Minisforum MS-01, CentOS Stream 9, Ceph Squid install
Table of Contents
doh!, install of my Ceph Squid 19 itself was in early March, but I forgot to publish the braindump I’d written to this blog.
It was really just following the official Using cephadm to Deploy a New Ceph Cluster for the latest version (squid) at the time of install.
Additionally, on the hottest summer days this year the MS-01 nodes were among the hardware nodes I switched off to keep the temperature in the work room at least below 40°C.
I actually ran a few minor update rounds (Dashboard / Administration / Upgrade) since the install in MArch happened. The original install was 19.2.1 today, 2025-09-06, it’s at 19.2.3 both GUI led updates were pleasantly hassle free. Nice!
Documentation pages to RTFM
- https://docs.ceph.com/en/squid/cephadm/install/#distribution-specific-installations in my case the subsection CentOS Stream
Bootstrap
As per the section Running the bootstrap command of the Ceph Squid docs.
[root@ms-01-01 ~]# cephadm bootstrap --mon-ip 192.168.40.151 --allow-fqdn-hostname --cluster-network "192.168.30.0/24"
Click to see full bootstrap output.
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 5.4.0 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: cee8622c-fb45-11ef-9938-5847ca797c06
Verifying IP 192.168.40.151 port 3300 ...
Verifying IP 192.168.40.151 port 6789 ...
Mon IP `192.168.40.151` is in CIDR network `192.168.40.0/24`
Mon IP `192.168.40.151` is in CIDR network `192.168.40.0/24`
Pulling container image quay.io/ceph/ceph:v19...
Ceph version: ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
firewalld ready
Enabling firewalld service ceph-mon in current zone...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting public_network to 192.168.40.0/24 in mon config section
Setting cluster_network to 192.168.30.0/24
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 0.0.0.0:9283 ...
Verifying port 0.0.0.0:8765 ...
Verifying port 0.0.0.0:8443 ...
firewalld ready
Enabling firewalld service ceph in current zone...
firewalld ready
Enabling firewalld port 9283/tcp in current zone...
Enabling firewalld port 8765/tcp in current zone...
Enabling firewalld port 8443/tcp in current zone...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr not available, waiting (4/15)...
mgr not available, waiting (5/15)...
mgr not available, waiting (6/15)...
mgr not available, waiting (7/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to /etc/ceph/ceph.pub
Adding key to root@localhost authorized_keys...
Adding host ms-01-01.storage.pcfe.net...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Deploying ceph-exporter service with default placement...
Deploying prometheus service with default placement...
Deploying grafana service with default placement...
Deploying node-exporter service with default placement...
Deploying alertmanager service with default placement...
Enabling the dashboard module...
Waiting for the mgr to restart...
Waiting for mgr epoch 9...
mgr epoch 9 is available
Generating a dashboard self-signed certificate...
Creating initial admin user...
Fetching dashboard port number...
firewalld ready
Ceph Dashboard is now available at:
URL: https://ms-01-01.storage.pcfe.net:8443/
User: admin
Password: [REDACTED]
Enabling client.admin keyring and conf on hosts with "admin" label
Saving cluster configuration to /var/lib/ceph/cee8622c-fb45-11ef-9938-5847ca797c06/config directory
You can access the Ceph CLI as following in case of multi-cluster or non-default config:
sudo /usr/sbin/cephadm shell --fsid cee8622c-fb45-11ef-9938-5847ca797c06 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
Or, if you are only running a single cluster on this host:
sudo /usr/sbin/cephadm shell
Please consider enabling telemetry to help improve Ceph:
ceph telemetry on
For more information see:
https://docs.ceph.com/en/latest/mgr/telemetry/
Bootstrap complete.
[root@ms-01-01 ~]#
Distribute ssh pubkey to all other nodes
Distribute the key that got generated during bootstrapping to the other 4 nodes.
[root@ms-01-01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ms-01-…
If you are not in a homelab, then please RTFM at both https://docs.ceph.com/en/squid/cephadm/install/#deployment-with-custom-ssh-keys and https://docs.ceph.com/en/squid/cephadm/install/#deployment-with-ca-signed-ssh-keys before you copy ssh keys willy nilly in prod.
Use cephadm shell on the bootstrap node
Now that it was bootstrapped, all further ceph command swill happen in a cephadm shell
For a start, I share some telemetry data with the community.
[root@ms-01-01 ~]# cephadm shell
[…]
[ceph: root@ms-01-01 /]#
[ceph: root@ms-01-01 /]# ceph version
ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)
[ceph: root@ms-01-01 /]# ceph telemetry on
Error EPERM: Telemetry data is licensed under the Community Data License Agreement - Sharing - Version 1.0 (https://cdla.io/sharing-1-0/).
To enable, add '--license sharing-1-0' to the 'ceph telemetry on' command.
[ceph: root@ms-01-01 /]# ceph telemetry on --license sharing-1-0
Telemetry is on.
Some channels are disabled, please enable with:
`ceph telemetry enable channel perf`
[ceph: root@ms-01-01 /]# ceph orch host ls
HOST ADDR LABELS STATUS
ms-01-01.storage.pcfe.net 192.168.40.151 _admin
1 hosts in cluster
Add hosts
[ceph: root@ms-01-01 /]# ceph orch host add ms-01-02.storage.pcfe.net 192.168.40.152 --labels _admin
Added host 'ms-01-02.storage.pcfe.net' with addr '192.168.40.152'
[ceph: root@ms-01-01 /]# ceph orch host add ms-01-03.storage.pcfe.net 192.168.40.153 --labels _admin
Added host 'ms-01-03.storage.pcfe.net' with addr '192.168.40.153'
[ceph: root@ms-01-01 /]# ceph orch host add ms-01-04.storage.pcfe.net 192.168.40.154 --labels _admin
Added host 'ms-01-04.storage.pcfe.net' with addr '192.168.40.154'
[ceph: root@ms-01-01 /]# ceph orch host add ms-01-05.storage.pcfe.net 192.168.40.155 --labels _admin
Added host 'ms-01-05.storage.pcfe.net' with addr '192.168.40.155'
[ceph: root@ms-01-01 /]# ceph log last cephadm ; ceph -W cephadm
NOTE:: later more labels were added, some in the Dashboard some in the cephadm shell.
Add unused storage
What I want is to consume all available and unused devices on my 5 nodes and that the OSDs are encrypted, the performance hit for luks is small and when one of the cheap consumer NVMe dies, I want to not have to faff around with wiping the device before I put it on the electronics recycling heap.
See also https://docs.ceph.com/en/squid/cephadm/install/#adding-storage
I do not want to use the all available devices default
What did not work for me was the default for all available devices.
Click to see details of me trying that
[ceph: root@ms-01-01 /]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@ms-01-01 /]# ceph log last cephadm ; ceph -W cephadm
[…]
2025-03-07T11:40:08.153124+0000 mgr.ms-01-01.pnupxa (mgr.14199) 704 : cephadm [INF] Marking host: ms-01-01.storage.pcfe.net for OSDSpec preview refresh.
2025-03-07T11:40:08.153163+0000 mgr.ms-01-01.pnupxa (mgr.14199) 705 : cephadm [INF] Marking host: ms-01-02.storage.pcfe.net for OSDSpec preview refresh.
2025-03-07T11:40:08.153190+0000 mgr.ms-01-01.pnupxa (mgr.14199) 706 : cephadm [INF] Marking host: ms-01-03.storage.pcfe.net for OSDSpec preview refresh.
2025-03-07T11:40:08.153211+0000 mgr.ms-01-01.pnupxa (mgr.14199) 707 : cephadm [INF] Marking host: ms-01-04.storage.pcfe.net for OSDSpec preview refresh.
2025-03-07T11:40:08.153233+0000 mgr.ms-01-01.pnupxa (mgr.14199) 708 : cephadm [INF] Marking host: ms-01-05.storage.pcfe.net for OSDSpec preview refresh.
2025-03-07T11:40:08.153476+0000 mgr.ms-01-01.pnupxa (mgr.14199) 709 : cephadm [INF] Saving service osd.all-available-devices spec with placement *
2025-03-07T11:40:19.586771+0000 mgr.ms-01-01.pnupxa (mgr.14199) 715 : cephadm [INF] Deploying daemon osd.4 on ms-01-03.storage.pcfe.net
2025-03-07T11:40:19.651087+0000 mgr.ms-01-01.pnupxa (mgr.14199) 717 : cephadm [INF] Deploying daemon osd.2 on ms-01-02.storage.pcfe.net
2025-03-07T11:40:19.697048+0000 mgr.ms-01-01.pnupxa (mgr.14199) 718 : cephadm [INF] Deploying daemon osd.3 on ms-01-05.storage.pcfe.net
2025-03-07T11:40:19.810508+0000 mgr.ms-01-01.pnupxa (mgr.14199) 719 : cephadm [INF] Deploying daemon osd.0 on ms-01-01.storage.pcfe.net
2025-03-07T11:40:19.951175+0000 mgr.ms-01-01.pnupxa (mgr.14199) 720 : cephadm [INF] Deploying daemon osd.1 on ms-01-04.storage.pcfe.net
2025-03-07T11:40:28.379302+0000 mgr.ms-01-01.pnupxa (mgr.14199) 725 : cephadm [INF] Deploying daemon osd.5 on ms-01-02.storage.pcfe.net
2025-03-07T11:40:28.510591+0000 mgr.ms-01-01.pnupxa (mgr.14199) 726 : cephadm [INF] Deploying daemon osd.6 on ms-01-03.storage.pcfe.net
2025-03-07T11:40:28.558902+0000 mgr.ms-01-01.pnupxa (mgr.14199) 727 : cephadm [INF] Deploying daemon osd.8 on ms-01-01.storage.pcfe.net
2025-03-07T11:40:28.669190+0000 mgr.ms-01-01.pnupxa (mgr.14199) 728 : cephadm [INF] Deploying daemon osd.7 on ms-01-05.storage.pcfe.net
2025-03-07T11:40:28.980062+0000 mgr.ms-01-01.pnupxa (mgr.14199) 729 : cephadm [INF] Deploying daemon osd.9 on ms-01-04.storage.pcfe.net
cluster:
id: cee8622c-fb45-11ef-9938-5847ca797c06
health: HEALTH_OK
services:
mon: 5 daemons, quorum ms-01-01,ms-01-02,ms-01-05,ms-01-03,ms-01-04 (age 4m)
mgr: ms-01-01.pnupxa(active, since 19m), standbys: ms-01-02.yuzhvl
osd: 10 osds: 1 up (since 1.01027s), 10 in (since 19s)
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
2025-03-07T11:40:47.650454+0000 mgr.ms-01-01.pnupxa [INF] Detected new or changed devices on ms-01-04.storage.pcfe.net
2025-03-07T11:40:47.660575+0000 mgr.ms-01-01.pnupxa [INF] Detected new or changed devices on ms-01-03.storage.pcfe.net
2025-03-07T11:40:47.689086+0000 mgr.ms-01-01.pnupxa [INF] Adjusting osd_memory_target on ms-01-04.storage.pcfe.net to 9447M
2025-03-07T11:40:47.698657+0000 mgr.ms-01-01.pnupxa [INF] Adjusting osd_memory_target on ms-01-03.storage.pcfe.net to 9449M
2025-03-07T11:40:47.724974+0000 mgr.ms-01-01.pnupxa [INF] Detected new or changed devices on ms-01-05.storage.pcfe.net
2025-03-07T11:40:47.744682+0000 mgr.ms-01-01.pnupxa [INF] Adjusting osd_memory_target on ms-01-05.storage.pcfe.net to 9449M
2025-03-07T11:40:47.831136+0000 mgr.ms-01-01.pnupxa [INF] Detected new or changed devices on ms-01-02.storage.pcfe.net
2025-03-07T11:40:47.851474+0000 mgr.ms-01-01.pnupxa [INF] Adjusting osd_memory_target on ms-01-02.storage.pcfe.net to 7401M
2025-03-07T11:40:48.522158+0000 mgr.ms-01-01.pnupxa [INF] Detected new or changed devices on ms-01-01.storage.pcfe.net
2025-03-07T11:40:48.559680+0000 mgr.ms-01-01.pnupxa [INF] Adjusting osd_memory_target on ms-01-01.storage.pcfe.net to 5863M
Awww, frack, that ends up with NOT encrypted devices.
It’s not what I want because this does not enable at rest encryption, so I disable that by setting it to unmanaged.
[ceph: root@ms-01-01 /]# ceph orch apply osd --all-available-devices --unmanaged=true
Scheduled osd.all-available-devices update...
Using a custom drive specification
And whipped up a primitive drive spec, mainly to get encrypted: true in.
See also: https://docs.ceph.com/en/squid/cephadm/services/osd/#advanced-osd-service-specifications
Create the drive spec
- node name starts with ms-01-
- device does not rotate
- OSD will be encrypted
[root@ms-01-01 ~]# cat ms-01-SSD-as-OSD-spec.yml
service_type: osd
service_id: osd_spec_MS-01_SSDs
placement:
host_pattern: 'ms-01-*'
encrypted: true
data_devices:
rotational: 0
Apply drive spec
Work around my earlier mistake
Because I first applied a drive spec that gave me non-encrypted OSDs I no longer had unused devices, now I could of course throw away the whole cluster, wipe all storage and start from zero, but why would I do that when I have many OSDs and very very little capacity used. I’ll just do them in a semi rolling fashion
Click to see details of me first applying the wrong spec and then doing nuke&pave on one OSD to free it as a test
[root@ms-01-01 ~]# cephadm shell --mount ms-01-SSD-as-OSD-spec.yml:/var/lib/ceph/osd/ms-01-SSD-as-OSD-spec.yml
Inferring fsid cee8622c-fb45-11ef-9938-5847ca797c06
Inferring config /var/lib/ceph/cee8622c-fb45-11ef-9938-5847ca797c06/mon.ms-01-01/config
Not using image 'f2efb0401a30ec7eda97b6da76b314bd081fcb910cc5dcd826bc7c72c9dfdd7d' as it's not in list of non-dangling images with ceph=True label
[ceph: root@ms-01-01 /]# cd /var/lib/ceph/osd/
[ceph: root@ms-01-01 osd]# ls
ms-01-SSD-as-OSD-spec.yml
[ceph: root@ms-01-01 osd]# cat ms-01-SSD-as-OSD-spec.yml
service_type: osd
service_id: osd_spec_MS-01_SSDs
placement:
host_pattern: 'ms-01-*'
encrypted: true
data_devices:
rotational: 0
[ceph: root@ms-01-01 osd]# ceph orch apply -i ms-01-SSD-as-OSD-spec.yml --dry-run
WARNING! Dry-Runs are snapshots of a certain point in time and are bound
to the current inventory setup. If any of these conditions change, the
preview will be invalid. Please make sure to have a minimal
timeframe between planning and applying the specs.
####################
SERVICESPEC PREVIEWS
####################
+---------+------+--------+-------------+
|SERVICE |NAME |ADD_TO |REMOVE_FROM |
+---------+------+--------+-------------+
+---------+------+--------+-------------+
################
OSDSPEC PREVIEWS
################
Preview data is being generated.. Please re-run this command in a bit.
Ah well with no devices free the dry run could not give anything, so remove the first of the non crypted OSDs and check if drive spec picks it up later
[ceph: root@ms-01-01 osd]# ceph orch osd rm 0 --zap
Scheduled OSD(s) for removal.
[ceph: root@ms-01-01 osd]# ceph log last cephadm ; ceph -W cephadm
[…]
2025-03-08T11:01:53.573439+0000 mgr.ms-01-02.yuzhvl [INF] osd.0 now down
2025-03-08T11:01:53.573568+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.0 from ms-01-01.storage.pcfe.net -- ports []
2025-03-08T11:01:56.001333+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.0
2025-03-08T11:01:56.016993+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.0 on ms-01-01.storage.pcfe.net
2025-03-08T11:01:56.032853+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.0 on ms-01-01.storage.pcfe.net
2025-03-08T11:01:56.032882+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.0 on ms-01-01.storage.pcfe.net
2025-03-08T11:01:58.489295+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.0 on ms-01-01.storage.pcfe.net
2025-03-08T11:02:03.008587+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-01.storage.pcfe.net
another dry run
[ceph: root@ms-01-01 osd]# ceph orch apply -i ms-01-SSD-as-OSD-spec.yml --dry-run
WARNING! Dry-Runs are snapshots of a certain point in time and are bound
to the current inventory setup. If any of these conditions change, the
preview will be invalid. Please make sure to have a minimal
timeframe between planning and applying the specs.
####################
SERVICESPEC PREVIEWS
####################
+---------+------+--------+-------------+
|SERVICE |NAME |ADD_TO |REMOVE_FROM |
+---------+------+--------+-------------+
+---------+------+--------+-------------+
################
OSDSPEC PREVIEWS
################
+---------+---------------------+---------------------------+--------------+----+-----+
|SERVICE |NAME |HOST |DATA |DB |WAL |
+---------+---------------------+---------------------------+--------------+----+-----+
|osd |osd_spec_MS-01_SSDs |ms-01-01.storage.pcfe.net |/dev/nvme0n1 |- |- |
+---------+---------------------+---------------------------+--------------+----+-----+
[ceph: root@ms-01-01 osd]#
looks good, apply
[ceph: root@ms-01-01 osd]# ceph orch apply -i ms-01-SSD-as-OSD-spec.yml
Scheduled osd.osd_spec_MS-01_SSDs update...
Check at OS level that the OSD 0 was created as crypted
[root@ms-01-01 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme0n1 259:0 0 1.9T 0 disk
└─ceph--39fa7697--930a--4e88--8280--c82f04bd4b42-osd--block--6af44afd--e59e--4ce3--8906--bf790f4a8e4d 253:2 0 1.9T 0 lvm
└─TyyWnn-bjFU-X7ua-Asd5-sKin-Jsh3-w5EJX4 253:9 0 1.9T 0 crypt
nvme1n1 259:1 0 1.8T 0 disk
└─ceph--5c22e2bb--67ec--4c1b--a59d--203e2083a0dd-osd--block--24f804d8--c3d1--4512--adf1--90b438072061 253:3 0 1.8T 0 lvm
nvme2n1 259:2 0 953.9G 0 disk
├─nvme2n1p1 259:3 0 600M 0 part /boot/efi
├─nvme2n1p2 259:4 0 1G 0 part /boot
└─nvme2n1p3 259:5 0 200G 0 part
├─VG_OS-LV_root 253:0 0 4G 0 lvm /
├─VG_OS-LV_swap 253:1 0 1G 0 lvm [SWAP]
├─VG_OS-LV_var_crash 253:4 0 40G 0 lvm /var/crash
├─VG_OS-LV_var 253:5 0 12G 0 lvm /var
├─VG_OS-LV_home 253:6 0 1G 0 lvm /home
├─VG_OS-LV_var_log 253:7 0 10G 0 lvm /var/log
└─VG_OS-LV_containers 253:8 0 10G 0 lvm /var/lib/containers/storage/overlay
/var/lib/containers
[root@ms-01-01 ~]#
yupp looks much better, nvme0n1 is indeed encrypted.
as expected the freshly rolled ous OSD is encrypted.
Click to see details of me deleting more of the non encrypted ones, did it in 2 halves
[ceph: root@ms-01-01 /]# ceph orch osd rm 1 --zap
Scheduled OSD(s) for removal.
[ceph: root@ms-01-01 /]# ceph orch osd rm 2 --zap
Scheduled OSD(s) for removal.
[ceph: root@ms-01-01 /]# ceph orch osd rm 3 --zap
Scheduled OSD(s) for removal.
[ceph: root@ms-01-01 /]# ceph orch osd rm 4 --zap
Scheduled OSD(s) for removal.
wait a bit, then check on progress
[ceph: root@ms-01-01 /]# ceph orch osd rm status
OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT
1 ms-01-04.storage.pcfe.net draining 7 False False True 2025-03-08 11:08:10.780531
2 ms-01-02.storage.pcfe.net draining 4 False False True 2025-03-08 11:08:13.833413
3 ms-01-05.storage.pcfe.net draining 6 False False True 2025-03-08 11:09:58.385193
4 ms-01-03.storage.pcfe.net draining 2 False False True 2025-03-08 11:08:57.038480
[ceph: root@ms-01-01 /]#
Went well, as expected
[ceph: root@ms-01-01 /]# ceph -W cephadm
cluster:
id: cee8622c-fb45-11ef-9938-5847ca797c06
health: HEALTH_OK
services:
mon: 5 daemons, quorum ms-01-01,ms-01-02,ms-01-05,ms-01-03,ms-01-04 (age 19h)
mgr: ms-01-02.yuzhvl(active, since 20h), standbys: ms-01-01.pnupxa
osd: 10 osds: 10 up (since 10m), 10 in (since 19h); 12 remapped pgs
data:
pools: 3 pools, 65 pgs
objects: 30.58k objects, 119 GiB
usage: 376 GiB used, 18 TiB / 18 TiB avail
pgs: 19208/91746 objects misplaced (20.936%)
53 active+clean
10 active+remapped+backfill_wait
2 active+remapped+backfilling
io:
recovery: 168 MiB/s, 41 objects/s
2025-03-08T11:19:10.697465+0000 mgr.ms-01-02.yuzhvl [INF] osd.1 now down
2025-03-08T11:19:10.697653+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.1 from ms-01-04.storage.pcfe.net -- ports []
2025-03-08T11:19:13.292850+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.1
2025-03-08T11:19:13.309298+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.1 on ms-01-04.storage.pcfe.net
2025-03-08T11:19:13.325684+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.1 on ms-01-04.storage.pcfe.net
2025-03-08T11:19:13.325714+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.1 on ms-01-04.storage.pcfe.net
2025-03-08T11:19:15.753203+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.1 on ms-01-04.storage.pcfe.net
2025-03-08T11:19:16.770071+0000 mgr.ms-01-02.yuzhvl [INF] osd.4 now down
2025-03-08T11:19:16.770239+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.4 from ms-01-03.storage.pcfe.net -- ports []
2025-03-08T11:19:19.413494+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.4
2025-03-08T11:19:19.428111+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.4 on ms-01-03.storage.pcfe.net
2025-03-08T11:19:19.442515+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.4 on ms-01-03.storage.pcfe.net
2025-03-08T11:19:19.442577+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.4 on ms-01-03.storage.pcfe.net
2025-03-08T11:19:21.889073+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.4 on ms-01-03.storage.pcfe.net
2025-03-08T11:19:26.229376+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-04.storage.pcfe.net
2025-03-08T11:19:26.239572+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-03.storage.pcfe.net
2025-03-08T11:19:41.571565+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.1 on ms-01-03.storage.pcfe.net
2025-03-08T11:19:47.716121+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.4 on ms-01-04.storage.pcfe.net
2025-03-08T11:20:07.614992+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-03.storage.pcfe.net
2025-03-08T11:20:07.791014+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-04.storage.pcfe.net
2025-03-08T11:21:05.066704+0000 mgr.ms-01-02.yuzhvl [INF] osd.2 now down
2025-03-08T11:21:05.066860+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.2 from ms-01-02.storage.pcfe.net -- ports []
2025-03-08T11:21:07.644311+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.2
2025-03-08T11:21:07.659369+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.2 on ms-01-02.storage.pcfe.net
2025-03-08T11:21:07.673985+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.2 on ms-01-02.storage.pcfe.net
2025-03-08T11:21:07.674032+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.2 on ms-01-02.storage.pcfe.net
2025-03-08T11:21:10.122593+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.2 on ms-01-02.storage.pcfe.net
2025-03-08T11:21:13.973207+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-02.storage.pcfe.net
2025-03-08T11:21:15.757390+0000 mgr.ms-01-02.yuzhvl [INF] Adjusting osd_memory_target on ms-01-02.storage.pcfe.net to 14802M
2025-03-08T11:21:33.589860+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.2 on ms-01-02.storage.pcfe.net
2025-03-08T11:21:53.611449+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-02.storage.pcfe.net
2025-03-08T11:21:55.474212+0000 mgr.ms-01-02.yuzhvl [INF] Adjusting osd_memory_target on ms-01-02.storage.pcfe.net to 7401M
2025-03-08T11:35:29.777052+0000 mgr.ms-01-02.yuzhvl [INF] osd.3 now down
2025-03-08T11:35:29.777269+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.3 from ms-01-05.storage.pcfe.net -- ports []
2025-03-08T11:35:32.284114+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.3
2025-03-08T11:35:32.300918+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.3 on ms-01-05.storage.pcfe.net
2025-03-08T11:35:32.316676+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.3 on ms-01-05.storage.pcfe.net
2025-03-08T11:35:32.316708+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.3 on ms-01-05.storage.pcfe.net
2025-03-08T11:35:34.790736+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.3 on ms-01-05.storage.pcfe.net
2025-03-08T11:35:38.702871+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-05.storage.pcfe.net
2025-03-08T11:35:53.638463+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.3 on ms-01-05.storage.pcfe.net
2025-03-08T11:36:13.618714+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-05.storage.pcfe.net
then wait for all PGs to be active+clean again before removing the second half.
[ceph: root@ms-01-01 /]# ceph -w
cluster:
id: cee8622c-fb45-11ef-9938-5847ca797c06
health: HEALTH_OK
services:
mon: 5 daemons, quorum ms-01-01,ms-01-02,ms-01-05,ms-01-03,ms-01-04 (age 20h)
mgr: ms-01-02.yuzhvl(active, since 21h), standbys: ms-01-01.pnupxa
osd: 10 osds: 10 up (since 11m), 10 in (since 20h)
data:
pools: 3 pools, 65 pgs
objects: 30.58k objects, 119 GiB
usage: 367 GiB used, 18 TiB / 18 TiB avail
pgs: 65 active+clean
io:
recovery: 49 MiB/s, 12 objects/s
and remove the second half of not encrypted OSDs so that the new drive spec (with encrypted: true) can pick it up.
[ceph: root@ms-01-01 /]# ceph orch osd rm 5 6 7 8 9 --zap
Scheduled OSD(s) for removal.
[ceph: root@ms-01-01 /]# ceph orch osd rm status
OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT
5 ms-01-02.storage.pcfe.net started 18 False False True
6 ms-01-03.storage.pcfe.net started 18 False False True
7 ms-01-05.storage.pcfe.net started 18 False False True
8 ms-01-01.storage.pcfe.net started 20 False False True
9 ms-01-04.storage.pcfe.net started 18 False False True
[ceph: root@ms-01-01 /]#
[ceph: root@ms-01-01 /]# ceph log last cephadm ; ceph -W cephadm
[…]
2025-03-08T11:51:10.129970+0000 mgr.ms-01-02.yuzhvl (mgr.14224) 53411 : cephadm [INF] osd.8 crush weight is 1.81939697265625
2025-03-08T11:51:10.133867+0000 mgr.ms-01-02.yuzhvl (mgr.14224) 53412 : cephadm [INF] osd.5 crush weight is 1.81939697265625
2025-03-08T11:51:10.137239+0000 mgr.ms-01-02.yuzhvl (mgr.14224) 53413 : cephadm [INF] osd.6 crush weight is 1.81939697265625
2025-03-08T11:51:10.140274+0000 mgr.ms-01-02.yuzhvl (mgr.14224) 53414 : cephadm [INF] osd.9 crush weight is 1.81939697265625
2025-03-08T11:51:10.143226+0000 mgr.ms-01-02.yuzhvl (mgr.14224) 53415 : cephadm [INF] osd.7 crush weight is 1.81939697265625
2025-03-08T11:51:17.802807+0000 mgr.ms-01-02.yuzhvl (mgr.14224) 53444 : cephadm [INF] osd.9 weight is now 0.0
cluster:
id: cee8622c-fb45-11ef-9938-5847ca797c06
health: HEALTH_OK
services:
mon: 5 daemons, quorum ms-01-01,ms-01-02,ms-01-05,ms-01-03,ms-01-04 (age 20h)
mgr: ms-01-02.yuzhvl(active, since 21h), standbys: ms-01-01.pnupxa
osd: 10 osds: 10 up (since 15m), 10 in (since 20h); 25 remapped pgs
data:
pools: 3 pools, 65 pgs
objects: 30.58k objects, 119 GiB
usage: 370 GiB used, 18 TiB / 18 TiB avail
pgs: 21144/91746 objects misplaced (23.046%)
40 active+clean
20 active+remapped+backfill_wait
5 active+remapped+backfilling
io:
recovery: 294 MiB/s, 73 objects/s
2025-03-08T11:52:21.699945+0000 mgr.ms-01-02.yuzhvl [INF] osd.8 weight is now 0.0
2025-03-08T11:54:34.358011+0000 mgr.ms-01-02.yuzhvl [INF] osd.7 weight is now 0.0
2025-03-08T11:55:37.250217+0000 mgr.ms-01-02.yuzhvl [INF] osd.6 weight is now 0.0
2025-03-08T11:56:38.679752+0000 mgr.ms-01-02.yuzhvl [INF] osd.5 weight is now 0.0
2025-03-08T12:12:15.978654+0000 mgr.ms-01-02.yuzhvl [INF] osd.9 now down
2025-03-08T12:12:15.978804+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.9 from ms-01-04.storage.pcfe.net -- ports []
2025-03-08T12:12:18.509455+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.9
2025-03-08T12:12:18.526608+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.9 on ms-01-04.storage.pcfe.net
2025-03-08T12:12:18.541942+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.9 on ms-01-04.storage.pcfe.net
2025-03-08T12:12:18.541979+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.9 on ms-01-04.storage.pcfe.net
2025-03-08T12:12:21.000208+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.9 on ms-01-04.storage.pcfe.net
2025-03-08T12:12:24.896547+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-04.storage.pcfe.net
2025-03-08T12:12:44.852913+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.9 on ms-01-04.storage.pcfe.net
2025-03-08T12:13:05.216192+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-04.storage.pcfe.net
2025-03-08T12:22:09.446113+0000 mgr.ms-01-02.yuzhvl [INF] osd.7 now down
2025-03-08T12:22:09.446277+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.7 from ms-01-05.storage.pcfe.net -- ports []
2025-03-08T12:22:11.919234+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.7
2025-03-08T12:22:11.936494+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.7 on ms-01-05.storage.pcfe.net
2025-03-08T12:22:11.952321+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.7 on ms-01-05.storage.pcfe.net
2025-03-08T12:22:11.952350+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.7 on ms-01-05.storage.pcfe.net
2025-03-08T12:22:14.389828+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.7 on ms-01-05.storage.pcfe.net
2025-03-08T12:22:18.292163+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-05.storage.pcfe.net
2025-03-08T12:22:40.156873+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.7 on ms-01-05.storage.pcfe.net
2025-03-08T12:23:01.152178+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-05.storage.pcfe.net
2025-03-08T12:25:17.113831+0000 mgr.ms-01-02.yuzhvl [INF] osd.8 now down
2025-03-08T12:25:17.113965+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.8 from ms-01-01.storage.pcfe.net -- ports []
2025-03-08T12:25:19.572978+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.8
2025-03-08T12:25:19.586673+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.8 on ms-01-01.storage.pcfe.net
2025-03-08T12:25:19.601872+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.8 on ms-01-01.storage.pcfe.net
2025-03-08T12:25:19.601901+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.8 on ms-01-01.storage.pcfe.net
2025-03-08T12:25:21.987384+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.8 on ms-01-01.storage.pcfe.net
2025-03-08T12:25:22.598562+0000 mgr.ms-01-02.yuzhvl [INF] osd.5 now down
2025-03-08T12:25:22.598706+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.5 from ms-01-02.storage.pcfe.net -- ports []
2025-03-08T12:25:25.216537+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.5
2025-03-08T12:25:25.232876+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.5 on ms-01-02.storage.pcfe.net
2025-03-08T12:25:25.248294+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.5 on ms-01-02.storage.pcfe.net
2025-03-08T12:25:25.248325+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.5 on ms-01-02.storage.pcfe.net
2025-03-08T12:25:27.666190+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.5 on ms-01-02.storage.pcfe.net
2025-03-08T12:25:28.680404+0000 mgr.ms-01-02.yuzhvl [INF] osd.6 now down
2025-03-08T12:25:28.680544+0000 mgr.ms-01-02.yuzhvl [INF] Removing daemon osd.6 from ms-01-03.storage.pcfe.net -- ports []
2025-03-08T12:25:31.282911+0000 mgr.ms-01-02.yuzhvl [INF] Removing key for osd.6
2025-03-08T12:25:31.300272+0000 mgr.ms-01-02.yuzhvl [INF] Successfully removed osd.6 on ms-01-03.storage.pcfe.net
2025-03-08T12:25:31.317684+0000 mgr.ms-01-02.yuzhvl [INF] Successfully purged osd.6 on ms-01-03.storage.pcfe.net
2025-03-08T12:25:31.317714+0000 mgr.ms-01-02.yuzhvl [INF] Zapping devices for osd.6 on ms-01-03.storage.pcfe.net
2025-03-08T12:25:33.716134+0000 mgr.ms-01-02.yuzhvl [INF] Successfully zapped devices for osd.6 on ms-01-03.storage.pcfe.net
2025-03-08T12:25:37.583541+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-02.storage.pcfe.net
2025-03-08T12:25:38.037671+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-03.storage.pcfe.net
2025-03-08T12:25:39.588782+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-01.storage.pcfe.net
2025-03-08T12:25:55.777847+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.8 on ms-01-03.storage.pcfe.net
2025-03-08T12:25:56.142570+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.6 on ms-01-02.storage.pcfe.net
2025-03-08T12:25:56.356299+0000 mgr.ms-01-02.yuzhvl [INF] Deploying daemon osd.5 on ms-01-01.storage.pcfe.net
2025-03-08T12:26:16.769890+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-03.storage.pcfe.net
2025-03-08T12:26:16.972150+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-02.storage.pcfe.net
2025-03-08T12:26:17.467832+0000 mgr.ms-01-02.yuzhvl [INF] Detected new or changed devices on ms-01-01.storage.pcfe.net
Standard procedure for custom drive spec
If I’d started with empty drives to use for OSDs, then this would just have been the commands
cephadm shell --mount <my YAML>ceph orch apply -i ms-01-SSD-as-OSD-spec.yml(maybe first a--dry-runthough)
[root@ms-01-01 ~]# cephadm shell --mount ms-01-SSD-as-OSD-spec.yml:/var/lib/ceph/osd/ms-01-SSD-as-OSD-spec.yml
[ceph: root@ms-01-01 /]# cd /var/lib/ceph/osd/
[ceph: root@ms-01-01 osd]# ls
ms-01-SSD-as-OSD-spec.yml
[ceph: root@ms-01-01 osd]# cat ms-01-SSD-as-OSD-spec.yml
service_type: osd
service_id: osd_spec_MS-01_SSDs
placement:
host_pattern: 'ms-01-*'
encrypted: true
data_devices:
rotational: 0
[ceph: root@ms-01-01 osd]# ceph orch apply -i ms-01-SSD-as-OSD-spec.yml
Scheduled osd.osd_spec_MS-01_SSDs update...
Check that OSDs are encrypted
After the above song and dance, of course I double checked. All OSDs are now encrypted at rest. Meaning that if I wear one out, I can remove it drom the node and throw it into recycling without worrying about anyone getting access to the data stored on that device.
Obviously only as long as I do not thror a not encrypted BaseOS storage of the node into the same recycling run.
NOTE:: this obviously does not protect the data in case of theft/recycling/sale of a whole node, the point here is really just peace of mind when I physically remove an OSD storage device from a cluster node mostly because I wore it out.
Click to see lsbkl output of all 5 nodes
pcfe@t3600 ~ $ for i in $(seq 1 5) ; do echo MS-01 number ${i} ; ssh ms-01-0${i} -l root lsblk ;echo ; done
MS-01 number 1
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme0n1 259:0 0 1.9T 0 disk
└─ceph--39fa7697--930a--4e88--8280--c82f04bd4b42-osd--block--6af44afd--e59e--4ce3--8906--bf790f4a8e4d 253:2 0 1.9T 0 lvm
└─TyyWnn-bjFU-X7ua-Asd5-sKin-Jsh3-w5EJX4 253:9 0 1.9T 0 crypt
nvme1n1 259:1 0 1.8T 0 disk
└─ceph--394db331--20da--401c--84c5--50397547d3c3-osd--block--e47aef1e--585c--4555--95a8--f8a06943097c 253:3 0 1.8T 0 lvm
└─Appi83-m73V-FjvK-pLJH-J8Ob-HsN0-uJpIaZ 253:10 0 1.8T 0 crypt
nvme2n1 259:2 0 953.9G 0 disk
├─nvme2n1p1 259:3 0 600M 0 part /boot/efi
├─nvme2n1p2 259:4 0 1G 0 part /boot
└─nvme2n1p3 259:5 0 200G 0 part
├─VG_OS-LV_root 253:0 0 4G 0 lvm /
├─VG_OS-LV_swap 253:1 0 1G 0 lvm [SWAP]
├─VG_OS-LV_var_crash 253:4 0 40G 0 lvm /var/crash
├─VG_OS-LV_var 253:5 0 12G 0 lvm /var
├─VG_OS-LV_home 253:6 0 1G 0 lvm /home
├─VG_OS-LV_var_log 253:7 0 10G 0 lvm /var/log
└─VG_OS-LV_containers 253:8 0 10G 0 lvm /var/lib/containers/storage/overlay
/var/lib/containers
MS-01 number 2
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme2n1 259:0 0 1.8T 0 disk
└─ceph--487b4a4c--b0b3--4a81--8f03--4f10c70be9d9-osd--block--ffbfb311--08e7--4d25--a397--ceea88027638 253:8 0 1.8T 0 lvm
└─hBVzZT-KBP3-okfr-UUxw-zIQb-9Nxo-HRMYBD 253:10 0 1.8T 0 crypt
nvme1n1 259:1 0 1.8T 0 disk
└─ceph--8d3763d6--e03a--4443--a1c5--0ab11b2d7f6b-osd--block--aba0fdee--23fd--4672--8cd6--65a61c082b8c 253:7 0 1.8T 0 lvm
└─zEGV9m-4iUT-p3RH-vaCX-oNFx-lpVv-SM9AUa 253:9 0 1.8T 0 crypt
nvme0n1 259:2 0 953.9G 0 disk
├─nvme0n1p1 259:3 0 600M 0 part /boot/efi
├─nvme0n1p2 259:4 0 1G 0 part /boot
└─nvme0n1p3 259:5 0 200G 0 part
├─VG_OS-LV_root 253:0 0 4G 0 lvm /
├─VG_OS-LV_swap 253:1 0 1G 0 lvm [SWAP]
├─VG_OS-LV_var_crash 253:2 0 40G 0 lvm /var/crash
├─VG_OS-LV_var 253:3 0 12G 0 lvm /var
├─VG_OS-LV_home 253:4 0 1G 0 lvm /home
├─VG_OS-LV_var_log 253:5 0 10G 0 lvm /var/log
└─VG_OS-LV_containers 253:6 0 10G 0 lvm /var/lib/containers/storage/overlay
/var/lib/containers
MS-01 number 3
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme0n1 259:0 0 1.8T 0 disk
└─ceph--9930a217--399a--452d--9089--110255862676-osd--block--88f72a42--f96d--4463--9be1--2a3e0f980212 253:7 0 1.8T 0 lvm
└─IKDjmd-Tmno-TPvv-zwDk-IaVz-ZclB-8lpdhA 253:9 0 1.8T 0 crypt
nvme1n1 259:1 0 1.8T 0 disk
└─ceph--a20aa91d--cea4--42f8--b1d1--0faab214a74d-osd--block--6f962829--1f98--4cb5--afc6--ba29e5367475 253:8 0 1.8T 0 lvm
└─CuKsDQ-EUhh-osRb-hVdC-syPC-p3lz-coVA4K 253:10 0 1.8T 0 crypt
nvme2n1 259:2 0 953.9G 0 disk
├─nvme2n1p1 259:3 0 600M 0 part /boot/efi
├─nvme2n1p2 259:4 0 1G 0 part /boot
└─nvme2n1p3 259:5 0 200G 0 part
├─VG_OS-LV_root 253:0 0 4G 0 lvm /
├─VG_OS-LV_swap 253:1 0 1G 0 lvm [SWAP]
├─VG_OS-LV_var_crash 253:2 0 40G 0 lvm /var/crash
├─VG_OS-LV_var 253:3 0 12G 0 lvm /var
├─VG_OS-LV_home 253:4 0 1G 0 lvm /home
├─VG_OS-LV_var_log 253:5 0 10G 0 lvm /var/log
└─VG_OS-LV_containers 253:6 0 10G 0 lvm /var/lib/containers/storage/overlay
/var/lib/containers
MS-01 number 4
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme2n1 259:0 0 1.8T 0 disk
└─ceph--5ac5826d--1ae5--4c18--a3b8--982d5e3cb33f-osd--block--994ef9b8--c453--4f61--9c91--8df05ac9ddd3 253:8 0 1.8T 0 lvm
└─mWGrQW-SEOu-i31F-DzrU-yEPv-xz2g-H28f3l 253:10 0 1.8T 0 crypt
nvme1n1 259:1 0 953.9G 0 disk
├─nvme1n1p1 259:2 0 600M 0 part /boot/efi
├─nvme1n1p2 259:3 0 1G 0 part /boot
└─nvme1n1p3 259:4 0 200G 0 part
├─VG_OS-LV_root 253:0 0 4G 0 lvm /
├─VG_OS-LV_swap 253:1 0 1G 0 lvm [SWAP]
├─VG_OS-LV_var_crash 253:2 0 40G 0 lvm /var/crash
├─VG_OS-LV_var 253:3 0 12G 0 lvm /var
├─VG_OS-LV_home 253:4 0 1G 0 lvm /home
├─VG_OS-LV_var_log 253:5 0 10G 0 lvm /var/log
└─VG_OS-LV_containers 253:6 0 10G 0 lvm /var/lib/containers/storage/overlay
/var/lib/containers
nvme0n1 259:5 0 1.9T 0 disk
└─ceph--ee5e0a69--45c3--4e49--8b6e--0023c1c16257-osd--block--67b39a1b--6b31--4daf--9fd2--ec70a734624d 253:7 0 1.9T 0 lvm
└─9aHi7q-XIep-d50G-ejp5-0XIg-GiaS-N2yrWE 253:9 0 1.9T 0 crypt
MS-01 number 5
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme1n1 259:0 0 1.8T 0 disk
└─ceph--2491bd1c--43e8--4476--b6cf--b41c00d6699e-osd--block--d7617b6e--5435--4a8e--9bb3--4260c7ccc9d8 253:8 0 1.8T 0 lvm
└─4BW01s-srfz-kLEw-y3Vq-iMiW-xJ0Z-Hup5Rb 253:10 0 1.8T 0 crypt
nvme0n1 259:1 0 1.8T 0 disk
└─ceph--e538ff2e--7854--4735--a9c3--be1aa5e04668-osd--block--f4a7f625--aa52--4c61--be87--a182252927cc 253:5 0 1.8T 0 lvm
└─NyhkOy-bB2Z-f9aT-YgY1-5z3d-MM2C-7zFMz3 253:9 0 1.8T 0 crypt
nvme2n1 259:2 0 953.9G 0 disk
├─nvme2n1p1 259:3 0 600M 0 part /boot/efi
├─nvme2n1p2 259:4 0 1G 0 part /boot
└─nvme2n1p3 259:5 0 200G 0 part
├─VG_OS-LV_root 253:0 0 4G 0 lvm /
├─VG_OS-LV_swap 253:1 0 1G 0 lvm [SWAP]
├─VG_OS-LV_var_crash 253:2 0 40G 0 lvm /var/crash
├─VG_OS-LV_var 253:3 0 12G 0 lvm /var
├─VG_OS-LV_home 253:4 0 1G 0 lvm /home
├─VG_OS-LV_var_log 253:6 0 10G 0 lvm /var/log
└─VG_OS-LV_containers 253:7 0 10G 0 lvm /var/lib/containers/storage/overlay
/var/lib/containers
pcfe@t3600 ~ $
MGR redirects to FQDN
I want this setting enabled so that when I access e.g. ms-01-01 but the dashboard is currently running on -02, that I get redirected to a fully qualified domain name, not an IP.
This makes my browser happier for TLS verification and it make my life easier WRT readability.
[ceph: root@ms-01-01 /]# ceph config set mgr mgr/dashboard/redirect_resolve_ip_addr True
Create additional pools
Create pools for benchmarks
ceph osd pool create testbench […]
ceph osd pool create rbdbench 32
ceph osd pool application enable rbdbench rbd
Create a CephFS
RTFM at https://docs.ceph.com/en/squid/cephfs/fs-volumes/#fs-volumes-and-subvolumes
For now one volume, subvolumes and subvolume groups will come later.
[ceph: root@ms-01-01 /]# ceph fs volume create cephfs --placement="label:mds"
Pool list
[ceph: root@ms-01-01 /]# ceph osd pool ls detail
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 10.00
pool 2 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 389 lfor 0/385/383 flags hashpspool stripe_width 0 application rbd read_balance_score 1.56
pool 3 'rbdbench' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 397 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.88
pool 4 'cephfs.cephfs.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 2630 lfor 0/0/2600 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 3.76
pool 5 'cephfs.cephfs.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 2606 lfor 0/0/2604 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 1.17
Performance baseline
The point here is to get numbers to compare to if I ever make changes. e.g. take the public_network interfaces from 1 Gigabit to 2.5 Gigabit if I ever get a 2.5 capable switch.
Rados, sequential write test, pool with 32 PGs
[ceph: root@ms-01-01 /]# date
Sat Sep 6 15:38:15 UTC 2025
[ceph: root@ms-01-01 /]# rados bench -p testbench 900 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 900 seconds or 0 objects
Object prefix: benchmark_data_ms-01-01.storage.pcfe.net_47
Click to see full run
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 46 30 119.986 120 0.891861 0.283605
2 16 81 65 129.986 140 0.728704 0.373012
3 16 110 94 125.32 116 0.357917 0.413796
4 16 147 131 130.986 148 0.151139 0.427921
5 16 179 163 130.387 128 0.0169029 0.460131
6 16 210 194 129.319 124 0.121417 0.456936
7 16 242 226 129.129 128 0.862833 0.473473
8 16 273 257 128.486 124 1.16294 0.473367
9 16 304 288 127.986 124 0.0874029 0.475227
10 16 336 320 127.986 128 0.0159942 0.473775
11 16 367 351 127.623 124 0.0881047 0.480436
12 16 397 381 126.987 120 1.24982 0.482688
13 16 429 413 127.064 128 0.241703 0.48436
14 16 459 443 126.558 120 0.12791 0.484436
15 16 495 479 127.72 144 0.0896826 0.480814
16 16 527 511 127.737 128 0.0123545 0.478949
17 16 563 547 128.691 144 0.182025 0.476148
18 16 595 579 128.652 128 0.275573 0.479161
19 16 625 609 128.196 120 0.247677 0.481757
2025-09-06T15:39:18.769105+0000 min lat: 0.0120121 max lat: 2.16865 avg lat: 0.48376
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 655 639 127.786 120 0.186276 0.48376
21 16 685 669 127.415 120 0.29716 0.483704
22 16 717 701 127.441 128 0.0879459 0.486324
23 16 745 729 126.769 112 1.52949 0.492499
24 16 780 764 127.319 140 0.15574 0.489639
25 16 809 793 126.866 116 1.76825 0.493005
26 16 842 826 127.063 132 1.79154 0.491684
27 16 873 857 126.95 124 0.0183032 0.490702
28 16 906 890 127.13 132 0.706151 0.490792
29 16 938 922 127.159 128 0.582968 0.493053
30 16 973 957 127.587 140 0.0132536 0.491336
31 16 1004 988 127.471 124 0.169061 0.494953
32 16 1033 1017 127.112 116 0.0919299 0.494588
33 16 1068 1052 127.502 140 0.15031 0.492852
34 16 1098 1082 127.281 120 0.0898387 0.491945
35 16 1131 1115 127.415 132 0.139664 0.492299
36 16 1166 1150 127.765 140 0.573958 0.494613
37 16 1199 1183 127.879 132 0.128009 0.493121
38 16 1228 1212 127.566 116 0.0936423 0.492605
39 16 1259 1243 127.474 124 1.80107 0.49548
2025-09-06T15:39:38.770982+0000 min lat: 0.0115416 max lat: 2.50556 avg lat: 0.493221
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1296 1280 127.987 148 1.5973 0.493221
41 16 1327 1311 127.889 124 0.213491 0.494597
42 16 1357 1341 127.701 120 0.0147104 0.494833
43 16 1389 1373 127.708 128 0.0211331 0.49667
44 16 1426 1410 128.169 148 0.0949964 0.494086
45 16 1455 1439 127.898 116 0.155738 0.494075
46 16 1484 1468 127.639 116 0.34259 0.496389
47 16 1513 1497 127.391 116 0.132117 0.495907
48 16 1548 1532 127.654 140 0.0129881 0.494806
49 16 1580 1564 127.66 128 0.0882796 0.496133
50 16 1614 1598 127.827 136 0.0916736 0.495361
51 16 1647 1631 127.908 132 1.11173 0.495569
52 16 1677 1661 127.756 120 0.507702 0.496079
53 16 1709 1693 127.76 128 0.129273 0.497577
54 16 1740 1724 127.691 124 0.0254375 0.495974
55 16 1772 1756 127.696 128 0.0227182 0.4963
56 16 1806 1790 127.844 136 0.329753 0.496575
57 16 1837 1821 127.776 124 0.0167499 0.495306
58 16 1872 1856 127.987 140 0.196348 0.494583
59 16 1905 1889 128.055 132 0.322432 0.49442
2025-09-06T15:39:58.773032+0000 min lat: 0.0112911 max lat: 2.50556 avg lat: 0.494783
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 1936 1920 127.987 124 0.0991983 0.494783
61 16 1973 1957 128.315 148 0.0123943 0.494378
62 16 2005 1989 128.31 128 1.01152 0.49553
63 16 2037 2021 128.304 128 0.127416 0.4931
64 16 2072 2056 128.487 140 0.0861447 0.491937
65 16 2108 2092 128.725 144 1.97594 0.492938
66 16 2139 2123 128.653 124 0.996572 0.493387
67 16 2169 2153 128.524 120 0.897124 0.495383
68 16 2197 2181 128.281 112 0.721435 0.495552
69 16 2226 2210 128.103 116 1.3611 0.49589
70 16 2260 2244 128.215 136 1.05833 0.496308
71 16 2287 2271 127.931 108 0.0877596 0.497459
72 16 2319 2303 127.931 128 1.23147 0.496159
73 16 2354 2338 128.097 140 1.76149 0.495753
74 16 2389 2373 128.257 140 0.0179643 0.495055
75 16 2423 2407 128.36 136 0.0859759 0.494071
76 16 2451 2435 128.145 112 0.0231612 0.494963
77 16 2480 2464 127.987 116 0.419215 0.495413
78 16 2515 2499 128.141 140 1.77961 0.49505
79 16 2548 2532 128.189 132 0.433241 0.495388
2025-09-06T15:40:18.775246+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.49461
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 2581 2565 128.237 132 0.139117 0.49461
81 16 2615 2599 128.332 136 0.0945735 0.49366
82 16 2647 2631 128.328 128 0.190526 0.493379
83 16 2677 2661 128.227 120 0.380594 0.494329
84 16 2706 2690 128.082 116 0.0924989 0.495199
85 16 2743 2727 128.316 148 0.0829465 0.494778
86 16 2774 2758 128.266 124 2.22245 0.49439
87 16 2803 2787 128.124 116 0.36343 0.495131
88 16 2835 2819 128.123 128 0.368028 0.495404
89 16 2873 2857 128.391 152 0.275797 0.495344
90 16 2902 2886 128.253 116 0.156844 0.495956
91 16 2933 2917 128.206 124 1.61382 0.495995
92 16 2964 2948 128.16 124 0.200034 0.496516
93 16 2994 2978 128.073 120 0.74711 0.497577
94 16 3024 3008 127.986 120 0.0832374 0.497393
95 16 3058 3042 128.071 136 0.319327 0.497268
96 16 3089 3073 128.028 124 0.998517 0.497572
97 16 3122 3106 128.069 132 0.36057 0.497108
98 16 3148 3132 127.823 104 1.19455 0.49836
99 16 3180 3164 127.825 128 0.460973 0.498516
2025-09-06T15:40:38.777483+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.498469
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3214 3198 127.906 136 0.190703 0.498469
101 16 3244 3228 127.828 120 0.0891681 0.498488
102 16 3279 3263 127.947 140 1.01252 0.498182
103 16 3309 3293 127.87 120 0.755003 0.497799
104 16 3341 3325 127.871 128 0.329617 0.498093
105 16 3371 3355 127.796 120 0.462132 0.497717
106 16 3405 3389 127.873 136 0.0921742 0.497935
107 16 3433 3417 127.725 112 0.0923818 0.498832
108 16 3462 3446 127.616 116 1.41373 0.498909
109 16 3493 3477 127.583 124 0.280848 0.498269
110 16 3525 3509 127.586 128 0.198916 0.49795
111 16 3554 3538 127.482 116 0.085032 0.498829
112 16 3586 3570 127.486 128 0.198039 0.498902
113 16 3614 3598 127.349 112 0.248161 0.498954
114 16 3650 3634 127.495 144 0.0151987 0.498726
115 16 3678 3662 127.36 112 0.337411 0.499597
116 16 3715 3699 127.538 148 0.101546 0.498792
117 16 3745 3729 127.474 120 0.139778 0.498472
118 16 3776 3760 127.444 124 0.0985602 0.499527
119 16 3807 3791 127.415 124 0.0877263 0.499744
2025-09-06T15:40:58.779686+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.499195
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 3837 3821 127.353 120 0.143885 0.499195
121 16 3869 3853 127.358 128 0.145949 0.499272
122 16 3901 3885 127.364 128 2.06391 0.499918
123 16 3931 3915 127.304 120 0.105985 0.500572
124 16 3970 3954 127.535 156 0.0309444 0.500174
125 16 4005 3989 127.634 140 0.536079 0.499227
126 16 4037 4021 127.637 128 0.56835 0.499251
127 16 4070 4054 127.671 132 0.257916 0.499163
128 16 4106 4090 127.799 144 0.194968 0.498771
129 16 4137 4121 127.769 124 0.281637 0.499064
130 16 4166 4150 127.679 116 0.233604 0.499265
131 16 4197 4181 127.65 124 0.422676 0.499544
132 16 4229 4213 127.653 128 0.136296 0.499425
133 16 4262 4246 127.685 132 0.165794 0.499302
134 16 4297 4281 127.777 140 0.611767 0.499259
135 16 4330 4314 127.808 132 1.05761 0.499475
136 16 4360 4344 127.751 120 0.0177166 0.499404
137 16 4390 4374 127.694 120 0.169101 0.49937
138 16 4425 4409 127.783 140 1.37626 0.498999
139 16 4456 4440 127.756 124 0.0868818 0.498992
2025-09-06T15:41:18.781998+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.4981
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 4488 4472 127.758 128 0.146172 0.4981
141 16 4523 4507 127.844 140 2.37268 0.498882
142 16 4553 4537 127.789 120 0.149856 0.49874
143 16 4584 4568 127.762 124 0.0153841 0.498249
144 16 4614 4598 127.708 120 0.189612 0.498561
145 16 4644 4628 127.655 120 0.087014 0.498361
146 16 4674 4658 127.603 120 0.101273 0.498314
147 16 4704 4688 127.551 120 0.349187 0.499679
148 16 4732 4716 127.446 112 0.932191 0.500301
149 16 4769 4753 127.584 148 0.0166335 0.500164
150 16 4802 4786 127.613 132 0.0146001 0.499685
151 16 4832 4816 127.562 120 2.05424 0.500096
152 16 4864 4848 127.565 128 0.0185872 0.500373
153 16 4891 4875 127.437 108 1.15425 0.499997
154 16 4929 4913 127.597 152 0.597443 0.500299
155 16 4959 4943 127.548 120 0.0184109 0.500234
156 16 4991 4975 127.55 128 0.647753 0.500019
157 16 5022 5006 127.528 124 0.0846472 0.49948
158 16 5054 5038 127.531 128 0.0179906 0.499787
159 16 5084 5068 127.483 120 1.78151 0.500272
2025-09-06T15:41:38.784176+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.50063
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 5112 5096 127.386 112 0.229029 0.50063
161 16 5147 5131 127.464 140 0.205066 0.500724
162 16 5181 5165 127.517 136 0.139766 0.49996
163 16 5214 5198 127.544 132 0.0151517 0.499815
164 16 5245 5229 127.523 124 0.18035 0.499337
165 16 5279 5263 127.574 136 0.212684 0.499407
166 16 5314 5298 127.649 140 0.0900562 0.498887
167 16 5349 5333 127.723 140 0.161326 0.498775
168 16 5385 5369 127.82 144 2.114 0.499046
169 16 5418 5402 127.844 132 1.50679 0.498985
170 16 5450 5434 127.845 128 0.0835925 0.498632
171 16 5481 5465 127.822 124 2.17135 0.498477
172 16 5509 5493 127.73 112 0.461062 0.499395
173 16 5540 5524 127.709 124 0.170932 0.499317
174 16 5570 5554 127.664 120 1.72015 0.499776
175 16 5605 5589 127.735 140 0.016511 0.499746
176 16 5635 5619 127.691 120 0.0188613 0.499901
177 16 5670 5654 127.76 140 0.0262272 0.499686
178 16 5705 5689 127.829 140 0.0185254 0.499313
179 16 5737 5721 127.83 128 0.0142448 0.499583
2025-09-06T15:41:58.786375+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.499317
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 5771 5755 127.875 136 0.0949172 0.499317
181 16 5803 5787 127.876 128 0.0187945 0.49914
182 16 5838 5822 127.942 140 0.403885 0.49917
183 16 5871 5855 127.964 132 0.0830073 0.498966
184 16 5902 5886 127.943 124 0.0830302 0.499013
185 16 5934 5918 127.943 128 0.868037 0.49916
186 16 5965 5949 127.921 124 1.06085 0.499356
187 16 5996 5980 127.9 124 0.0854325 0.499033
188 16 6026 6010 127.858 120 0.206039 0.498886
189 16 6057 6041 127.838 124 0.0976375 0.49875
190 16 6090 6074 127.86 132 0.513917 0.498319
191 16 6122 6106 127.86 128 0.0880232 0.498888
192 16 6154 6138 127.861 128 1.73397 0.499036
193 16 6183 6167 127.8 116 0.267093 0.499223
194 16 6221 6205 127.924 152 1.63174 0.498967
195 16 6254 6238 127.945 132 0.533932 0.498767
196 16 6290 6274 128.027 144 0.101415 0.498503
197 16 6319 6303 127.966 116 0.0139238 0.498474
198 16 6353 6337 128.006 136 1.13811 0.499147
199 16 6383 6367 127.966 120 0.536272 0.499058
2025-09-06T15:42:18.788755+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.498567
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 6414 6398 127.946 124 0.301265 0.498567
201 16 6445 6429 127.926 124 0.568706 0.498866
202 16 6485 6469 128.085 160 0.192172 0.498296
203 16 6517 6501 128.085 128 0.460974 0.498558
204 16 6546 6530 128.025 116 1.01028 0.498661
205 16 6576 6560 127.986 120 0.0855708 0.498712
206 16 6605 6589 127.928 116 0.621335 0.498904
207 16 6643 6627 128.044 152 0.227024 0.498146
208 16 6672 6656 127.986 116 0.119048 0.498013
209 16 6710 6694 128.101 152 0.0153679 0.49794
210 16 6743 6727 128.119 132 0.738219 0.498168
211 16 6772 6756 128.062 116 0.100046 0.498637
212 16 6806 6790 128.099 136 0.754633 0.498487
213 16 6840 6824 128.136 136 0.147949 0.497936
214 16 6873 6857 128.154 132 0.0148063 0.498241
215 16 6905 6889 128.154 128 0.214634 0.498032
216 16 6939 6923 128.19 136 0.226452 0.498139
217 16 6970 6954 128.17 124 1.42773 0.498302
218 16 7002 6986 128.17 128 0.170738 0.497888
219 16 7032 7016 128.132 120 1.66462 0.498185
2025-09-06T15:42:38.790733+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.498067
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 7064 7048 128.132 128 1.31962 0.498067
221 16 7094 7078 128.095 120 0.278874 0.498398
222 16 7123 7107 128.04 116 0.11889 0.498375
223 16 7156 7140 128.058 132 1.75955 0.498411
224 16 7184 7168 127.986 112 0.629593 0.498789
225 16 7219 7203 128.039 140 0.157019 0.498765
226 16 7252 7236 128.057 132 0.163003 0.498666
227 16 7283 7267 128.039 124 0.463427 0.498789
228 16 7318 7302 128.091 140 0.585858 0.498435
229 16 7354 7338 128.161 144 0.242101 0.498385
230 16 7385 7369 128.143 124 0.222028 0.498215
231 16 7414 7398 128.09 116 0.280185 0.498316
232 16 7447 7431 128.107 132 0.343408 0.498583
233 16 7479 7463 128.106 128 1.14125 0.498413
234 16 7506 7490 128.02 108 0.0839596 0.498283
235 16 7537 7521 128.003 124 0.411275 0.498843
236 16 7570 7554 128.02 132 1.71495 0.498826
237 16 7602 7586 128.02 128 0.546866 0.498875
238 16 7633 7617 128.003 124 0.616218 0.498814
239 16 7666 7650 128.02 132 0.808912 0.498968
2025-09-06T15:42:58.792669+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.499115
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 7697 7681 128.003 124 1.01153 0.499115
241 16 7729 7713 128.003 128 0.169586 0.499001
242 16 7763 7747 128.036 136 0.0836757 0.498608
243 16 7797 7781 128.069 136 0.974431 0.49876
244 16 7828 7812 128.052 124 0.0933906 0.498493
245 16 7862 7846 128.084 136 0.0264225 0.498804
246 16 7896 7880 128.116 136 1.35708 0.498611
247 16 7932 7916 128.181 144 0.953986 0.498218
248 16 7961 7945 128.131 116 0.53038 0.498478
249 16 7992 7976 128.115 124 0.0904629 0.498187
250 16 8022 8006 128.082 120 1.44914 0.498691
251 16 8055 8039 128.098 132 0.478966 0.498668
252 16 8083 8067 128.034 112 1.27091 0.498785
253 16 8121 8105 128.129 152 0.771445 0.498566
254 16 8157 8141 128.191 144 0.514055 0.498057
255 16 8195 8179 128.284 152 0.860361 0.497881
256 16 8229 8213 128.314 136 0.0925479 0.497722
257 16 8263 8247 128.344 136 0.0913154 0.497788
258 16 8296 8280 128.358 132 0.0166663 0.497453
259 16 8326 8310 128.326 120 0.740491 0.497972
2025-09-06T15:43:18.794988+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.497889
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 8356 8340 128.294 120 1.3518 0.497889
261 16 8386 8370 128.262 120 0.0823364 0.49791
262 16 8417 8401 128.246 124 1.54553 0.49786
263 16 8450 8434 128.26 132 0.0838941 0.497792
264 16 8482 8466 128.259 128 1.72307 0.497549
265 16 8515 8499 128.273 132 0.356187 0.497651
266 16 8543 8527 128.212 112 0.0153906 0.497953
267 16 8581 8565 128.301 152 0.298913 0.497681
268 16 8620 8604 128.404 156 0.0991551 0.497596
269 16 8651 8635 128.388 124 0.0174738 0.49748
270 16 8684 8668 128.401 132 0.632525 0.497568
271 16 8720 8704 128.458 144 0.0179211 0.497476
272 16 8752 8736 128.457 128 0.355126 0.497472
273 16 8784 8768 128.455 128 0.083615 0.49719
274 16 8816 8800 128.453 128 0.404962 0.497328
275 16 8853 8837 128.524 148 0.0822306 0.497097
276 16 8884 8868 128.508 124 0.137836 0.497125
277 16 8915 8899 128.492 124 0.938899 0.497151
278 16 8946 8930 128.475 124 1.57643 0.497049
279 16 8974 8958 128.416 112 0.201861 0.497013
2025-09-06T15:43:38.797064+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.497314
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 9007 8991 128.429 132 0.267373 0.497314
281 16 9037 9021 128.399 120 0.230352 0.497376
282 16 9075 9059 128.483 152 1.33239 0.497304
283 16 9104 9088 128.438 116 0.119168 0.497561
284 16 9137 9121 128.451 132 0.627274 0.497566
285 16 9167 9151 128.421 120 0.192958 0.497364
286 16 9200 9184 128.434 132 0.158574 0.497227
287 16 9234 9218 128.46 136 0.16397 0.497378
288 16 9264 9248 128.431 120 0.729445 0.497506
289 16 9296 9280 128.429 128 1.05838 0.497642
290 16 9328 9312 128.428 128 0.327132 0.497384
291 16 9361 9345 128.44 132 0.318189 0.497225
292 16 9390 9374 128.397 116 0.219333 0.497388
293 16 9423 9407 128.409 132 0.425543 0.497768
294 16 9451 9435 128.354 112 1.43109 0.497962
295 16 9485 9469 128.38 136 0.189871 0.497842
296 16 9516 9500 128.365 124 0.230003 0.497657
297 16 9544 9528 128.31 112 0.955559 0.498058
298 16 9576 9560 128.308 128 0.0124968 0.497977
299 16 9613 9597 128.374 148 0.0123177 0.497772
2025-09-06T15:43:58.798869+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.497815
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 9647 9631 128.4 136 0.811663 0.497815
301 16 9678 9662 128.385 124 0.0884557 0.49765
302 16 9710 9694 128.384 128 0.0216289 0.497641
303 16 9742 9726 128.382 128 0.174616 0.497752
304 16 9777 9761 128.42 140 0.015451 0.497691
305 16 9809 9793 128.419 128 0.216147 0.49751
306 16 9842 9826 128.431 132 0.101977 0.497521
307 16 9875 9859 128.442 132 0.112421 0.497377
308 16 9906 9890 128.428 124 0.11111 0.49751
309 16 9942 9926 128.478 144 0.649199 0.497281
310 16 9979 9963 128.541 148 1.69193 0.496995
311 16 10009 9993 128.514 120 0.220615 0.497058
312 16 10047 10031 128.589 152 0.197369 0.49661
313 16 10077 10061 128.561 120 0.424528 0.496527
314 16 10113 10097 128.61 144 0.698878 0.496785
315 16 10145 10129 128.609 128 0.409623 0.496865
316 16 10175 10159 128.581 120 0.0907244 0.496909
317 16 10211 10195 128.63 144 0.151109 0.496434
318 16 10245 10229 128.653 136 0.92775 0.496823
319 16 10277 10261 128.651 128 1.18775 0.49678
2025-09-06T15:44:18.801007+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.49664
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 10313 10297 128.699 144 0.577956 0.49664
321 16 10344 10328 128.684 124 0.0136668 0.49665
322 16 10373 10357 128.645 116 0.128139 0.496726
323 16 10407 10391 128.667 136 0.1921 0.496567
324 16 10442 10426 128.702 140 0.0853432 0.496328
325 16 10474 10458 128.7 128 0.227602 0.496532
326 16 10503 10487 128.661 116 0.0931666 0.496592
327 16 10535 10519 128.659 128 1.43914 0.496656
328 16 10565 10549 128.633 120 1.21759 0.496883
329 16 10594 10578 128.594 116 0.0168948 0.496886
330 16 10623 10607 128.556 116 0.127112 0.497045
331 16 10654 10638 128.542 124 0.468855 0.497003
332 16 10688 10672 128.565 136 0.318109 0.496861
333 16 10718 10702 128.539 120 0.0895297 0.4968
334 16 10746 10730 128.489 112 0.0931543 0.496932
335 16 10781 10765 128.524 140 0.102196 0.497246
336 16 10810 10794 128.486 116 0.309746 0.497038
337 16 10840 10824 128.461 120 1.48846 0.497417
338 16 10875 10859 128.495 140 0.0125648 0.497074
339 16 10903 10887 128.447 112 0.24323 0.497
2025-09-06T15:44:38.803036+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.497293
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 10932 10916 128.41 116 0.159828 0.497293
341 16 10969 10953 128.467 148 0.512563 0.497498
342 16 11000 10984 128.454 124 0.430946 0.49749
343 16 11034 11018 128.476 136 0.0190478 0.4975
344 16 11064 11048 128.451 120 1.43754 0.497451
345 16 11096 11080 128.45 128 0.123126 0.497735
346 16 11126 11110 128.426 120 0.0199677 0.497568
347 16 11160 11144 128.447 136 0.248343 0.497401
348 16 11193 11177 128.458 132 1.33479 0.497461
349 16 11225 11209 128.456 128 1.55478 0.497472
350 16 11255 11239 128.432 120 0.402596 0.497454
351 16 11284 11268 128.397 116 0.209659 0.497276
352 16 11327 11311 128.52 172 0.0162636 0.496935
353 16 11357 11341 128.496 120 0.303009 0.496808
354 16 11389 11373 128.495 128 2.09929 0.4972
355 16 11417 11401 128.448 112 0.329918 0.497181
356 16 11450 11434 128.458 132 0.201195 0.497147
357 16 11483 11467 128.468 132 0.012181 0.497155
358 16 11513 11497 128.445 120 0.270215 0.497146
359 16 11544 11528 128.432 124 2.08333 0.49738
2025-09-06T15:44:58.804827+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.497608
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 11574 11558 128.409 120 0.13663 0.497608
361 16 11604 11588 128.385 120 0.143828 0.497511
362 16 11640 11624 128.428 144 1.93239 0.497602
363 16 11673 11657 128.438 132 1.44862 0.497344
364 16 11709 11693 128.481 144 0.45765 0.497235
365 16 11738 11722 128.447 116 0.187503 0.497381
366 16 11769 11753 128.435 124 1.54588 0.497409
367 16 11801 11785 128.433 128 0.14537 0.497391
368 16 11834 11818 128.443 132 0.0837397 0.49768
369 16 11863 11847 128.409 116 1.19477 0.497808
370 16 11896 11880 128.419 132 0.0154446 0.497747
371 16 11926 11910 128.396 120 0.291201 0.497643
372 16 11958 11942 128.395 128 0.543976 0.497705
373 16 11995 11979 128.448 148 0.0158311 0.497739
374 16 12027 12011 128.446 128 1.44982 0.497627
375 16 12057 12041 128.424 120 0.0830153 0.497706
376 16 12088 12072 128.412 124 0.839145 0.497897
377 16 12122 12106 128.432 136 0.0169174 0.497736
378 16 12155 12139 128.441 132 1.1392 0.49759
379 16 12185 12169 128.419 120 0.175248 0.497634
2025-09-06T15:45:18.806935+0000 min lat: 0.0112911 max lat: 2.5627 avg lat: 0.49747
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 12221 12205 128.46 144 0.133709 0.49747
381 16 12254 12238 128.469 132 0.734467 0.497619
382 16 12285 12269 128.458 124 0.0242917 0.497503
383 16 12317 12301 128.456 128 0.148004 0.497484
384 16 12351 12335 128.476 136 0.372017 0.49757
385 16 12385 12369 128.496 136 0.0842423 0.497467
386 16 12419 12403 128.515 136 0.610525 0.49743
387 16 12450 12434 128.503 124 0.622395 0.497484
388 16 12480 12464 128.481 120 0.16895 0.497589
389 16 12515 12499 128.511 140 0.107047 0.497332
390 16 12556 12540 128.602 164 0.0124197 0.497078
391 16 12592 12576 128.641 144 0.33531 0.496892
392 16 12621 12605 128.609 116 0.0831915 0.496986
393 16 12652 12636 128.597 124 0.231673 0.497114
394 16 12683 12667 128.585 124 1.42598 0.496965
395 16 12713 12697 128.564 120 0.185326 0.4969
396 16 12744 12728 128.552 124 0.0329641 0.496685
397 16 12778 12762 128.571 136 0.0830932 0.496686
398 16 12810 12794 128.569 128 0.797652 0.49691
399 16 12840 12824 128.548 120 2.05504 0.497196
2025-09-06T15:45:38.809001+0000 min lat: 0.0112911 max lat: 2.90998 avg lat: 0.497298
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 12871 12855 128.536 124 0.6905 0.497298
401 16 12902 12886 128.525 124 0.254844 0.497479
402 16 12933 12917 128.514 124 0.205747 0.497421
403 16 12966 12950 128.522 132 0.370429 0.497308
404 16 12995 12979 128.491 116 0.677479 0.497547
405 16 13026 13010 128.48 124 0.0870828 0.49738
406 16 13058 13042 128.479 128 0.95667 0.497583
407 16 13096 13080 128.537 152 1.6651 0.4974
408 16 13135 13119 128.604 156 1.11176 0.497185
409 16 13164 13148 128.573 116 0.177138 0.49717
410 16 13195 13179 128.562 124 0.399098 0.497271
411 16 13222 13206 128.512 108 0.0220697 0.497366
412 16 13265 13249 128.618 172 0.0120533 0.497085
413 16 13297 13281 128.616 128 1.09701 0.49718
414 16 13329 13313 128.614 128 1.28757 0.497078
415 16 13360 13344 128.603 124 0.567254 0.497114
416 16 13391 13375 128.592 124 0.229586 0.497204
417 16 13423 13407 128.591 128 0.115248 0.496987
418 16 13459 13443 128.628 144 0.192905 0.497026
419 16 13492 13476 128.636 132 1.24071 0.496936
2025-09-06T15:45:58.811246+0000 min lat: 0.0112911 max lat: 2.90998 avg lat: 0.497053
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 13519 13503 128.586 108 1.47235 0.497053
421 16 13550 13534 128.575 124 0.212801 0.4972
422 16 13579 13563 128.546 116 1.08266 0.497322
423 16 13611 13595 128.544 128 1.52997 0.497483
424 16 13641 13625 128.524 120 0.150946 0.497302
425 16 13676 13660 128.551 140 0.596906 0.497405
426 16 13711 13695 128.578 140 0.2281 0.497193
427 16 13742 13726 128.567 124 0.0839347 0.497224
428 16 13775 13759 128.575 132 0.201946 0.497209
429 16 13810 13794 128.602 140 0.32731 0.497051
430 16 13841 13825 128.591 124 0.949474 0.497035
431 16 13873 13857 128.59 128 0.279448 0.497047
432 16 13907 13891 128.607 136 0.789029 0.497195
433 16 13939 13923 128.605 128 0.88506 0.497013
434 16 13976 13960 128.65 148 0.221014 0.496969
435 16 14012 13996 128.685 144 0.0118398 0.49664
436 16 14043 14027 128.674 124 0.188171 0.496684
437 16 14074 14058 128.664 124 0.412753 0.49664
438 16 14109 14093 128.69 140 0.0876013 0.496772
439 16 14137 14121 128.651 112 0.0227811 0.496833
2025-09-06T15:46:18.813698+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496803
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 14171 14155 128.668 136 0.629819 0.496803
441 16 14201 14185 128.648 120 0.0876253 0.496817
442 16 14233 14217 128.647 128 0.522567 0.497065
443 16 14266 14250 128.654 132 0.0923337 0.496944
444 16 14299 14283 128.662 132 0.239631 0.496894
445 16 14333 14317 128.678 136 0.0167242 0.496877
446 16 14366 14350 128.686 132 0.0132008 0.496753
447 16 14400 14384 128.702 136 0.516377 0.496719
448 16 14433 14417 128.71 132 0.0906252 0.49649
449 16 14466 14450 128.717 132 0.0875175 0.496337
450 16 14501 14485 128.742 140 1.70374 0.496583
451 16 14533 14517 128.74 128 0.13055 0.496465
452 16 14564 14548 128.73 124 0.421772 0.496476
453 16 14597 14581 128.737 132 0.480648 0.496586
454 16 14631 14615 128.753 136 0.728483 0.496505
455 16 14660 14644 128.725 116 0.0258437 0.496583
456 16 14699 14683 128.785 156 1.38215 0.496443
457 16 14726 14710 128.739 108 0.225671 0.496422
458 16 14755 14739 128.711 116 0.935365 0.496658
459 16 14787 14771 128.71 128 0.185154 0.496743
2025-09-06T15:46:38.815971+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496528
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 14818 14802 128.699 124 0.629326 0.496528
461 16 14853 14837 128.724 140 0.382282 0.496571
462 16 14884 14868 128.714 124 0.236527 0.496563
463 16 14914 14898 128.695 120 0.416287 0.496686
464 16 14946 14930 128.693 128 0.159529 0.496667
465 16 14975 14959 128.666 116 1.65493 0.496633
466 16 15005 14989 128.647 120 0.39065 0.496549
467 16 15043 15027 128.697 152 0.318626 0.496765
468 16 15072 15056 128.67 116 0.27181 0.496826
469 16 15104 15088 128.669 128 0.0873646 0.496884
470 16 15135 15119 128.659 124 1.11515 0.496908
471 16 15165 15149 128.64 120 0.0867459 0.496877
472 16 15196 15180 128.63 124 0.20517 0.497097
473 16 15228 15212 128.629 128 0.196976 0.497028
474 16 15260 15244 128.628 128 0.155721 0.496837
475 16 15288 15272 128.593 112 0.11416 0.496959
476 16 15320 15304 128.591 128 0.0880278 0.49698
477 16 15351 15335 128.582 124 0.599031 0.497028
478 16 15382 15366 128.572 124 0.165262 0.49717
479 16 15412 15396 128.554 120 0.0910894 0.497152
2025-09-06T15:46:58.818123+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497084
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 15444 15428 128.553 128 0.177656 0.497084
481 16 15481 15465 128.593 148 0.194738 0.497015
482 16 15510 15494 128.567 116 1.85793 0.49712
483 16 15543 15527 128.574 132 0.324268 0.496883
484 16 15579 15563 128.606 144 0.220855 0.496859
485 16 15608 15592 128.58 116 0.157102 0.496885
486 16 15641 15625 128.587 132 1.02381 0.497195
487 16 15672 15656 128.578 124 0.0873863 0.497203
488 16 15702 15686 128.56 120 0.221675 0.497358
489 16 15737 15721 128.583 140 1.19506 0.497225
490 16 15768 15752 128.574 124 1.58886 0.497159
491 16 15807 15791 128.63 156 0.120142 0.496906
492 16 15838 15822 128.62 124 0.208816 0.496872
493 16 15874 15858 128.651 144 1.70134 0.496848
494 16 15905 15889 128.642 124 0.013919 0.49679
495 16 15938 15922 128.649 132 0.107668 0.496578
496 16 15969 15953 128.639 124 0.20202 0.496691
497 16 16000 15984 128.63 124 2.19179 0.496876
498 16 16030 16014 128.613 120 0.0935627 0.4969
499 16 16063 16047 128.619 132 0.586365 0.496987
2025-09-06T15:47:18.820825+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497189
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 16093 16077 128.602 120 0.928535 0.497189
501 16 16125 16109 128.601 128 0.587212 0.497124
502 16 16157 16141 128.6 128 0.0149436 0.496984
503 16 16188 16172 128.59 124 0.525985 0.497092
504 16 16223 16207 128.613 140 0.177496 0.49698
505 16 16256 16240 128.62 132 1.49771 0.497001
506 16 16284 16268 128.587 112 0.204646 0.497055
507 16 16323 16307 128.641 156 0.162624 0.497101
508 16 16353 16337 128.624 120 1.14793 0.497145
509 16 16381 16365 128.591 112 0.258723 0.497014
510 16 16415 16399 128.606 136 0.0191178 0.49716
511 16 16450 16434 128.628 140 0.0163753 0.497114
512 16 16480 16464 128.611 120 1.45301 0.496958
513 16 16513 16497 128.618 132 0.299722 0.4968
514 16 16545 16529 128.616 128 0.470141 0.497005
515 16 16575 16559 128.6 120 0.0836169 0.497205
516 16 16604 16588 128.575 116 0.795945 0.497203
517 16 16636 16620 128.574 128 0.152648 0.497248
518 16 16668 16652 128.573 128 1.7621 0.497177
519 16 16701 16685 128.58 132 0.0286827 0.497207
2025-09-06T15:47:38.823024+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497116
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 16733 16717 128.578 128 0.121051 0.497116
521 16 16764 16748 128.57 124 1.92792 0.497185
522 16 16796 16780 128.568 128 0.223131 0.497065
523 16 16829 16813 128.575 132 2.03957 0.497055
524 16 16860 16844 128.566 124 0.368705 0.497162
525 16 16892 16876 128.565 128 0.0291149 0.497224
526 16 16927 16911 128.587 140 0.289156 0.497117
527 16 16956 16940 128.563 116 0.154645 0.497165
528 16 16987 16971 128.554 124 0.0191086 0.497211
529 16 17018 17002 128.546 124 0.016989 0.497038
530 16 17052 17036 128.56 136 0.0873115 0.496995
531 16 17085 17069 128.566 132 0.261964 0.497159
532 16 17115 17099 128.55 120 0.271947 0.497064
533 16 17148 17132 128.556 132 0.0223805 0.497098
534 16 17179 17163 128.548 124 2.03793 0.497349
535 16 17209 17193 128.532 120 0.380499 0.497416
536 16 17241 17225 128.531 128 0.395876 0.497297
537 16 17273 17257 128.53 128 1.56496 0.497343
538 16 17310 17294 128.566 148 0.0836351 0.49699
539 16 17339 17323 128.543 116 0.261275 0.497154
2025-09-06T15:47:58.825300+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497092
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 17374 17358 128.564 140 0.158989 0.497092
541 16 17405 17389 128.555 124 0.215266 0.497162
542 16 17438 17422 128.562 132 0.352414 0.49733
543 16 17472 17456 128.575 136 1.7696 0.497264
544 16 17506 17490 128.589 136 0.0252598 0.497077
545 16 17538 17522 128.588 128 0.0955861 0.497009
546 16 17573 17557 128.609 140 1.66585 0.497175
547 16 17607 17591 128.622 136 0.189143 0.496961
548 16 17638 17622 128.614 124 0.108806 0.496882
549 16 17671 17655 128.62 132 2.31094 0.496916
550 16 17701 17685 128.604 120 0.0906144 0.497151
551 16 17728 17712 128.567 108 0.0891716 0.49709
552 16 17761 17745 128.573 132 0.093646 0.497087
553 16 17789 17773 128.543 112 0.143058 0.497338
554 16 17823 17807 128.557 136 0.603812 0.497307
555 16 17853 17837 128.541 120 0.109883 0.497243
556 16 17884 17868 128.533 124 0.0156527 0.497444
557 16 17913 17897 128.51 116 0.0838089 0.497448
558 16 17948 17932 128.531 140 1.5958 0.497477
559 16 17978 17962 128.516 120 0.202312 0.497409
2025-09-06T15:48:18.827278+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497323
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 18014 17998 128.543 144 0.611702 0.497323
561 16 18045 18029 128.535 124 0.0833382 0.497345
562 16 18079 18063 128.548 136 0.156749 0.497305
563 16 18114 18098 128.569 140 0.0152491 0.4973
564 16 18143 18127 128.546 116 1.95806 0.497362
565 16 18176 18160 128.553 132 0.379853 0.497464
566 16 18209 18193 128.559 132 0.0878705 0.497461
567 16 18240 18224 128.551 124 0.083344 0.497533
568 16 18270 18254 128.535 120 0.432692 0.497496
569 16 18299 18283 128.513 116 0.0146442 0.497573
570 16 18332 18316 128.52 132 1.27941 0.497523
571 16 18366 18350 128.533 136 0.421052 0.49734
572 16 18394 18378 128.504 112 0.20878 0.497499
573 16 18422 18406 128.475 112 0.0953588 0.497533
574 16 18454 18438 128.474 128 1.88759 0.497579
575 16 18486 18470 128.473 128 0.0906426 0.49748
576 16 18525 18509 128.521 156 0.0117905 0.497391
577 16 18554 18538 128.499 116 0.026821 0.497546
578 16 18585 18569 128.491 124 0.55262 0.497686
579 16 18625 18609 128.546 160 1.18513 0.497467
2025-09-06T15:48:38.829200+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497211
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 18660 18644 128.566 140 0.0887577 0.497211
581 16 18694 18678 128.578 136 1.86063 0.497239
582 16 18726 18710 128.577 128 0.0204899 0.497217
583 16 18754 18738 128.549 112 0.0166127 0.497212
584 16 18786 18770 128.548 128 0.0827811 0.497199
585 16 18818 18802 128.547 128 0.129177 0.497309
586 16 18852 18836 128.56 136 2.06812 0.497305
587 16 18881 18865 128.538 116 0.192596 0.49741
588 16 18916 18900 128.558 140 0.021311 0.497254
589 16 18952 18936 128.584 144 0.0842292 0.497262
590 16 18982 18966 128.569 120 0.203662 0.497172
591 16 19013 18997 128.561 124 2.305 0.497252
592 16 19046 19030 128.567 132 1.88673 0.497218
593 16 19084 19068 128.607 152 0.355858 0.496991
594 16 19114 19098 128.592 120 0.191087 0.497129
595 16 19147 19131 128.598 132 0.260669 0.497217
596 16 19180 19164 128.604 132 0.911384 0.497343
597 16 19212 19196 128.603 128 0.0959994 0.497227
598 16 19243 19227 128.595 124 0.112064 0.497089
599 16 19278 19262 128.614 140 0.285722 0.497112
2025-09-06T15:48:58.831279+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497174
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 16 19310 19294 128.613 128 0.564131 0.497174
601 16 19341 19325 128.605 124 1.99262 0.497139
602 16 19374 19358 128.611 132 0.493302 0.49707
603 16 19411 19395 128.643 148 0.553165 0.496956
604 16 19442 19426 128.635 124 0.15799 0.497093
605 16 19470 19454 128.608 112 0.706058 0.497189
606 16 19504 19488 128.62 136 0.0877895 0.497176
607 16 19534 19518 128.606 120 1.39541 0.49723
608 16 19566 19550 128.605 128 0.780547 0.497219
609 16 19598 19582 128.604 128 0.958955 0.497244
610 16 19627 19611 128.583 116 0.899976 0.497336
611 16 19663 19647 128.608 144 1.23997 0.497237
612 16 19698 19682 128.627 140 0.0131782 0.497181
613 16 19733 19717 128.645 140 0.15652 0.497012
614 16 19764 19748 128.638 124 0.819587 0.497135
615 16 19795 19779 128.63 124 0.529949 0.49701
616 16 19826 19810 128.623 124 0.0120411 0.497145
617 16 19858 19842 128.621 128 1.73184 0.497184
618 16 19890 19874 128.62 128 0.702031 0.497218
619 16 19923 19907 128.626 132 1.12445 0.497201
2025-09-06T15:49:18.833681+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.49716
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
620 16 19957 19941 128.638 136 0.15215 0.49716
621 16 19991 19975 128.65 136 1.10722 0.497151
622 16 20023 20007 128.649 128 0.0845331 0.496995
623 16 20052 20036 128.628 116 0.0900372 0.496987
624 16 20087 20071 128.646 140 0.0856471 0.496966
625 16 20119 20103 128.645 128 1.80381 0.497067
626 16 20151 20135 128.644 128 0.101881 0.49704
627 16 20186 20170 128.662 140 0.149242 0.496841
628 16 20222 20206 128.687 144 0.366895 0.496914
629 16 20252 20236 128.673 120 1.78885 0.497
630 16 20284 20268 128.672 128 0.350274 0.497013
631 16 20316 20300 128.671 128 0.227148 0.497012
632 16 20348 20332 128.67 128 1.06538 0.49703
633 16 20376 20360 128.643 112 0.20394 0.497043
634 16 20409 20393 128.649 132 0.0166973 0.497111
635 16 20443 20427 128.66 136 0.219014 0.497036
636 16 20475 20459 128.659 128 0.180861 0.496958
637 16 20508 20492 128.664 132 0.227043 0.49701
638 16 20542 20526 128.676 136 0.268174 0.496931
639 16 20578 20562 128.7 144 0.293443 0.496817
2025-09-06T15:49:38.835859+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496911
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
640 16 20607 20591 128.68 116 0.0308784 0.496911
641 16 20640 20624 128.685 132 0.125695 0.496928
642 16 20676 20660 128.709 144 0.402102 0.496851
643 16 20709 20693 128.714 132 0.121779 0.496815
644 16 20742 20726 128.719 132 0.298877 0.496934
645 16 20771 20755 128.699 116 1.01643 0.496949
646 16 20802 20786 128.692 124 1.59932 0.496923
647 16 20834 20818 128.691 128 0.228951 0.496876
648 16 20864 20848 128.677 120 0.127424 0.496905
649 16 20898 20882 128.689 136 0.012764 0.496891
650 16 20931 20915 128.694 132 0.555876 0.497001
651 16 20962 20946 128.687 124 0.501452 0.497003
652 16 20994 20978 128.686 128 0.357937 0.496874
653 16 21027 21011 128.691 132 0.712629 0.496922
654 16 21059 21043 128.689 128 0.0919336 0.496951
655 16 21089 21073 128.676 120 0.227276 0.496975
656 16 21117 21101 128.651 112 0.163457 0.497062
657 16 21149 21133 128.65 128 0.452582 0.497044
658 16 21175 21159 128.612 104 0.345423 0.497064
659 16 21207 21191 128.611 128 0.359749 0.497043
2025-09-06T15:49:58.838143+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496999
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
660 16 21243 21227 128.635 144 0.0119636 0.496999
661 16 21281 21265 128.67 152 0.0834842 0.496886
662 16 21313 21297 128.669 128 0.0859965 0.496878
663 16 21347 21331 128.68 136 0.0868878 0.496864
664 16 21382 21366 128.697 140 0.137291 0.496798
665 16 21414 21398 128.696 128 0.0899432 0.49699
666 16 21444 21428 128.683 120 0.158628 0.496937
667 16 21476 21460 128.682 128 0.0854752 0.496916
668 16 21510 21494 128.693 136 1.10989 0.496973
669 16 21543 21527 128.698 132 0.973859 0.496867
670 16 21579 21563 128.72 144 1.01193 0.496886
671 16 21609 21593 128.707 120 0.199175 0.4969
672 16 21642 21626 128.712 132 0.455941 0.496838
673 16 21678 21662 128.735 144 0.319036 0.496787
674 16 21711 21695 128.74 132 0.35557 0.496824
675 16 21742 21726 128.733 124 1.4648 0.496785
676 16 21781 21765 128.773 156 0.086695 0.496519
677 16 21813 21797 128.772 128 0.0984366 0.496508
678 16 21847 21831 128.783 136 1.52147 0.496547
679 16 21875 21859 128.758 112 0.123523 0.496519
2025-09-06T15:50:18.840588+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496447
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
680 16 21906 21890 128.751 124 0.388716 0.496447
681 16 21945 21929 128.791 156 0.462731 0.496491
682 16 21977 21961 128.79 128 0.193746 0.496538
683 16 22011 21995 128.8 136 0.244371 0.496526
684 16 22041 22025 128.787 120 0.111875 0.496502
685 16 22076 22060 128.804 140 0.16684 0.49648
686 16 22106 22090 128.791 120 0.890448 0.49659
687 16 22141 22125 128.807 140 0.191691 0.496505
688 16 22172 22156 128.8 124 0.366525 0.496479
689 16 22206 22190 128.81 136 1.53857 0.496555
690 16 22237 22221 128.803 124 0.669494 0.496476
691 16 22269 22253 128.802 128 0.0871688 0.496537
692 16 22303 22287 128.813 136 0.236077 0.496521
693 16 22331 22315 128.788 112 1.42682 0.496554
694 16 22362 22346 128.781 124 1.11015 0.496622
695 16 22398 22382 128.803 144 0.384562 0.496515
696 16 22431 22415 128.808 132 0.0832375 0.496519
697 16 22464 22448 128.812 132 0.146332 0.496472
698 16 22494 22478 128.8 120 1.40816 0.496536
699 16 22527 22511 128.804 132 0.0936015 0.496516
2025-09-06T15:50:38.842907+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496485
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
700 16 22561 22545 128.815 136 0.174489 0.496485
701 16 22590 22574 128.796 116 1.4611 0.496474
702 16 22621 22605 128.789 124 0.0871898 0.496475
703 16 22652 22636 128.783 124 0.0838748 0.496395
704 16 22679 22663 128.753 108 0.535887 0.496632
705 16 22712 22696 128.758 132 0.087955 0.49666
706 16 22747 22731 128.774 140 0.0890702 0.496638
707 15 22779 22764 128.778 132 0.0875567 0.496649
708 16 22813 22797 128.783 132 1.37929 0.496529
709 16 22846 22830 128.787 132 0.0217707 0.496489
710 16 22877 22861 128.78 124 0.420075 0.496555
711 16 22908 22892 128.774 124 0.0914168 0.49651
712 16 22940 22924 128.773 128 0.194288 0.496498
713 16 22976 22960 128.794 144 0.0834247 0.496492
714 16 23012 22996 128.815 144 0.0167785 0.496477
715 16 23047 23031 128.831 140 1.31584 0.496381
716 16 23082 23066 128.846 140 1.26216 0.496318
717 16 23112 23096 128.834 120 0.128126 0.496309
718 16 23142 23126 128.822 120 0.495135 0.496318
719 16 23173 23157 128.815 124 0.295205 0.496372
2025-09-06T15:50:58.845264+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496362
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
720 16 23205 23189 128.814 128 0.0133885 0.496362
721 16 23239 23223 128.824 136 0.78904 0.496277
722 16 23268 23252 128.806 116 0.667653 0.496527
723 16 23301 23285 128.81 132 0.15411 0.496543
724 16 23334 23318 128.815 132 0.250982 0.496471
725 16 23365 23349 128.808 124 0.521842 0.496339
726 16 23402 23386 128.834 148 0.0838884 0.496444
727 16 23430 23414 128.811 112 0.0912053 0.49645
728 16 23465 23449 128.827 140 0.0160954 0.496404
729 16 23499 23483 128.836 136 0.158878 0.496505
730 16 23530 23514 128.83 124 0.0175636 0.496409
731 16 23563 23547 128.834 132 0.234138 0.496404
732 16 23594 23578 128.827 124 1.15021 0.496508
733 16 23627 23611 128.832 132 0.417956 0.496433
734 16 23655 23639 128.809 112 0.190337 0.496495
735 16 23685 23669 128.797 120 0.550822 0.496505
736 16 23719 23703 128.807 136 0.561391 0.4965
737 16 23754 23738 128.822 140 0.209948 0.496347
738 16 23787 23771 128.826 132 0.0179907 0.496361
739 16 23817 23801 128.814 120 0.0881252 0.496386
2025-09-06T15:51:18.847793+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496551
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
740 16 23847 23831 128.802 120 0.124325 0.496551
741 16 23882 23866 128.817 140 0.018145 0.496493
742 16 23914 23898 128.816 128 0.0880226 0.496466
743 16 23950 23934 128.837 144 0.0881598 0.496383
744 16 23983 23967 128.841 132 0.186385 0.496365
745 16 24014 23998 128.834 124 0.0176771 0.496384
746 16 24050 24034 128.855 144 0.0833495 0.496319
747 16 24083 24067 128.859 132 1.48442 0.496403
748 16 24115 24099 128.858 128 1.43534 0.496401
749 16 24149 24133 128.867 136 0.262983 0.496309
750 16 24180 24164 128.861 124 0.0874134 0.496286
751 16 24211 24195 128.854 124 0.205091 0.496339
752 16 24241 24225 128.842 120 0.246256 0.496232
753 16 24271 24255 128.831 120 2.21482 0.496318
754 16 24303 24287 128.829 128 0.119742 0.496375
755 16 24333 24317 128.818 120 0.0904597 0.496338
756 16 24366 24350 128.822 132 0.42795 0.496291
757 16 24399 24383 128.826 132 2.59693 0.496464
758 16 24430 24414 128.82 124 0.0178529 0.49651
759 16 24463 24447 128.824 132 0.0846903 0.496464
2025-09-06T15:51:38.850062+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496402
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
760 16 24493 24477 128.812 120 0.509347 0.496402
761 16 24524 24508 128.806 124 0.357603 0.496541
762 16 24552 24536 128.784 112 1.64092 0.49657
763 16 24584 24568 128.783 128 0.435011 0.496585
764 16 24614 24598 128.771 120 0.916258 0.496624
765 16 24645 24629 128.765 124 0.0132452 0.496643
766 16 24679 24663 128.774 136 0.086325 0.496644
767 16 24709 24693 128.763 120 0.182785 0.496711
768 16 24736 24720 128.736 108 1.30047 0.496805
769 16 24768 24752 128.735 128 0.16885 0.496846
770 16 24799 24783 128.729 124 0.0908904 0.496743
771 16 24831 24815 128.728 128 0.513463 0.496755
772 16 24864 24848 128.732 132 1.78014 0.496769
773 16 24897 24881 128.736 132 0.135393 0.496665
774 16 24928 24912 128.73 124 0.141492 0.496685
775 16 24959 24943 128.724 124 0.0250499 0.496823
776 16 24991 24975 128.723 128 0.0119032 0.496847
777 16 25020 25004 128.707 116 0.0857588 0.496821
778 16 25054 25038 128.716 136 0.0164854 0.49677
779 16 25085 25069 128.71 124 0.388014 0.49686
2025-09-06T15:51:58.852300+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496877
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
780 16 25118 25102 128.714 132 0.111373 0.496877
781 16 25147 25131 128.698 116 0.396779 0.496892
782 16 25180 25164 128.702 132 0.133056 0.496941
783 16 25211 25195 128.696 124 1.01611 0.497033
784 16 25242 25226 128.69 124 0.101869 0.496959
785 16 25272 25256 128.679 120 0.0874754 0.496972
786 16 25302 25286 128.668 120 0.723337 0.49711
787 16 25331 25315 128.652 116 0.0168556 0.497178
788 16 25357 25341 128.62 104 0.886662 0.497275
789 16 25390 25374 128.625 132 1.2442 0.497306
790 16 25423 25407 128.629 132 1.26531 0.497244
791 16 25454 25438 128.623 124 0.149689 0.497251
792 16 25487 25471 128.627 132 0.873668 0.497355
793 16 25518 25502 128.621 124 0.430606 0.49723
794 16 25551 25535 128.626 132 0.528231 0.497298
795 16 25580 25564 128.61 116 0.137408 0.49729
796 16 25614 25598 128.619 136 0.559987 0.497242
797 16 25646 25630 128.618 128 0.581304 0.497287
798 16 25677 25661 128.612 124 0.43385 0.497208
799 16 25709 25693 128.612 128 1.33966 0.497326
2025-09-06T15:52:18.854533+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497211
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
800 16 25744 25728 128.626 140 1.43384 0.497211
801 16 25776 25760 128.625 128 0.172351 0.497136
802 16 25808 25792 128.624 128 0.977078 0.497281
803 16 25838 25822 128.614 120 1.72357 0.49721
804 16 25873 25857 128.628 140 0.269135 0.497191
805 16 25902 25886 128.612 116 0.0883461 0.497236
806 16 25938 25922 128.631 144 0.0169808 0.497166
807 16 25974 25958 128.65 144 1.75482 0.497093
808 16 26002 25986 128.629 112 0.408635 0.49712
809 16 26037 26021 128.644 140 0.0162427 0.497082
810 16 26066 26050 128.628 116 0.267328 0.497039
811 16 26103 26087 128.652 148 0.268523 0.497128
812 16 26132 26116 128.636 116 0.0926032 0.49717
813 16 26160 26144 128.616 112 0.703411 0.497313
814 16 26194 26178 128.625 136 0.698206 0.497333
815 16 26227 26211 128.629 132 0.197742 0.497272
816 16 26258 26242 128.623 124 0.135187 0.497255
817 16 26289 26273 128.618 124 0.1126 0.497193
818 16 26322 26306 128.622 132 0.762397 0.497249
819 16 26363 26347 128.665 164 0.0171378 0.497178
2025-09-06T15:52:38.856545+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497187
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
820 16 26395 26379 128.664 128 0.599347 0.497187
821 16 26428 26412 128.668 132 0.6571 0.497151
822 16 26463 26447 128.682 140 1.16671 0.497104
823 16 26498 26482 128.696 140 0.338615 0.496984
824 16 26530 26514 128.695 128 0.717878 0.497026
825 16 26564 26548 128.704 136 0.0133314 0.497018
826 16 26597 26581 128.707 132 0.109907 0.496872
827 16 26630 26614 128.711 132 0.961285 0.496981
828 16 26660 26644 128.701 120 0.965745 0.497026
829 16 26690 26674 128.69 120 0.0881294 0.497023
830 16 26722 26706 128.69 128 1.18792 0.497033
831 16 26753 26737 128.684 124 0.0835127 0.496988
832 16 26788 26772 128.697 140 1.25384 0.496967
833 16 26815 26799 128.673 108 0.131317 0.497008
834 16 26847 26831 128.672 128 1.285 0.497141
835 16 26882 26866 128.685 140 0.103099 0.496908
836 16 26914 26898 128.685 128 0.599605 0.496912
837 16 26946 26930 128.684 128 0.219788 0.497017
838 16 26979 26963 128.688 132 0.280519 0.496895
839 16 27009 26993 128.677 120 0.0848683 0.49695
2025-09-06T15:52:58.858510+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.496968
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
840 16 27045 27029 128.695 144 0.227195 0.496968
841 16 27077 27061 128.695 128 0.338462 0.496926
842 16 27113 27097 128.713 144 1.83516 0.496881
843 16 27144 27128 128.707 124 0.0830275 0.496864
844 16 27175 27159 128.702 124 0.0138857 0.496822
845 16 27210 27194 128.715 140 0.184648 0.496845
846 16 27242 27226 128.714 128 0.105456 0.49683
847 16 27272 27256 128.704 120 0.124289 0.496928
848 16 27304 27288 128.703 128 0.2104 0.496874
849 16 27337 27321 128.707 132 0.330146 0.496863
850 16 27373 27357 128.725 144 1.21452 0.496957
851 16 27405 27389 128.724 128 0.0160896 0.49694
852 16 27438 27422 128.728 132 0.0870598 0.496865
853 16 27468 27452 128.717 120 1.13649 0.496912
854 16 27502 27486 128.726 136 1.30123 0.496931
855 16 27536 27520 128.734 136 0.621463 0.496879
856 16 27565 27549 128.72 116 0.648204 0.496945
857 16 27592 27576 128.695 108 1.12094 0.496994
858 16 27622 27606 128.685 120 0.0867658 0.496989
859 16 27656 27640 128.694 136 0.0176688 0.496998
2025-09-06T15:53:18.860908+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497107
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
860 16 27686 27670 128.684 120 0.87022 0.497107
861 16 27716 27700 128.674 120 0.202417 0.497041
862 16 27749 27733 128.677 132 0.104684 0.496995
863 16 27778 27762 128.663 116 0.231146 0.497071
864 16 27810 27794 128.662 128 0.0961636 0.497177
865 16 27839 27823 128.647 116 0.957915 0.497136
866 16 27872 27856 128.651 132 0.621128 0.497192
867 16 27905 27889 128.655 132 0.214006 0.49715
868 16 27936 27920 128.65 124 0.818133 0.497258
869 16 27967 27951 128.644 124 0.246626 0.497097
870 16 28001 27985 128.653 136 0.0870381 0.497146
871 16 28033 28017 128.652 128 0.431196 0.497206
872 16 28065 28049 128.651 128 0.115195 0.497104
873 16 28096 28080 128.646 124 1.56901 0.497268
874 16 28131 28115 128.659 140 0.610961 0.497198
875 16 28160 28144 128.644 116 0.646003 0.497296
876 16 28191 28175 128.639 124 0.0868408 0.497228
877 16 28221 28205 128.629 120 0.0193528 0.497262
878 16 28251 28235 128.619 120 0.0830888 0.497271
879 16 28287 28271 128.637 144 0.0879058 0.497193
2025-09-06T15:53:38.863231+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497356
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
880 16 28314 28298 128.613 108 1.79126 0.497356
881 16 28343 28327 128.599 116 0.017847 0.497383
882 16 28374 28358 128.594 124 0.0885008 0.497391
883 16 28405 28389 128.588 124 0.236418 0.497454
884 16 28441 28425 128.606 144 0.0180073 0.497363
885 16 28475 28459 128.614 136 0.18639 0.497292
886 16 28510 28494 128.627 140 0.340808 0.497261
887 16 28536 28520 128.599 104 0.0854572 0.497357
888 16 28566 28550 128.59 120 0.0878152 0.497391
889 16 28595 28579 128.575 116 0.136222 0.497443
890 16 28629 28613 128.584 136 0.232702 0.49743
891 16 28664 28648 128.596 140 1.58094 0.497404
892 16 28694 28678 128.587 120 0.0206609 0.497411
893 16 28727 28711 128.591 132 0.0944843 0.497407
894 16 28759 28743 128.59 128 0.0942855 0.497424
895 16 28792 28776 128.594 132 0.172671 0.49738
896 16 28821 28805 128.58 116 0.0936381 0.49748
897 16 28853 28837 128.579 128 1.50884 0.497508
898 16 28886 28870 128.583 132 0.0136993 0.497517
899 16 28918 28902 128.582 128 0.0891547 0.497503
2025-09-06T15:53:58.865532+0000 min lat: 0.0112637 max lat: 2.90998 avg lat: 0.497532
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
900 13 28947 28934 128.581 128 0.804857 0.497532
Total time run: 900.508
Total writes made: 28947
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 128.581
Stddev Bandwidth: 10.3096
Max bandwidth (MB/sec): 172
Min bandwidth (MB/sec): 104
Average IOPS: 32
Stddev IOPS: 2.5774
Max IOPS: 43
Min IOPS: 26
Average Latency(s): 0.497607
Stddev Latency(s): 0.560887
Max latency(s): 2.90998
Min latency(s): 0.0112637
[ceph: root@ms-01-01 /]#
Rados, sequential read test, pool with 32 PGs
[ceph: root@ms-01-01 /]# date
Sat Sep 6 18:02:39 UTC 2025
[ceph: root@ms-01-01 /]# rados bench -p testbench 900 seq
hints = 1
Click to see full run
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 1 1 0 0 0 - 0
1 16 51 35 139.971 140 0.00576719 0.279735
2 16 81 65 129.978 120 0.615285 0.400994
3 16 111 95 126.646 120 0.262298 0.440092
4 16 147 131 130.98 144 1.06346 0.449015
5 16 176 160 127.981 116 1.16096 0.45495
6 16 209 193 128.648 132 0.00941819 0.460813
7 16 241 225 128.553 128 0.278738 0.471168
8 16 272 256 127.982 124 0.585682 0.473817
9 16 305 289 128.423 132 0.397055 0.475108
10 16 336 320 127.979 124 0.0100224 0.479312
11 16 368 352 127.979 128 0.349567 0.479956
12 16 398 382 127.313 120 0.233218 0.482224
13 16 427 411 126.442 116 0.141797 0.484991
14 16 461 445 127.123 136 0.0940371 0.486826
15 16 494 478 127.447 132 0.984991 0.487235
16 16 525 509 127.231 124 0.909859 0.488067
17 16 563 547 128.686 152 0.165475 0.482307
18 16 593 577 128.203 120 0.436253 0.486661
19 16 625 609 128.192 128 0.811796 0.487417
2025-09-06T18:03:02.348388+0000 min lat: 0.00306773 max lat: 1.89244 avg lat: 0.49023
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 655 639 127.781 120 0.281749 0.49023
21 16 685 669 127.41 120 0.437773 0.488087
22 16 716 700 127.254 124 0.410089 0.489592
23 16 748 732 127.286 128 0.362424 0.492715
24 16 780 764 127.315 128 0.256071 0.492451
25 16 810 794 127.022 120 0.294221 0.496227
26 16 842 826 127.059 128 0.280833 0.492168
27 16 871 855 126.649 116 1.64619 0.495698
28 16 905 889 126.982 136 0.349129 0.495225
29 16 938 922 127.154 132 1.37743 0.495949
30 16 973 957 127.582 140 0.00500597 0.492614
31 16 1004 988 127.466 124 0.205431 0.495119
32 16 1033 1017 127.107 116 1.08622 0.496186
33 16 1068 1052 127.497 140 0.125499 0.493873
34 16 1097 1081 127.159 116 0.187805 0.492131
35 16 1131 1115 127.411 136 0.535902 0.495211
36 16 1165 1149 127.649 136 0.648156 0.495292
37 16 1196 1180 127.55 124 0.174556 0.493856
38 16 1228 1212 127.561 128 0.465592 0.493075
39 16 1262 1246 127.777 136 0.0102957 0.495957
2025-09-06T18:03:22.351012+0000 min lat: 0.00306773 max lat: 2.0271 avg lat: 0.494969
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1295 1279 127.882 132 0.912497 0.494969
41 16 1323 1307 127.494 112 0.912498 0.496502
42 16 1356 1340 127.601 132 0.00948159 0.497076
43 16 1389 1373 127.703 132 0.0108206 0.497531
44 16 1425 1409 128.073 144 0.227257 0.494487
45 16 1455 1439 127.893 120 0.397512 0.494719
46 16 1483 1467 127.548 112 0.808184 0.49758
47 16 1512 1496 127.302 116 0.61087 0.498028
48 16 1547 1531 127.566 140 0.20288 0.497715
49 16 1578 1562 127.493 124 0.620353 0.498016
50 16 1616 1600 127.982 152 0.00426473 0.495593
51 16 1646 1630 127.825 120 1.04344 0.495678
52 16 1676 1660 127.674 120 0.339092 0.496449
53 16 1709 1693 127.755 132 0.434183 0.498095
54 16 1737 1721 127.463 112 0.00930642 0.497088
55 16 1771 1755 127.618 136 0.00946941 0.497972
56 16 1804 1788 127.696 132 0.259619 0.497103
57 16 1837 1821 127.771 132 0.00950038 0.49549
58 16 1869 1853 127.775 128 0.229229 0.495607
59 16 1903 1887 127.914 136 0.377198 0.495499
2025-09-06T18:03:42.353886+0000 min lat: 0.00300177 max lat: 2.0271 avg lat: 0.495681
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 1937 1921 128.049 136 0.229569 0.495681
61 16 1971 1955 128.179 136 0.179857 0.495987
62 16 2003 1987 128.176 128 0.148077 0.496047
63 16 2036 2020 128.236 132 0.00321485 0.494439
64 16 2073 2057 128.545 148 0.00506035 0.492693
65 16 2105 2089 128.536 128 0.00996371 0.493904
66 16 2137 2121 128.528 128 0.21248 0.494269
67 16 2166 2150 128.34 116 0.33767 0.495082
68 16 2195 2179 128.159 116 0.737051 0.496149
69 16 2224 2208 127.982 116 0.543227 0.496498
70 16 2256 2240 127.982 128 0.176781 0.496147
71 16 2285 2269 127.813 116 0.266752 0.497433
72 16 2316 2300 127.76 124 0.372841 0.497255
73 16 2353 2337 128.037 148 0.139255 0.496314
74 16 2385 2369 128.036 128 0.266525 0.496781
75 16 2420 2404 128.196 140 0.297324 0.496606
76 16 2448 2432 127.982 112 0.717249 0.497082
77 16 2480 2464 127.982 128 0.694474 0.497304
78 16 2514 2498 128.085 136 1.20036 0.496787
79 16 2547 2531 128.134 132 1.22903 0.497212
2025-09-06T18:04:02.356418+0000 min lat: 0.00300177 max lat: 2.0271 avg lat: 0.495893
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 2579 2563 128.132 128 0.60856 0.495893
81 16 2614 2598 128.279 140 0.216403 0.49574
82 16 2647 2631 128.324 132 1.40591 0.495954
83 16 2675 2659 128.127 112 0.183819 0.495189
84 16 2706 2690 128.078 124 0.192226 0.495735
85 16 2740 2724 128.171 136 0.00416184 0.494977
86 16 2773 2757 128.215 132 2.12443 0.494323
87 16 2802 2786 128.075 116 0.356535 0.495284
88 16 2834 2818 128.074 128 1.81285 0.496497
89 16 2870 2854 128.252 144 0.889957 0.496552
90 16 2902 2886 128.25 128 0.63383 0.496893
91 16 2931 2915 128.115 116 0.285331 0.496523
92 16 2963 2947 128.113 128 0.481716 0.497276
93 16 2992 2976 127.983 116 0.405642 0.497421
94 16 3024 3008 127.983 128 0.221838 0.497783
95 16 3057 3041 128.025 132 0.593946 0.497604
96 16 3090 3074 128.066 132 0.00412563 0.497358
97 16 3119 3103 127.942 116 0.00427057 0.497278
98 16 3147 3131 127.779 112 0.479791 0.498667
99 16 3178 3162 127.741 124 0.277126 0.498719
2025-09-06T18:04:22.358522+0000 min lat: 0.00300177 max lat: 2.36401 avg lat: 0.498278
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3214 3198 127.903 144 1.15525 0.498278
101 16 3245 3229 127.865 124 0.00952695 0.498026
102 16 3278 3262 127.905 132 0.191092 0.49801
103 16 3308 3292 127.828 120 0.640412 0.497861
104 16 3339 3323 127.791 124 1.95322 0.497655
105 16 3370 3354 127.755 124 0.36248 0.497762
106 16 3401 3385 127.719 124 0.230054 0.497732
107 16 3432 3416 127.685 124 1.6113 0.498838
108 16 3461 3445 127.576 116 1.37972 0.49905
109 16 3491 3475 127.507 120 1.53033 0.498807
110 16 3523 3507 127.511 128 0.256755 0.49812
111 16 3554 3538 127.479 124 1.62511 0.499213
112 16 3586 3570 127.484 128 0.15323 0.498856
113 16 3613 3597 127.311 108 0.197853 0.499316
114 16 3647 3631 127.387 136 0.699708 0.500205
115 16 3677 3661 127.323 120 0.00527622 0.499934
116 16 3714 3698 127.501 148 0.184277 0.498855
117 16 3743 3727 127.403 116 0.74199 0.499486
118 16 3774 3758 127.374 124 0.197858 0.500192
119 16 3806 3790 127.379 128 0.106141 0.500039
2025-09-06T18:04:42.360597+0000 min lat: 0.00299179 max lat: 2.36401 avg lat: 0.499846
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 3833 3817 127.217 108 0.209887 0.499846
121 16 3865 3849 127.224 128 0.28772 0.499589
122 16 3900 3884 127.328 140 0.369694 0.499959
123 16 3928 3912 127.203 112 0.901468 0.501408
124 16 3966 3950 127.403 152 0.246088 0.500682
125 16 4002 3986 127.536 144 0.513193 0.499997
126 16 4034 4018 127.54 128 0.152934 0.499787
127 16 4066 4050 127.543 128 0.580802 0.500011
128 16 4104 4088 127.734 152 0.208105 0.499095
129 16 4133 4117 127.643 116 0.121422 0.499389
130 16 4163 4147 127.584 120 0.0093297 0.499208
131 16 4196 4180 127.618 132 0.503652 0.499427
132 16 4227 4211 127.59 124 0.00991318 0.499936
133 16 4259 4243 127.593 128 0.678579 0.499814
134 16 4296 4280 127.745 148 0.687167 0.499521
135 16 4325 4309 127.658 116 0.0100017 0.499755
136 16 4357 4341 127.661 128 0.681882 0.499721
137 16 4389 4373 127.663 128 0.437386 0.499762
138 15 4422 4407 127.723 136 0.266809 0.499486
139 16 4454 4438 127.696 124 0.785598 0.499721
2025-09-06T18:05:02.363174+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.499186
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 4486 4470 127.698 128 0.978208 0.499186
141 16 4519 4503 127.729 132 0.383326 0.499622
142 16 4551 4535 127.73 128 0.392751 0.499306
143 16 4580 4564 127.648 116 1.63569 0.499092
144 16 4612 4596 127.651 128 0.154411 0.499328
145 16 4641 4625 127.57 116 0.117135 0.49903
146 16 4670 4654 127.491 116 0.149681 0.499358
147 16 4701 4685 127.467 124 1.44813 0.499939
148 16 4729 4713 127.362 112 0.193016 0.500454
149 16 4762 4746 127.393 132 0.00999606 0.50068
150 16 4796 4780 127.451 136 1.49075 0.500327
151 16 4829 4813 127.481 132 0.00429479 0.500366
152 15 4860 4845 127.484 128 0.487669 0.500598
153 16 4890 4874 127.409 116 0.253615 0.500241
154 16 4926 4910 127.516 144 1.58565 0.500283
155 16 4956 4940 127.468 120 1.02842 0.50047
156 16 4989 4973 127.497 132 0.250503 0.500153
157 16 5019 5003 127.449 120 0.811717 0.49989
158 16 5051 5035 127.452 128 1.79872 0.500299
159 16 5082 5066 127.431 124 0.242845 0.500416
2025-09-06T18:05:22.365656+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.501239
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 15 5111 5096 127.384 120 0.932433 0.501239
161 16 5145 5129 127.413 132 0.588867 0.500704
162 16 5180 5164 127.49 140 0.184245 0.500433
163 16 5209 5193 127.42 116 0.648174 0.500812
164 16 5243 5227 127.472 136 0.35894 0.500498
165 16 5275 5259 127.475 128 0.498188 0.500551
166 16 5312 5296 127.598 148 0.229462 0.500109
167 16 5345 5329 127.625 132 0.648958 0.499956
168 16 5382 5366 127.746 148 0.00464392 0.499642
169 16 5412 5396 127.7 120 0.00487543 0.499385
170 16 5448 5432 127.796 144 0.483526 0.499331
171 16 5477 5461 127.727 116 0.134459 0.498981
172 16 5508 5492 127.705 124 1.20563 0.49972
173 16 5536 5520 127.614 112 1.41229 0.499839
174 16 5567 5551 127.593 124 0.150071 0.5001
175 16 5598 5582 127.573 124 1.337 0.500327
176 16 5632 5616 127.62 136 0.227541 0.499891
177 16 5663 5647 127.6 124 0.880482 0.500197
178 16 5700 5684 127.714 148 0.686781 0.49984
179 16 5733 5717 127.738 132 0.689276 0.499841
2025-09-06T18:05:42.367930+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.499441
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 5769 5753 127.828 144 0.00934018 0.499441
181 16 5798 5782 127.763 116 0.417504 0.499759
182 16 5833 5817 127.83 140 0.487476 0.499255
183 16 5866 5850 127.853 132 0.00436459 0.498963
184 16 5896 5880 127.81 120 0.724651 0.499236
185 16 5931 5915 127.876 140 0.609206 0.498968
186 16 5961 5945 127.834 120 0.158926 0.498613
187 16 5993 5977 127.834 128 0.158989 0.499061
188 16 6022 6006 127.771 116 0.216444 0.499332
189 16 6054 6038 127.772 128 0.301984 0.499349
190 16 6087 6071 127.795 132 0.317744 0.499137
191 16 6119 6103 127.796 128 0.936912 0.499401
192 16 6151 6135 127.797 128 1.02494 0.499482
193 16 6180 6164 127.735 116 0.544841 0.499793
194 16 6216 6200 127.819 144 0.00325828 0.499267
195 16 6247 6231 127.8 124 0.00964512 0.499378
196 16 6284 6268 127.903 148 0.527441 0.498964
197 16 6315 6299 127.883 124 0.653334 0.499203
198 16 6348 6332 127.903 132 0.60937 0.499184
199 16 6379 6363 127.884 124 0.702848 0.4991
2025-09-06T18:06:02.370072+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.498833
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 6411 6395 127.884 128 0.00439229 0.498833
201 16 6441 6425 127.845 120 0.188819 0.498802
202 16 6479 6463 127.964 152 0.254095 0.498431
203 16 6510 6494 127.945 124 1.70776 0.498496
204 16 6542 6526 127.945 128 1.91442 0.498379
205 16 6574 6558 127.945 128 0.236667 0.498629
206 16 6602 6586 127.868 112 0.378327 0.498789
207 16 6638 6622 127.946 144 0.217619 0.498294
208 16 6669 6653 127.927 124 2.02291 0.498433
209 16 6707 6691 128.042 152 0.170598 0.49804
210 16 6738 6722 128.022 124 0.179312 0.498469
211 16 6769 6753 128.003 124 1.0306 0.49887
212 16 6802 6786 128.022 132 0.563567 0.498485
213 16 6836 6820 128.059 136 0.335909 0.498259
214 16 6868 6852 128.059 128 1.41516 0.498427
215 16 6901 6885 128.077 132 0.372781 0.498417
216 16 6935 6919 128.114 136 0.00354265 0.498329
217 16 6965 6949 128.076 120 1.28527 0.498382
218 16 6998 6982 128.094 132 0.152524 0.498051
219 16 7027 7011 128.039 116 0.286006 0.498115
2025-09-06T18:06:22.372424+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.498139
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 7059 7043 128.039 128 0.00392951 0.498139
221 16 7090 7074 128.021 124 0.278302 0.498314
222 16 7119 7103 127.966 116 0.00933923 0.498315
223 16 7150 7134 127.948 124 1.5835 0.498636
224 16 7181 7165 127.931 124 0.60661 0.498852
225 16 7214 7198 127.949 132 0.162822 0.498868
226 16 7248 7232 127.984 136 0.432917 0.498761
227 16 7279 7263 127.967 124 0.312608 0.498858
228 16 7316 7300 128.055 148 0.379037 0.49854
229 16 7348 7332 128.054 128 0.222594 0.498587
230 16 7381 7365 128.071 132 0.638524 0.498586
231 16 7410 7394 128.019 116 0.0102252 0.498793
232 16 7442 7426 128.019 128 0.342715 0.498807
233 16 7475 7459 128.036 132 0.283528 0.498647
234 16 7502 7486 127.95 108 0.32061 0.498819
235 16 7532 7516 127.916 120 0.423818 0.499084
236 16 7564 7548 127.917 128 1.05935 0.499046
237 16 7597 7581 127.934 132 0.3387 0.499032
238 16 7628 7612 127.917 124 0.470145 0.499028
239 16 7661 7645 127.934 132 0.276346 0.499079
2025-09-06T18:06:42.374509+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.499308
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 7691 7675 127.901 120 0.630996 0.499308
241 16 7724 7708 127.918 132 0.653379 0.499234
242 16 7759 7743 127.968 140 0.382368 0.499024
243 16 7794 7778 128.017 140 0.00389913 0.498749
244 16 7823 7807 127.968 116 1.42538 0.49891
245 16 7855 7839 127.968 128 1.20369 0.498941
246 16 7892 7876 128.05 148 0.391859 0.498501
247 16 7925 7909 128.066 132 0.15492 0.498133
248 16 7956 7940 128.049 124 1.61748 0.49845
249 16 7987 7971 128.033 124 0.00471052 0.498296
250 16 8017 8001 128.001 120 1.9789 0.498616
251 16 8049 8033 128.001 128 0.719657 0.49844
252 16 8079 8063 127.969 120 0.146483 0.498867
253 16 8116 8100 128.048 148 0.328742 0.498303
254 16 8153 8137 128.126 148 0.287973 0.497678
255 16 8186 8170 128.141 132 0.208681 0.498128
256 16 8223 8207 128.219 148 1.42616 0.497903
257 16 8258 8242 128.265 140 0.157802 0.497907
258 16 8288 8272 128.233 120 0.227412 0.49754
259 16 8321 8305 128.247 132 0.134095 0.497593
2025-09-06T18:07:02.376904+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.497811
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 8350 8334 128.2 116 0.597617 0.497811
261 16 8381 8365 128.184 124 1.46212 0.497966
262 16 8412 8396 128.168 124 0.159406 0.497709
263 16 8446 8430 128.197 136 2.19868 0.497516
264 16 8477 8461 128.181 124 0.147281 0.497576
265 16 8509 8493 128.181 128 0.621998 0.497842
266 16 8538 8522 128.135 116 0.267175 0.498179
267 16 8573 8557 128.179 140 1.26651 0.498039
268 16 8616 8600 128.343 172 0.00420792 0.497629
269 16 8647 8631 128.327 124 0.00442937 0.497774
270 16 8680 8664 128.34 132 0.010723 0.497621
271 16 8711 8695 128.324 124 0.576568 0.49769
272 16 8746 8730 128.367 140 0.931825 0.497696
273 16 8777 8761 128.351 124 0.387395 0.497583
274 16 8814 8798 128.423 148 0.0101271 0.497352
275 16 8846 8830 128.421 128 0.728713 0.4974
276 16 8879 8863 128.434 132 0.760732 0.497348
277 16 8909 8893 128.403 120 0.581887 0.497206
278 16 8941 8925 128.402 128 0.57315 0.497442
279 16 8970 8954 128.357 116 1.18111 0.497594
2025-09-06T18:07:22.378922+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.497688
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 8999 8983 128.313 116 1.12803 0.497688
281 16 9033 9017 128.341 136 1.15891 0.497727
282 16 9064 9048 128.325 124 0.00320911 0.497727
283 16 9098 9082 128.352 136 0.234972 0.497717
284 16 9130 9114 128.351 128 0.404145 0.497589
285 16 9163 9147 128.364 132 0.313078 0.497574
286 16 9195 9179 128.362 128 0.185359 0.497249
287 16 9230 9214 128.403 140 0.349621 0.497439
288 16 9257 9241 128.332 108 0.489215 0.497554
289 16 9291 9275 128.358 136 0.889491 0.497648
290 16 9321 9305 128.33 120 1.13856 0.497582
291 16 9355 9339 128.356 136 0.118936 0.497343
292 16 9386 9370 128.341 124 0.488898 0.497485
293 16 9417 9401 128.326 124 0.20372 0.497714
294 16 9445 9429 128.27 112 0.427066 0.497997
295 16 9479 9463 128.297 136 0.265628 0.497932
296 16 9511 9495 128.296 128 0.418079 0.497851
297 16 9537 9521 128.214 104 0.429063 0.498098
298 16 9568 9552 128.2 124 0.23329 0.498192
299 16 9606 9590 128.279 152 0.910775 0.497995
2025-09-06T18:07:42.381048+0000 min lat: 0.00297007 max lat: 2.36401 avg lat: 0.497992
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 9637 9621 128.265 124 0.608789 0.497992
301 16 9673 9657 128.317 144 0.255794 0.497664
302 16 9703 9687 128.289 120 1.12719 0.497914
303 16 9737 9721 128.315 136 0.737864 0.497867
304 16 9767 9751 128.287 120 0.262968 0.497586
305 16 9804 9788 128.352 148 0.94377 0.497755
306 16 9835 9819 128.338 124 0.487529 0.497775
307 16 9870 9854 128.376 140 0.00387504 0.49759
308 16 9899 9883 128.335 116 0.569328 0.497772
309 16 9934 9918 128.373 140 0.442329 0.497707
310 16 9974 9958 128.475 160 0.00426233 0.49715
311 16 10004 9988 128.448 120 0.154078 0.497264
312 16 10041 10025 128.51 148 0.00293449 0.496826
313 16 10071 10055 128.483 120 0.520624 0.496829
314 16 10107 10091 128.533 144 1.78664 0.496991
315 16 10137 10121 128.505 120 1.25159 0.4971
316 16 10170 10154 128.516 132 0.680811 0.497233
317 16 10207 10191 128.578 148 0.0030961 0.496805
318 16 10239 10223 128.576 128 0.12769 0.496662
319 16 10270 10254 128.562 124 0.273306 0.496537
2025-09-06T18:08:02.383278+0000 min lat: 0.00293449 max lat: 2.36401 avg lat: 0.496638
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 10305 10289 128.597 140 0.646938 0.496638
321 16 10338 10322 128.608 132 0.35997 0.496654
322 16 10366 10350 128.556 112 0.116863 0.496595
323 16 10402 10386 128.604 144 1.58206 0.496481
324 16 10436 10420 128.627 136 0.00312157 0.496438
325 16 10467 10451 128.613 124 0.41607 0.496649
326 16 10497 10481 128.586 120 0.281999 0.496806
327 16 10528 10512 128.572 124 0.167662 0.496854
328 16 10558 10542 128.546 120 0.459874 0.49711
329 16 10589 10573 128.532 124 0.536428 0.497116
330 16 10617 10601 128.482 112 0.289528 0.497259
331 16 10647 10631 128.456 120 0.186033 0.497293
332 16 10682 10666 128.491 140 0.990862 0.49706
333 16 10713 10697 128.477 124 0.250918 0.497082
334 16 10742 10726 128.44 116 0.243726 0.497182
335 16 10774 10758 128.439 128 0.822844 0.49746
336 16 10805 10789 128.425 124 0.147175 0.497183
337 16 10833 10817 128.377 112 0.179403 0.497404
338 16 10867 10851 128.399 136 0.130057 0.497052
339 16 10897 10881 128.374 120 0.162936 0.496973
2025-09-06T18:08:22.385343+0000 min lat: 0.00293449 max lat: 2.37439 avg lat: 0.497198
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 10927 10911 128.35 120 0.321889 0.497198
341 16 10962 10946 128.384 140 0.318759 0.497307
342 16 10995 10979 128.394 132 0.00998479 0.497422
343 16 11024 11008 128.358 116 0.00958292 0.497612
344 16 11058 11042 128.38 136 0.00394575 0.497569
345 16 11087 11071 128.344 116 0.802075 0.497839
346 16 11120 11104 128.355 132 0.781992 0.497708
347 16 11152 11136 128.354 128 0.361345 0.497657
348 16 11183 11167 128.341 124 1.19181 0.497768
349 16 11219 11203 128.386 144 0.462863 0.497449
350 16 11248 11232 128.351 116 1.18204 0.49769
351 16 11275 11259 128.293 108 0.23239 0.497753
352 16 11311 11295 128.337 144 0.00412308 0.497561
353 16 11351 11335 128.427 160 0.241098 0.497104
354 16 11380 11364 128.392 116 0.105852 0.49726
355 16 11411 11395 128.379 124 0.004077 0.497323
356 16 11440 11424 128.345 116 0.295806 0.497335
357 16 11474 11458 128.366 136 1.36883 0.497611
358 16 11505 11489 128.354 124 0.347334 0.497475
359 16 11537 11521 128.353 128 1.76748 0.497587
2025-09-06T18:08:42.387535+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.497728
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 11568 11552 128.341 124 0.251515 0.497728
361 16 11596 11580 128.295 112 0.58145 0.497844
362 16 11630 11614 128.317 136 0.00938028 0.49783
363 16 11665 11649 128.349 140 1.10464 0.497648
364 16 11699 11683 128.37 136 0.230716 0.49737
365 16 11732 11716 128.38 132 0.336426 0.497389
366 16 11762 11746 128.357 120 0.356947 0.497469
367 16 11792 11776 128.334 120 0.246938 0.497574
368 16 11822 11806 128.311 120 0.126272 0.497851
369 16 11856 11840 128.332 136 0.678884 0.49775
370 16 11887 11871 128.32 124 1.51797 0.497687
371 16 11920 11904 128.33 132 0.138966 0.497656
372 16 11952 11936 128.329 128 1.99888 0.497817
373 16 11983 11967 128.318 124 0.00486749 0.497818
374 16 12017 12001 128.338 136 0.247335 0.497399
375 16 12051 12035 128.358 136 0.00401746 0.497509
376 16 12081 12065 128.336 120 0.302152 0.497711
377 16 12114 12098 128.346 132 0.0033209 0.497667
378 16 12147 12131 128.356 132 0.24 0.497573
379 16 12177 12161 128.333 120 1.52235 0.49764
2025-09-06T18:09:02.389397+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.497534
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 12212 12196 128.364 140 0.0046087 0.497534
381 16 12246 12230 128.384 136 0.746804 0.497757
382 16 12277 12261 128.373 124 0.628044 0.497782
383 16 12310 12294 128.382 132 0.222153 0.497561
384 16 12342 12326 128.381 128 0.154559 0.49742
385 16 12377 12361 128.411 140 0.272726 0.497449
386 16 12410 12394 128.42 132 0.0103072 0.497426
387 16 12440 12424 128.399 120 0.391835 0.497366
388 16 12470 12454 128.377 120 0.470033 0.497568
389 16 12504 12488 128.397 136 1.45165 0.497424
390 16 12541 12525 128.447 148 1.66159 0.497426
391 16 12583 12567 128.548 168 0.249008 0.496928
392 16 12613 12597 128.526 120 0.611948 0.497084
393 16 12646 12630 128.535 132 0.0900817 0.497108
394 16 12672 12656 128.473 104 0.144182 0.497042
395 16 12705 12689 128.481 132 0.173422 0.496869
396 16 12736 12720 128.47 124 0.167057 0.496983
397 16 12767 12751 128.459 124 0.413334 0.497113
398 16 12802 12786 128.488 140 0.00421749 0.49736
399 16 12832 12816 128.467 120 0.547067 0.497295
2025-09-06T18:09:22.391081+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.497417
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 12860 12844 128.425 112 0.20035 0.497417
401 16 12894 12878 128.444 136 0.300884 0.497506
402 16 12924 12908 128.423 120 0.650455 0.497617
403 16 12955 12939 128.412 124 0.483085 0.497552
404 16 12984 12968 128.381 116 0.37074 0.497539
405 16 13017 13001 128.39 132 1.02046 0.497653
406 16 13049 13033 128.389 128 0.381209 0.497661
407 16 13084 13068 128.418 140 0.00510433 0.497403
408 16 13123 13107 128.485 156 0.898466 0.497262
409 16 13156 13140 128.494 132 0.274762 0.497247
410 16 13187 13171 128.483 124 0.451803 0.497308
411 16 13216 13200 128.453 116 0.362619 0.49745
412 16 13252 13236 128.49 144 0.278338 0.497328
413 16 13285 13269 128.499 132 0.422513 0.497206
414 16 13319 13303 128.517 136 0.751218 0.497228
415 16 13349 13333 128.496 120 0.679691 0.49728
416 16 13382 13366 128.505 132 0.198557 0.497268
417 16 13413 13397 128.494 124 0.42791 0.497144
418 16 13446 13430 128.502 132 0.18337 0.497163
419 16 13483 13467 128.549 148 0.00940943 0.497045
2025-09-06T18:09:42.393164+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.497132
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 13512 13496 128.519 116 0.188586 0.497132
421 16 13539 13523 128.47 108 0.412928 0.497319
422 16 13570 13554 128.459 124 1.1894 0.49742
423 16 13602 13586 128.458 128 0.759103 0.497457
424 16 13634 13618 128.457 128 0.0039798 0.497371
425 16 13665 13649 128.447 124 0.445506 0.497423
426 16 13701 13685 128.483 144 0.449915 0.497339
427 16 13734 13718 128.491 132 0.00442037 0.497332
428 16 13767 13751 128.499 132 0.225952 0.497251
429 16 13802 13786 128.526 140 0.289564 0.497016
430 16 13832 13816 128.506 120 1.51509 0.497158
431 16 13863 13847 128.496 124 0.611599 0.497208
432 16 13896 13880 128.504 132 0.00442923 0.497194
433 16 13930 13914 128.521 136 0.991144 0.497074
434 16 13965 13949 128.548 140 0.210871 0.497006
435 16 14000 13984 128.574 140 1.36111 0.496777
436 16 14033 14017 128.582 132 0.593111 0.496784
437 16 14064 14048 128.571 124 1.79177 0.496686
438 16 14096 14080 128.57 128 1.9373 0.496929
439 16 14127 14111 128.56 124 0.409606 0.497037
2025-09-06T18:10:02.395110+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.496873
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 14162 14146 128.585 140 0.333377 0.496873
441 16 14194 14178 128.584 128 0.659246 0.49701
442 16 14223 14207 128.556 116 0.235254 0.497142
443 16 14257 14241 128.572 136 0.566517 0.49702
444 16 14289 14273 128.571 128 1.06056 0.497045
445 16 14322 14306 128.579 132 0.123148 0.496853
446 16 14355 14339 128.586 132 0.167563 0.496948
447 16 14390 14374 128.612 140 0.00942157 0.496858
448 16 14423 14407 128.619 132 0.198323 0.496648
449 16 14456 14440 128.627 132 1.74442 0.49655
450 16 14493 14477 128.67 148 0.25956 0.496533
451 16 14525 14509 128.668 128 0.00961006 0.496509
452 16 14554 14538 128.64 116 0.260049 0.496455
453 16 14588 14572 128.657 136 0.795083 0.496583
454 16 14620 14604 128.655 128 0.719707 0.496504
455 16 14652 14636 128.654 128 0.665609 0.496614
456 16 14689 14673 128.696 148 0.331472 0.496422
457 16 14718 14702 128.668 116 0.516364 0.496535
458 16 14746 14730 128.632 112 0.709876 0.496753
459 16 14776 14760 128.613 120 0.600357 0.496775
2025-09-06T18:10:22.397169+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.496844
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 14806 14790 128.594 120 0.184671 0.496844
461 16 14842 14826 128.628 144 0.16644 0.49679
462 16 14872 14856 128.609 120 1.36522 0.496756
463 16 14906 14890 128.625 136 1.3291 0.496853
464 16 14934 14918 128.589 112 0.107218 0.496809
465 16 14966 14950 128.588 128 0.182398 0.49665
466 16 14995 14979 128.561 116 0.216869 0.496707
467 15 15031 15016 128.602 148 0.00317107 0.496749
468 16 15061 15045 128.575 116 0.753498 0.496893
469 16 15097 15081 128.608 144 0.00512218 0.496919
470 16 15126 15110 128.581 116 1.00771 0.496943
471 16 15156 15140 128.563 120 1.2514 0.496968
472 16 15185 15169 128.536 116 0.408152 0.49715
473 16 15217 15201 128.535 128 0.293741 0.497095
474 16 15248 15232 128.526 124 0.0103043 0.497054
475 16 15279 15263 128.516 124 0.301884 0.497099
476 16 15310 15294 128.506 124 1.28326 0.497195
477 16 15342 15326 128.505 128 0.304381 0.497129
478 16 15371 15355 128.479 116 0.248527 0.497386
479 16 15404 15388 128.487 132 0.298936 0.497271
2025-09-06T18:10:42.399750+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.497136
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 15435 15419 128.477 124 0.195316 0.497136
481 16 15468 15452 128.484 132 0.346396 0.497212
482 16 15500 15484 128.483 128 1.64571 0.49719
483 16 15534 15518 128.499 136 0.224342 0.497128
484 16 15569 15553 128.523 140 0.175996 0.4971
485 16 15600 15584 128.513 124 0.492342 0.497199
486 16 15631 15615 128.504 124 0.00989235 0.497282
487 16 15662 15646 128.495 124 0.240696 0.49724
488 16 15692 15676 128.477 120 0.357431 0.497365
489 16 15722 15706 128.46 120 0.0103692 0.497439
490 16 15754 15738 128.459 128 0.338325 0.497354
491 16 15795 15779 128.531 164 0.00476621 0.497076
492 16 15828 15812 128.538 132 0.278356 0.496901
493 16 15860 15844 128.537 128 0.151095 0.496907
494 16 15893 15877 128.544 132 0.00938923 0.496878
495 16 15928 15912 128.567 140 1.86682 0.496876
496 16 15959 15943 128.558 124 0.213942 0.496948
497 16 15988 15972 128.533 116 0.112799 0.496948
498 16 16022 16006 128.548 136 0.780793 0.497022
499 16 16050 16034 128.515 112 0.00925413 0.497204
2025-09-06T18:11:02.401800+0000 min lat: 0.00293449 max lat: 2.58677 avg lat: 0.497338
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 16080 16064 128.498 120 0.809226 0.497338
501 16 16114 16098 128.512 136 0.295834 0.497283
502 16 16147 16131 128.519 132 1.14744 0.497259
503 16 16181 16165 128.534 136 0.746442 0.497218
504 16 16210 16194 128.509 116 0.424045 0.497287
505 16 16243 16227 128.516 132 0.143943 0.49709
506 16 16275 16259 128.515 128 1.54463 0.497075
507 16 16305 16289 128.498 120 0.168674 0.49707
508 16 16342 16326 128.537 148 1.66319 0.497186
509 16 16372 16356 128.52 120 0.207433 0.497178
510 16 16400 16384 128.488 112 0.550999 0.497291
511 16 16437 16421 128.526 148 0.164468 0.497246
512 16 16472 16456 128.548 140 1.27 0.497025
513 16 16503 16487 128.539 124 0.187189 0.496938
514 16 16533 16517 128.523 120 2.1235 0.497068
515 16 16563 16547 128.506 120 1.53499 0.497213
516 16 16595 16579 128.505 128 0.317535 0.497196
517 16 16626 16610 128.496 124 0.00927053 0.497198
518 16 16655 16639 128.472 116 0.444921 0.497262
519 16 16690 16674 128.494 140 0.127166 0.497244
2025-09-06T18:11:22.403507+0000 min lat: 0.00291441 max lat: 2.58677 avg lat: 0.497203
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 16723 16707 128.501 132 0.356817 0.497203
521 16 16751 16735 128.469 112 0.212603 0.49732
522 16 16787 16771 128.499 144 1.67788 0.497206
523 16 16819 16803 128.498 128 1.70007 0.497199
524 16 16848 16832 128.474 116 0.391044 0.497213
525 16 16879 16863 128.466 124 0.504305 0.49739
526 16 16914 16898 128.488 140 0.114174 0.497298
527 16 16946 16930 128.487 128 0.164095 0.497207
528 16 16974 16958 128.455 112 0.178093 0.497213
529 16 17007 16991 128.462 132 0.220819 0.497258
530 16 17036 17020 128.439 116 1.73876 0.497325
531 16 17072 17056 128.468 144 0.00446915 0.497221
532 16 17105 17089 128.474 132 0.0100984 0.497302
533 16 17134 17118 128.451 116 0.442438 0.497368
534 16 17168 17152 128.465 136 0.586345 0.497403
535 16 17200 17184 128.464 128 0.00446841 0.497412
536 16 17229 17213 128.441 116 0.406347 0.497432
537 16 17260 17244 128.433 124 1.87135 0.497431
538 16 17296 17280 128.462 144 0.00376238 0.497202
539 16 17328 17312 128.461 128 0.134363 0.497312
2025-09-06T18:11:42.405461+0000 min lat: 0.00291441 max lat: 2.58677 avg lat: 0.497366
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 17359 17343 128.452 124 0.301831 0.497366
541 16 17393 17377 128.466 136 1.12597 0.497437
542 16 17422 17406 128.443 116 0.00516023 0.497583
543 16 17460 17444 128.487 152 0.832456 0.497443
544 16 17493 17477 128.493 132 0.668396 0.497399
545 16 17526 17510 128.5 132 0.29177 0.497228
546 16 17558 17542 128.499 128 0.209682 0.497328
547 16 17594 17578 128.527 144 0.463534 0.497163
548 16 17624 17608 128.511 120 0.00932494 0.497003
549 16 17659 17643 128.532 140 1.938 0.497098
550 16 17685 17669 128.488 104 1.81873 0.497165
551 16 17717 17701 128.487 128 0.255139 0.497356
552 16 17751 17735 128.5 136 0.1688 0.497142
553 16 17780 17764 128.478 116 0.264821 0.497443
554 16 17812 17796 128.477 128 1.4639 0.497441
555 16 17842 17826 128.461 120 0.00531299 0.497339
556 16 17872 17856 128.446 120 0.16497 0.497339
557 16 17901 17885 128.424 116 1.26066 0.497556
558 16 17932 17916 128.416 124 0.41316 0.497665
559 16 17969 17953 128.451 148 0.0103564 0.497552
2025-09-06T18:12:02.407207+0000 min lat: 0.00291441 max lat: 2.58677 avg lat: 0.497469
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 18002 17986 128.457 132 0.451508 0.497469
561 16 18029 18013 128.421 108 0.272648 0.49754
562 16 18069 18053 128.477 160 0.00310979 0.497458
563 16 18099 18083 128.462 120 1.04354 0.497444
564 16 18133 18117 128.475 136 0.00432079 0.497516
565 16 18163 18147 128.46 120 0.73703 0.497569
566 16 18195 18179 128.459 128 0.610731 0.497535
567 16 18228 18212 128.466 132 1.04296 0.497543
568 16 18256 18240 128.437 112 0.221794 0.497525
569 16 18288 18272 128.436 128 0.277707 0.49758
570 16 18321 18305 128.442 132 0.476984 0.497555
571 16 18355 18339 128.455 136 0.305267 0.497396
572 16 18383 18367 128.426 112 0.215524 0.497386
573 16 18411 18395 128.398 112 0.12944 0.497608
574 16 18442 18426 128.39 124 0.146728 0.497584
575 16 18471 18455 128.369 116 1.74543 0.497728
576 16 18508 18492 128.403 148 0.637231 0.497643
577 16 18541 18525 128.409 132 0.988765 0.49774
578 16 18572 18556 128.401 124 1.36235 0.497728
579 16 18608 18592 128.428 144 0.180485 0.497658
2025-09-06T18:12:22.409018+0000 min lat: 0.00291441 max lat: 2.58677 avg lat: 0.497377
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 18644 18628 128.455 144 0.225034 0.497377
581 16 18681 18665 128.489 148 0.277628 0.497166
582 16 18711 18695 128.474 120 0.175279 0.497284
583 16 18742 18726 128.466 124 0.12403 0.497227
584 16 18777 18761 128.486 140 0.393684 0.497191
585 16 18808 18792 128.478 124 0.004276 0.497436
586 16 18837 18821 128.457 116 0.31828 0.497418
587 16 18867 18851 128.443 120 0.212868 0.497595
588 16 18899 18883 128.442 128 0.26975 0.497515
589 16 18938 18922 128.489 156 0.729405 0.497448
590 16 18967 18951 128.467 116 0.277506 0.497323
591 16 18999 18983 128.467 128 0.189254 0.497343
592 16 19032 19016 128.472 132 0.371961 0.497404
593 16 19071 19055 128.519 156 0.1841 0.497135
594 16 19103 19087 128.518 128 0.222556 0.497222
595 16 19133 19117 128.504 120 0.0042575 0.497356
596 16 19165 19149 128.503 128 0.490665 0.497375
597 16 19197 19181 128.502 128 1.10989 0.497391
598 16 19228 19212 128.494 124 1.40869 0.497253
599 16 19264 19248 128.52 144 0.00499039 0.497101
2025-09-06T18:12:42.410876+0000 min lat: 0.0029092 max lat: 2.58677 avg lat: 0.497095
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 15 19298 19283 128.539 140 0.215206 0.497095
601 16 19328 19312 128.518 116 0.627554 0.497154
602 16 19362 19346 128.531 136 0.188996 0.497192
603 16 19393 19377 128.523 124 0.198215 0.497057
604 15 19430 19415 128.562 152 0.185431 0.497055
605 16 19458 19442 128.528 108 0.409243 0.497241
606 16 19489 19473 128.521 124 0.548066 0.497261
607 16 19521 19505 128.52 128 1.51215 0.497263
608 16 19552 19536 128.512 124 0.214283 0.497237
609 16 19585 19569 128.518 132 0.182495 0.49729
610 16 19614 19598 128.497 116 0.39932 0.497339
611 16 19651 19635 128.529 148 0.00471426 0.49734
612 16 19684 19668 128.535 132 0.225195 0.497286
613 16 19718 19702 128.547 136 0.368972 0.497151
614 16 19751 19735 128.553 132 0.00518921 0.49721
615 16 19782 19766 128.545 124 0.179398 0.497117
616 16 19810 19794 128.518 112 0.00452079 0.497266
617 16 19845 19829 128.537 140 1.32393 0.497254
618 16 19877 19861 128.536 128 0.0048083 0.497315
619 16 19908 19892 128.529 124 1.24167 0.497288
2025-09-06T18:13:02.413325+0000 min lat: 0.0029092 max lat: 2.58677 avg lat: 0.497173
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
620 16 19945 19929 128.56 148 0.178552 0.497173
621 16 19977 19961 128.559 128 0.00954097 0.497243
622 16 20009 19993 128.558 128 0.394515 0.49719
623 16 20038 20022 128.538 116 0.825439 0.497259
624 16 20073 20057 128.556 140 0.358026 0.497163
625 16 20105 20089 128.556 128 0.337663 0.497203
626 16 20136 20120 128.548 124 0.156441 0.497215
627 16 20174 20158 128.586 152 0.00956967 0.49695
628 16 20207 20191 128.591 132 0.283944 0.497063
629 16 20239 20223 128.59 128 1.19294 0.497002
630 15 20269 20254 128.583 124 0.898571 0.497116
631 16 20299 20283 128.563 116 1.32028 0.497156
632 16 20334 20318 128.581 140 0.151176 0.497075
633 16 20364 20348 128.567 120 0.314777 0.497157
634 16 20393 20377 128.547 116 0.559239 0.497228
635 16 20430 20414 128.578 148 0.371777 0.497144
636 16 20460 20444 128.565 120 0.389895 0.497118
637 16 20494 20478 128.576 136 0.471799 0.496986
638 16 20527 20511 128.582 132 0.125774 0.497019
639 16 20561 20545 128.593 136 0.353528 0.49702
2025-09-06T18:13:22.415323+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.49705
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
640 16 20593 20577 128.592 128 0.242985 0.49705
641 16 20626 20610 128.598 132 0.501938 0.497057
642 16 20662 20646 128.621 144 0.900247 0.49696
643 16 20696 20680 128.633 136 0.190519 0.496823
644 16 20727 20711 128.626 124 0.00453893 0.49687
645 16 20757 20741 128.612 120 0.356759 0.496907
646 16 20787 20771 128.599 120 1.6677 0.496939
647 16 20819 20803 128.598 128 0.32035 0.496891
648 16 20852 20836 128.603 132 0.477224 0.496956
649 16 20884 20868 128.602 128 0.153023 0.496921
650 16 20916 20900 128.601 128 0.00959983 0.496959
651 16 20947 20931 128.594 124 0.269298 0.497082
652 16 20979 20963 128.593 128 0.33172 0.496983
653 16 21013 20997 128.605 136 0.469143 0.497067
654 16 21044 21028 128.598 124 0.122382 0.496946
655 16 21074 21058 128.584 120 0.291679 0.497025
656 16 21103 21087 128.565 116 0.517548 0.497162
657 16 21134 21118 128.558 124 0.408259 0.497201
658 16 21165 21149 128.551 124 0.523012 0.49724
659 16 21193 21177 128.526 112 1.06902 0.497306
2025-09-06T18:13:42.417076+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.497292
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
660 16 21223 21207 128.513 120 0.191887 0.497292
661 16 21262 21246 128.555 156 0.00956121 0.497196
662 16 21299 21283 128.584 148 0.0650634 0.496954
663 16 21330 21314 128.577 124 0.215255 0.49685
664 16 21367 21351 128.606 148 2.07179 0.496859
665 16 21398 21382 128.6 124 0.931835 0.496978
666 16 21431 21415 128.605 132 1.30033 0.496981
667 16 21460 21444 128.586 116 0.00987722 0.497004
668 16 21493 21477 128.591 132 0.872785 0.49707
669 16 21530 21514 128.62 148 0.725586 0.496934
670 16 21561 21545 128.613 124 0.176933 0.49689
671 16 21594 21578 128.618 132 0.00413939 0.49696
672 16 21628 21612 128.629 136 0.00937935 0.496941
673 16 21658 21642 128.616 120 1.67573 0.496893
674 16 21693 21677 128.633 140 0.346008 0.496861
675 16 21728 21712 128.65 140 0.00957494 0.496894
676 16 21760 21744 128.649 128 0.951695 0.49686
677 16 21799 21783 128.689 156 0.491442 0.496723
678 16 21829 21813 128.676 120 1.14673 0.4967
679 16 21862 21846 128.681 132 0.223762 0.49661
2025-09-06T18:14:02.419549+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496638
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
680 16 21894 21878 128.68 128 1.6374 0.496638
681 16 21930 21914 128.703 144 0.142292 0.496563
682 16 21963 21947 128.707 132 0.00485172 0.496554
683 16 21994 21978 128.7 124 0.444104 0.496678
684 16 22027 22011 128.705 132 0.51119 0.496691
685 16 22059 22043 128.704 128 0.550832 0.496662
686 16 22092 22076 128.709 132 0.804719 0.496635
687 16 22126 22110 128.72 136 0.219605 0.496516
688 16 22154 22138 128.695 112 0.436212 0.496581
689 16 22191 22175 128.723 148 0.671074 0.496484
690 16 22223 22207 128.722 128 0.275451 0.496467
691 16 22251 22235 128.698 112 1.8259 0.496511
692 16 22286 22270 128.714 140 0.4962 0.496535
693 16 22316 22300 128.702 120 0.163581 0.496572
694 16 22348 22332 128.701 128 1.48503 0.496635
695 16 22380 22364 128.7 128 1.15408 0.496575
696 16 22415 22399 128.716 140 0.494578 0.496585
697 16 22449 22433 128.726 136 0.179676 0.496485
698 16 22480 22464 128.72 124 0.442427 0.496593
699 16 22513 22497 128.724 132 0.00957192 0.496549
2025-09-06T18:14:22.421243+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496549
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
700 16 22545 22529 128.723 128 0.275863 0.496549
701 16 22574 22558 128.705 116 0.303003 0.49659
702 16 22602 22586 128.681 112 0.534422 0.49653
703 16 22636 22620 128.692 136 0.44303 0.496636
704 16 22667 22651 128.685 124 0.304948 0.496726
705 16 22699 22683 128.684 128 0.381443 0.496784
706 16 22728 22712 128.666 116 0.29457 0.496729
707 16 22763 22747 128.682 140 0.940405 0.496801
708 16 22796 22780 128.687 132 0.159662 0.496719
709 15 22826 22811 128.68 124 0.232827 0.496669
710 16 22863 22847 128.702 144 0.163895 0.496553
711 16 22892 22876 128.684 116 0.155947 0.496713
712 16 22925 22909 128.688 132 0.222762 0.496598
713 16 22959 22943 128.699 136 0.346232 0.496502
714 16 22992 22976 128.703 132 0.897818 0.496611
715 16 23029 23013 128.73 148 0.370455 0.496495
716 16 23064 23048 128.746 140 1.2132 0.496396
717 16 23098 23082 128.756 136 0.246506 0.496254
718 16 23129 23113 128.749 124 0.365073 0.496335
719 16 23157 23141 128.726 112 1.85838 0.496445
2025-09-06T18:14:42.423286+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496496
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
720 16 23188 23172 128.719 124 0.0957588 0.496496
721 16 23220 23204 128.718 128 0.00445857 0.496445
722 16 23251 23235 128.712 124 0.223392 0.496647
723 16 23282 23266 128.705 124 0.469905 0.496617
724 16 23317 23301 128.721 140 0.541303 0.496521
725 16 23350 23334 128.725 132 0.273359 0.496601
726 16 23384 23368 128.735 136 0.120779 0.496537
727 16 23414 23398 128.723 120 0.222828 0.496453
728 16 23448 23432 128.733 136 0.245954 0.496331
729 16 23483 23467 128.749 140 0.00940112 0.496276
730 16 23514 23498 128.742 124 0.232406 0.496379
731 16 23542 23526 128.719 112 1.81049 0.496427
732 16 23576 23560 128.729 136 0.432407 0.49647
733 16 23607 23591 128.723 124 0.334011 0.496532
734 16 23641 23625 128.733 136 0.574354 0.496594
735 16 23671 23655 128.721 120 0.724521 0.496602
736 16 23703 23687 128.72 128 1.0496 0.496614
737 16 23735 23719 128.719 128 0.263671 0.496569
738 16 23772 23756 128.745 148 0.00475692 0.496481
739 16 23801 23785 128.728 116 0.189403 0.496458
2025-09-06T18:15:02.424997+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496606
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
740 16 23830 23814 128.71 116 1.0814 0.496606
741 16 23862 23846 128.71 128 0.664569 0.496664
742 16 23899 23883 128.735 148 0.00462682 0.496582
743 16 23931 23915 128.734 128 0.00935809 0.496551
744 16 23964 23948 128.739 132 0.637936 0.49652
745 16 23998 23982 128.749 136 0.366649 0.49648
746 16 24030 24014 128.748 128 0.00398182 0.496465
747 16 24065 24049 128.763 140 0.36963 0.496393
748 16 24096 24080 128.756 124 0.260242 0.496367
749 16 24133 24117 128.782 148 0.476668 0.496355
750 16 24164 24148 128.775 124 0.689863 0.496313
751 16 24194 24178 128.764 120 0.553792 0.496387
752 16 24226 24210 128.763 128 0.649896 0.496446
753 16 24256 24240 128.751 120 0.733163 0.496483
754 16 24287 24271 128.745 124 0.433617 0.496505
755 16 24316 24300 128.728 116 1.11075 0.496563
756 16 24350 24334 128.737 136 0.846018 0.496552
757 16 24381 24365 128.731 124 0.00384142 0.496557
758 16 24411 24395 128.72 120 0.514282 0.496632
759 16 24446 24430 128.735 140 0.0093646 0.496599
2025-09-06T18:15:22.427243+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496444
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
760 16 24478 24462 128.734 128 0.244911 0.496444
761 16 24507 24491 128.717 116 0.522814 0.496472
762 16 24539 24523 128.716 128 0.413511 0.496636
763 16 24569 24553 128.704 120 1.00365 0.496691
764 16 24597 24581 128.683 112 0.779107 0.496755
765 16 24627 24611 128.671 120 0.35555 0.496732
766 16 24662 24646 128.686 140 0.310008 0.496748
767 16 24690 24674 128.664 112 0.913338 0.496795
768 16 24721 24705 128.658 124 1.15482 0.496861
769 16 24750 24734 128.642 116 0.00494844 0.496869
770 16 24783 24767 128.646 132 0.788786 0.496894
771 16 24812 24796 128.63 116 0.00416075 0.496924
772 16 24844 24828 128.629 128 1.53599 0.496903
773 16 24881 24865 128.654 148 0.248943 0.496759
774 16 24912 24896 128.648 124 0.419131 0.49676
775 16 24940 24924 128.626 112 0.390904 0.496929
776 16 24970 24954 128.615 120 0.215752 0.496969
777 16 25004 24988 128.625 136 0.16379 0.496896
778 16 25036 25020 128.624 128 0.224807 0.49689
779 16 25067 25051 128.618 124 0.358542 0.496859
2025-09-06T18:15:42.429209+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496927
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
780 16 25099 25083 128.617 128 1.83034 0.496927
781 16 25132 25116 128.621 132 0.414996 0.496949
782 16 25164 25148 128.62 128 0.443258 0.496933
783 16 25194 25178 128.609 120 0.193056 0.497022
784 16 25223 25207 128.593 116 0.503691 0.497087
785 16 25254 25238 128.587 124 0.612914 0.497088
786 16 25284 25268 128.577 120 0.861878 0.497184
787 16 25312 25296 128.555 112 0.815574 0.497204
788 16 25342 25326 128.545 120 0.221453 0.497268
789 16 25372 25356 128.534 120 0.299789 0.497362
790 16 25405 25389 128.538 132 0.00440725 0.497327
791 16 25438 25422 128.542 132 0.432757 0.497305
792 16 25468 25452 128.532 120 0.801785 0.497335
793 16 25501 25485 128.536 132 0.00396724 0.497345
794 16 25533 25517 128.535 128 0.332969 0.497369
795 15 25563 25548 128.53 124 0.244494 0.497404
796 16 25598 25582 128.539 136 1.09812 0.497361
797 16 25630 25614 128.538 128 1.0083 0.497367
798 16 25660 25644 128.528 120 0.00443719 0.497309
799 16 25690 25674 128.517 120 0.193485 0.497376
2025-09-06T18:16:02.431412+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.497351
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
800 16 25722 25706 128.516 128 1.22193 0.497351
801 16 25760 25744 128.545 152 0.170412 0.497321
802 16 25790 25774 128.535 120 0.330782 0.497357
803 16 25821 25805 128.529 124 0.210534 0.49724
804 16 25853 25837 128.528 128 1.52072 0.497341
805 16 25886 25870 128.533 132 0.3297 0.497337
806 16 25912 25896 128.502 104 0.781525 0.497414
807 16 25953 25937 128.546 164 0.357451 0.497305
808 16 25987 25971 128.556 136 0.511665 0.497295
809 16 26018 26002 128.55 124 0.596082 0.497304
810 16 26049 26033 128.544 124 1.28158 0.497302
811 16 26081 26065 128.544 128 0.156761 0.497153
812 16 26114 26098 128.548 132 0.25792 0.497219
813 16 26144 26128 128.537 120 0.140097 0.497305
814 16 26172 26156 128.517 112 0.420569 0.497361
815 16 26209 26193 128.541 148 1.65198 0.497279
816 16 26241 26225 128.54 128 0.281951 0.497258
817 16 26273 26257 128.539 128 0.420018 0.497234
818 16 26307 26291 128.549 136 0.187523 0.497253
819 16 26337 26321 128.538 120 0.490617 0.497229
2025-09-06T18:16:22.433435+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.497075
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
820 16 26377 26361 128.576 160 0.00392565 0.497075
821 16 26410 26394 128.581 132 0.734829 0.497178
822 16 26442 26426 128.58 128 0.00407791 0.497176
823 16 26475 26459 128.584 132 0.40133 0.49711
824 16 26513 26497 128.612 152 0.425043 0.49704
825 16 26547 26531 128.621 136 0.464611 0.496976
826 15 26577 26562 128.616 124 0.937893 0.497032
827 16 26615 26599 128.639 148 0.398004 0.49687
828 16 26641 26625 128.609 104 0.286453 0.496906
829 16 26672 26656 128.604 124 0.907882 0.497087
830 16 26703 26687 128.598 124 1.26354 0.497104
831 16 26732 26716 128.583 116 0.152784 0.497122
832 16 26770 26754 128.611 152 0.355234 0.496969
833 16 26799 26783 128.596 116 0.171364 0.497075
834 16 26829 26813 128.586 120 0.346634 0.497072
835 16 26859 26843 128.575 120 0.217705 0.497142
836 15 26896 26881 128.603 152 0.308229 0.496968
837 16 26927 26911 128.593 120 0.773591 0.497103
838 16 26959 26943 128.592 128 0.00375755 0.497084
839 16 26990 26974 128.587 124 0.203202 0.49697
2025-09-06T18:16:42.436401+0000 min lat: 0.00286085 max lat: 2.58677 avg lat: 0.496851
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
840 16 27027 27011 128.61 148 0.16814 0.496851
841 16 27059 27043 128.609 128 0.00439834 0.496926
842 16 27090 27074 128.604 124 1.64512 0.497005
843 16 27126 27110 128.622 144 0.00400234 0.496932
844 16 27157 27141 128.616 124 0.283174 0.49694
845 16 27190 27174 128.62 132 0.128569 0.496869
846 16 27221 27205 128.615 124 0.199545 0.496968
847 16 27252 27236 128.609 124 1.56657 0.496966
848 16 27283 27267 128.604 124 1.24649 0.496999
849 16 27321 27305 128.632 152 0.637402 0.496872
850 16 27353 27337 128.631 128 0.223792 0.496922
851 16 27385 27369 128.63 128 0.148846 0.496943
852 16 27418 27402 128.634 132 1.18254 0.496925
853 16 27451 27435 128.638 132 0.00994704 0.496933
854 16 27482 27466 128.632 124 0.00976012 0.496957
855 16 27512 27496 128.622 120 0.00493533 0.496998
856 16 27548 27532 128.64 144 0.338676 0.496927
857 16 27576 27560 128.621 112 0.153618 0.496936
858 16 27605 27589 128.606 116 0.803978 0.497038
859 16 27638 27622 128.61 132 0.523701 0.497039
2025-09-06T18:17:02.438696+0000 min lat: 0.00286085 max lat: 2.74757 avg lat: 0.497126
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
860 16 27666 27650 128.591 112 1.12031 0.497126
861 16 27699 27683 128.595 132 1.19636 0.497092
862 16 27728 27712 128.58 116 0.00924344 0.497038
863 16 27759 27743 128.575 124 2.32524 0.497079
864 16 27792 27776 128.579 132 0.00962865 0.497139
865 16 27823 27807 128.573 124 0.22464 0.49715
866 16 27853 27837 128.563 120 0.447585 0.49712
867 16 27885 27869 128.563 128 0.00426165 0.497173
868 16 27917 27901 128.562 128 0.514195 0.497227
869 16 27949 27933 128.561 128 0.192849 0.497186
870 16 27979 27963 128.552 120 1.65747 0.497172
871 16 28014 27998 128.565 140 0.2004 0.497075
872 16 28047 28031 128.569 132 0.394598 0.497161
873 16 28076 28060 128.554 116 0.170698 0.497216
874 16 28109 28093 128.558 132 0.29469 0.497259
875 16 28141 28125 128.558 128 0.993899 0.497255
876 16 28172 28156 128.552 124 0.243069 0.49721
877 16 28203 28187 128.547 124 1.39689 0.497296
878 16 28233 28217 128.537 120 0.530768 0.497293
879 16 28264 28248 128.532 124 0.189211 0.497325
2025-09-06T18:17:22.440688+0000 min lat: 0.00286085 max lat: 2.74757 avg lat: 0.497323
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
880 16 28296 28280 128.532 128 0.16957 0.497323
881 16 28325 28309 128.517 116 0.220706 0.497413
882 16 28354 28338 128.503 116 1.45203 0.49747
883 16 28387 28371 128.507 132 0.322121 0.49751
884 16 28421 28405 128.516 136 0.330126 0.497429
885 16 28452 28436 128.51 124 1.43398 0.497384
886 16 28488 28472 128.528 144 0.00948231 0.497297
887 16 28519 28503 128.523 124 0.27187 0.497184
888 16 28549 28533 128.513 120 0.234242 0.49725
889 16 28579 28563 128.504 120 0.811712 0.497372
890 16 28611 28595 128.503 128 0.631672 0.497492
891 16 28643 28627 128.502 128 0.344119 0.49747
892 16 28675 28659 128.502 128 0.131958 0.49744
893 16 28706 28690 128.497 124 0.382912 0.497336
894 16 28739 28723 128.501 132 0.469578 0.497424
895 16 28774 28758 128.513 140 0.358718 0.497343
896 16 28804 28788 128.504 120 0.7592 0.497435
897 16 28832 28816 128.486 112 1.15076 0.49752
898 16 28865 28849 128.489 132 0.334928 0.497518
899 16 28897 28881 128.489 128 0.227154 0.497494
2025-09-06T18:17:42.442778+0000 min lat: 0.00286085 max lat: 2.74757 avg lat: 0.497484
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
900 12 28925 28913 128.488 128 0.131318 0.497484
Total time run: 900.45
Total reads made: 28925
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 128.491
Average IOPS: 32
Stddev IOPS: 2.64739
Max IOPS: 43
Min IOPS: 26
Average Latency(s): 0.497611
Max latency(s): 2.74757
Min latency(s): 0.00286085
Rados, random read, pool with 32 PGs
[ceph: root@ms-01-01 /]# date
Sat Sep 6 18:20:43 UTC 2025
[ceph: root@ms-01-01 /]# rados bench -p testbench 900 rand
hints = 1
Click to see full run
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 2 2 0 0 0 - 0
1 16 45 29 115.977 116 0.946455 0.334365
2 16 81 65 129.978 144 0.151799 0.402858
3 16 112 96 127.981 124 1.18352 0.432497
4 16 143 127 126.981 124 1.3917 0.435819
5 16 178 162 129.582 140 1.46459 0.445311
6 16 208 192 127.982 120 0.178275 0.45001
7 16 243 227 129.696 140 0.00718082 0.466022
8 15 275 260 129.981 132 1.41221 0.469916
9 16 306 290 128.87 120 0.300853 0.47135
10 16 340 324 129.582 136 0.00488823 0.471236
11 16 368 352 127.982 112 0.909512 0.483933
12 16 402 386 128.649 136 0.00465737 0.477073
13 16 437 421 129.519 140 0.239667 0.478698
14 16 469 453 129.41 128 1.33108 0.479356
15 16 497 481 128.248 112 0.319827 0.484032
16 16 530 514 128.482 132 0.247962 0.483375
17 16 559 543 127.746 116 0.400901 0.487744
18 16 594 578 128.426 140 0.935051 0.483598
19 16 629 613 129.034 140 0.839456 0.487638
2025-09-06T18:21:08.560249+0000 min lat: 0.00179416 max lat: 1.92525 avg lat: 0.484987
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 658 642 128.382 116 0.170675 0.484987
21 16 693 677 128.934 140 1.64293 0.48565
22 16 724 708 128.709 124 0.399857 0.488586
23 16 755 739 128.503 124 0.394626 0.488468
24 16 787 771 128.482 128 0.500534 0.489694
25 16 819 803 128.462 128 1.36508 0.486698
26 16 849 833 128.136 120 1.71311 0.486285
27 16 880 864 127.982 124 0.13139 0.492606
28 16 908 892 127.411 112 0.49139 0.493764
29 16 940 924 127.431 128 1.33473 0.494019
30 16 970 954 127.183 120 0.236683 0.49613
31 16 999 983 126.822 116 0.376773 0.495851
32 16 1029 1013 126.608 120 0.131229 0.496592
33 16 1064 1048 127.013 140 0.721859 0.497289
34 16 1094 1078 126.807 120 0.680327 0.498672
35 16 1124 1108 126.612 120 0.129594 0.49848
36 16 1156 1140 126.65 128 0.260548 0.499325
37 16 1191 1175 127.01 140 0.950264 0.498749
38 16 1226 1210 127.352 140 0.00300055 0.496089
39 16 1262 1246 127.778 144 0.0096846 0.493494
2025-09-06T18:21:28.562582+0000 min lat: 0.00179416 max lat: 2.05151 avg lat: 0.492609
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1299 1283 128.283 148 0.0029702 0.492609
41 16 1327 1311 127.886 112 0.701586 0.493137
42 16 1362 1346 128.174 140 0.215376 0.49327
43 16 1396 1380 128.356 136 0.316092 0.493723
44 16 1424 1408 127.983 112 0.4967 0.495102
45 16 1456 1440 127.983 128 0.868343 0.494754
46 16 1489 1473 128.07 132 0.114139 0.494572
47 16 1524 1508 128.324 140 0.622904 0.493379
48 16 1556 1540 128.317 128 0.395353 0.493841
49 16 1585 1569 128.065 116 1.12993 0.495359
50 16 1618 1602 128.144 132 0.759356 0.495252
51 16 1649 1633 128.062 124 0.20947 0.495789
52 16 1681 1665 128.061 128 0.288825 0.495784
53 16 1713 1697 128.06 128 1.22006 0.494936
54 16 1744 1728 127.984 124 0.271546 0.494275
55 16 1782 1766 128.42 152 2.33718 0.49515
56 16 1813 1797 128.341 124 0.221193 0.49545
57 16 1846 1830 128.405 132 0.408704 0.494118
58 16 1871 1855 127.915 100 0.206133 0.494185
59 16 1905 1889 128.052 136 0.608614 0.495752
2025-09-06T18:21:48.564823+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.496001
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 1940 1924 128.251 140 0.638511 0.496001
61 16 1969 1953 128.05 116 0.988119 0.496579
62 16 2001 1985 128.049 128 1.13147 0.49677
63 16 2041 2025 128.556 160 0.824897 0.495086
64 16 2073 2057 128.547 128 0.00413838 0.495086
65 16 2104 2088 128.477 124 0.940774 0.494608
66 16 2139 2123 128.651 140 0.568543 0.49462
67 16 2171 2155 128.641 128 0.761599 0.494895
68 16 2206 2190 128.808 140 0.352015 0.494522
69 16 2236 2220 128.68 120 0.691905 0.494641
70 16 2269 2253 128.727 132 0.178285 0.493757
71 16 2298 2282 128.548 116 0.209528 0.494817
72 16 2331 2315 128.596 132 0.71842 0.49443
73 16 2367 2351 128.806 144 0.633731 0.494508
74 16 2397 2381 128.687 120 0.724632 0.49424
75 16 2427 2411 128.571 120 0.217372 0.494385
76 16 2461 2445 128.669 136 0.00975976 0.494507
77 16 2497 2481 128.868 144 0.948524 0.493288
78 16 2527 2511 128.754 120 0.474774 0.493999
79 16 2563 2547 128.947 144 0.00296521 0.494124
2025-09-06T18:22:08.566992+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.494351
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 2593 2577 128.835 120 0.00998061 0.494351
81 16 2627 2611 128.923 136 0.26053 0.493034
82 16 2660 2644 128.96 132 0.0041278 0.49355
83 16 2687 2671 128.708 108 0.209411 0.494385
84 16 2718 2702 128.651 124 0.214842 0.493519
85 16 2750 2734 128.644 128 0.991513 0.494759
86 16 2779 2763 128.496 116 0.48483 0.495547
87 16 2816 2800 128.72 148 1.0391 0.494766
88 16 2852 2836 128.894 144 1.33733 0.493983
89 16 2886 2870 128.973 136 0.144908 0.493069
90 16 2920 2904 129.051 136 1.44358 0.492961
91 16 2952 2936 129.04 128 0.00351137 0.492733
92 16 2985 2969 129.072 132 1.68695 0.492663
93 16 3019 3003 129.146 136 0.510286 0.492775
94 16 3053 3037 129.219 136 1.09382 0.493047
95 16 3083 3067 129.122 120 1.17341 0.492773
96 16 3113 3097 129.026 120 0.502812 0.493636
97 16 3147 3131 129.098 136 0.286989 0.492621
98 16 3180 3164 129.128 132 0.00946412 0.492579
99 16 3218 3202 129.358 152 0.747498 0.492664
2025-09-06T18:22:28.569218+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.492763
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3249 3233 129.305 124 0.372276 0.492763
101 16 3283 3267 129.371 136 1.0016 0.492541
102 16 3317 3301 129.436 136 1.08775 0.492317
103 16 3351 3335 129.499 136 0.184754 0.491702
104 16 3381 3365 129.408 120 1.26675 0.492134
105 16 3413 3397 129.394 128 0.0106307 0.492805
106 16 3445 3429 129.381 128 1.04456 0.492575
107 16 3478 3462 129.405 132 0.196917 0.491999
108 16 3509 3493 129.355 124 0.476888 0.492411
109 16 3539 3523 129.269 120 0.557053 0.492509
110 16 3571 3555 129.258 128 0.277216 0.491915
111 16 3604 3588 129.282 132 0.127825 0.492271
112 16 3636 3620 129.271 128 1.58018 0.492753
113 16 3664 3648 129.118 112 0.591644 0.493138
114 16 3693 3677 129.002 116 1.497 0.493721
115 16 3726 3710 129.028 132 0.1885 0.4936
116 16 3760 3744 129.088 136 1.38589 0.493821
117 16 3789 3773 128.976 116 0.882342 0.494318
118 16 3823 3807 129.036 136 0.00511954 0.494104
119 16 3855 3839 129.027 128 0.0095501 0.494178
2025-09-06T18:22:48.571281+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.494404
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 3882 3866 128.852 108 0.284555 0.494404
121 16 3912 3896 128.779 120 0.290215 0.494104
122 16 3945 3929 128.805 132 0.344239 0.494458
123 16 3980 3964 128.896 140 0.0046088 0.494267
124 16 4013 3997 128.921 132 1.23598 0.494125
125 16 4045 4029 128.913 128 1.40003 0.494237
126 16 4077 4061 128.906 128 0.00367598 0.494337
127 16 4109 4093 128.899 128 0.894082 0.494527
128 16 4142 4126 128.923 132 0.559329 0.494463
129 16 4175 4159 128.946 132 0.349044 0.493989
130 16 4208 4192 128.97 132 0.145473 0.49432
131 16 4240 4224 128.962 128 0.956884 0.494224
132 16 4271 4255 128.925 124 1.52672 0.494109
133 16 4303 4287 128.917 128 1.66106 0.493823
134 16 4333 4317 128.851 120 0.34692 0.494447
135 16 4364 4348 128.815 124 0.157463 0.494363
136 16 4394 4378 128.75 120 1.23643 0.495595
137 16 4422 4406 128.628 112 0.00419456 0.49581
138 16 4461 4445 128.826 156 0.383102 0.495293
139 16 4492 4476 128.791 124 0.205475 0.495058
2025-09-06T18:23:08.573442+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.495406
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 4523 4507 128.757 124 1.30392 0.495406
141 16 4556 4540 128.78 132 0.133783 0.495038
142 16 4587 4571 128.746 124 0.426306 0.495213
143 16 4616 4600 128.657 116 0.171095 0.495771
144 16 4644 4628 128.541 112 0.762945 0.496191
145 16 4676 4660 128.537 128 0.516176 0.496139
146 15 4711 4696 128.643 144 0.233451 0.495784
147 16 4742 4726 128.584 120 0.154194 0.495735
148 16 4774 4758 128.58 128 0.147627 0.495399
149 16 4803 4787 128.495 116 0.158759 0.495721
150 16 4837 4821 128.545 136 1.72453 0.496043
151 16 4867 4851 128.489 120 0.825901 0.496556
152 16 4901 4885 128.538 136 1.39913 0.496193
153 16 4934 4918 128.56 132 0.146294 0.496083
154 16 4965 4949 128.531 124 0.507093 0.495363
155 16 4999 4983 128.579 136 0.16174 0.495709
156 16 5030 5014 128.549 124 0.139627 0.495535
157 16 5063 5047 128.571 132 0.571443 0.496089
158 16 5095 5079 128.568 128 0.347691 0.49633
159 16 5126 5110 128.539 124 0.253666 0.496215
2025-09-06T18:23:28.575737+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.49617
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 5160 5144 128.585 136 1.08115 0.49617
161 16 5190 5174 128.532 120 0.501026 0.496242
162 16 5223 5207 128.553 132 0.751707 0.496462
163 16 5258 5242 128.623 140 0.00969016 0.495906
164 16 5288 5272 128.571 120 0.381706 0.495999
165 16 5324 5308 128.664 144 0.169395 0.495791
166 16 5353 5337 128.588 116 0.177165 0.495859
167 16 5382 5366 128.512 116 0.177833 0.496177
168 16 5413 5397 128.485 124 1.11535 0.496541
169 16 5444 5428 128.459 124 0.12443 0.496446
170 16 5478 5462 128.503 136 0.19199 0.495887
171 16 5507 5491 128.43 116 1.97573 0.496706
172 16 5539 5523 128.427 128 1.11569 0.496987
173 16 5572 5556 128.448 132 1.22677 0.496703
174 16 5601 5585 128.376 116 0.189057 0.496661
175 16 5634 5618 128.397 132 0.118491 0.496933
176 16 5666 5650 128.394 128 0.752882 0.496778
177 16 5698 5682 128.392 128 1.61013 0.497023
178 16 5732 5716 128.435 136 0.101837 0.496795
179 16 5764 5748 128.432 128 0.00959418 0.496393
2025-09-06T18:23:48.577846+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.496385
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 5798 5782 128.474 136 1.61284 0.496385
181 16 5826 5810 128.383 112 0.262099 0.496502
182 16 5863 5847 128.491 148 0.891377 0.496596
183 16 5893 5877 128.444 120 0.0094383 0.496525
184 16 5925 5909 128.442 128 0.00944839 0.496628
185 16 5955 5939 128.396 120 0.663242 0.496726
186 16 5986 5970 128.373 124 0.516342 0.4968
187 16 6019 6003 128.392 132 0.179988 0.496856
188 16 6049 6033 128.347 120 1.67309 0.497213
189 16 6082 6066 128.366 132 0.196339 0.49714
190 16 6110 6094 128.28 112 1.55856 0.497167
191 16 6139 6123 128.216 116 0.156225 0.497232
192 16 6179 6163 128.381 160 0.00307982 0.497211
193 16 6210 6194 128.358 124 0.10009 0.496853
194 16 6239 6223 128.295 116 0.00961365 0.497516
195 16 6268 6252 128.231 116 0.00969469 0.497593
196 16 6303 6287 128.291 140 0.529988 0.497639
197 16 6336 6320 128.31 132 0.850143 0.497468
198 16 6368 6352 128.309 128 0.570711 0.497475
199 16 6401 6385 128.327 132 0.877029 0.497499
2025-09-06T18:24:08.580258+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.497449
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 6432 6416 128.305 124 0.897945 0.497449
201 16 6463 6447 128.284 124 0.800495 0.497522
202 16 6491 6475 128.203 112 0.220941 0.497713
203 16 6526 6510 128.261 140 0.169825 0.497401
204 16 6558 6542 128.26 128 0.264377 0.497628
205 16 6594 6578 128.337 144 0.236943 0.497295
206 16 6629 6613 128.393 140 0.199815 0.497162
207 16 6661 6645 128.391 128 1.4958 0.497038
208 16 6693 6677 128.389 128 0.737803 0.497065
209 16 6723 6707 128.349 120 0.29945 0.49744
210 16 6754 6738 128.328 124 0.231791 0.497284
211 16 6790 6774 128.402 144 0.00460067 0.497211
212 16 6820 6804 128.363 120 1.25609 0.497399
213 16 6850 6834 128.323 120 0.00418781 0.497331
214 16 6886 6870 128.397 144 0.170761 0.497064
215 16 6916 6900 128.357 120 0.121822 0.496989
216 16 6948 6932 128.356 128 1.56089 0.497002
217 16 6982 6966 128.391 136 0.43261 0.496975
218 16 7016 7000 128.426 136 0.56955 0.496696
219 16 7046 7030 128.387 120 0.552879 0.497116
2025-09-06T18:24:28.582424+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.497387
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 7075 7059 128.331 116 0.170507 0.497387
221 16 7104 7088 128.275 116 1.32798 0.497532
222 16 7133 7117 128.22 116 0.348238 0.497664
223 16 7165 7149 128.219 128 1.11767 0.498058
224 16 7194 7178 128.164 116 0.511028 0.497928
225 16 7225 7209 128.146 124 1.58515 0.498057
226 16 7255 7239 128.109 120 1.57742 0.498154
227 16 7289 7273 128.144 136 0.159036 0.497763
228 16 7321 7305 128.143 128 1.49895 0.497809
229 16 7354 7338 128.16 132 0.196115 0.497368
230 16 7385 7369 128.142 124 1.5907 0.497826
231 16 7427 7411 128.315 168 1.06215 0.497535
232 16 7455 7439 128.244 112 1.33585 0.497895
233 16 7486 7470 128.226 124 0.201014 0.497985
234 16 7517 7501 128.208 124 0.419794 0.497977
235 16 7548 7532 128.19 124 0.642755 0.498122
236 16 7577 7561 128.138 116 0.447203 0.498227
237 16 7608 7592 128.121 124 0.505713 0.498515
238 16 7644 7628 128.187 144 1.37865 0.498076
239 16 7672 7656 128.12 112 0.215345 0.498351
2025-09-06T18:24:48.584253+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498413
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 7704 7688 128.119 128 0.44655 0.498413
241 16 7736 7720 128.118 128 0.0095287 0.49834
242 16 7770 7754 128.151 136 0.312365 0.498192
243 16 7799 7783 128.101 116 0.186793 0.498197
244 16 7831 7815 128.1 128 0.00976769 0.498387
245 16 7866 7850 128.149 140 0.744113 0.498367
246 16 7898 7882 128.148 128 1.09274 0.498359
247 16 7928 7912 128.115 120 0.771925 0.498289
248 16 7958 7942 128.082 120 0.185194 0.498479
249 16 7991 7975 128.098 132 0.149698 0.498665
250 16 8020 8004 128.05 116 0.145959 0.498522
251 16 8056 8040 128.113 144 1.39841 0.498289
252 16 8091 8075 128.16 140 0.341722 0.498246
253 16 8123 8107 128.16 128 1.29678 0.498181
254 16 8152 8136 128.112 116 0.271019 0.498384
255 16 8181 8165 128.064 116 0.937672 0.498516
256 16 8216 8200 128.111 140 0.444886 0.49827
257 16 8250 8234 128.141 136 0.380401 0.498106
258 16 8287 8271 128.218 148 0.0045436 0.497991
259 16 8317 8301 128.186 120 0.611223 0.49803
2025-09-06T18:25:08.586493+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498067
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 8350 8334 128.201 132 0.0101976 0.498067
261 16 8382 8366 128.2 128 1.04709 0.498295
262 16 8409 8393 128.123 108 1.27154 0.498406
263 16 8443 8427 128.153 136 0.241117 0.498411
264 16 8472 8456 128.107 116 0.170082 0.498369
265 16 8506 8490 128.137 136 0.00994715 0.498201
266 16 8539 8523 128.151 132 0.715941 0.49844
267 16 8574 8558 128.195 140 0.140435 0.498149
268 16 8606 8590 128.195 128 1.08167 0.49798
269 15 8636 8621 128.179 124 0.168536 0.498029
270 16 8670 8654 128.193 132 0.377636 0.498331
271 16 8700 8684 128.163 120 0.304312 0.498187
272 16 8734 8718 128.192 136 0.855053 0.498077
273 16 8764 8748 128.162 120 1.49923 0.498309
274 16 8795 8779 128.146 124 1.07114 0.498403
275 16 8825 8809 128.117 120 1.33039 0.498513
276 16 8856 8840 128.102 124 1.0153 0.498665
277 16 8888 8872 128.101 128 0.266581 0.498333
278 16 8922 8906 128.13 136 0.150895 0.498368
279 16 8954 8938 128.129 128 0.00399762 0.49828
2025-09-06T18:25:28.588706+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498103
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 8987 8971 128.143 132 0.00448132 0.498103
281 16 9019 9003 128.142 128 0.202703 0.49796
282 16 9047 9031 128.085 112 0.194229 0.498191
283 16 9075 9059 128.028 112 0.382489 0.498693
284 16 9107 9091 128.028 128 0.41276 0.498735
285 16 9145 9129 128.112 152 0.3976 0.498416
286 16 9173 9157 128.056 112 0.00943455 0.498539
287 16 9202 9186 128.014 116 0.308213 0.498759
288 16 9230 9214 127.958 112 0.8936 0.499062
289 16 9263 9247 127.972 132 1.25778 0.498924
290 16 9293 9277 127.944 120 0.255986 0.498873
291 16 9328 9312 127.986 140 0.322974 0.499022
292 16 9358 9342 127.958 120 0.0105642 0.498938
293 16 9389 9373 127.945 124 0.318374 0.499173
294 16 9420 9404 127.931 124 0.121503 0.498982
295 16 9448 9432 127.877 112 0.167404 0.499404
296 16 9482 9466 127.905 136 0.454141 0.499127
297 15 9515 9500 127.932 136 0.393165 0.499284
298 16 9543 9527 127.865 108 0.267608 0.499356
299 16 9578 9562 127.906 140 0.934571 0.499439
2025-09-06T18:25:48.590746+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499393
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 9609 9593 127.892 124 0.265582 0.499393
301 16 9639 9623 127.866 120 0.511249 0.499411
302 16 9669 9653 127.84 120 0.19636 0.499404
303 16 9702 9686 127.854 132 0.00399089 0.499379
304 16 9738 9722 127.907 144 0.00493717 0.49907
305 16 9771 9755 127.92 132 0.581672 0.499197
306 16 9802 9786 127.907 124 1.30593 0.499292
307 16 9837 9821 127.947 140 0.142791 0.499211
308 16 9867 9851 127.921 120 0.183639 0.499245
309 16 9900 9884 127.934 132 1.25266 0.499281
310 16 9929 9913 127.896 116 0.865585 0.499464
311 16 9958 9942 127.857 116 0.00999793 0.499492
312 16 9991 9975 127.87 132 0.0094562 0.499456
313 16 10020 10004 127.833 116 0.370611 0.499639
314 16 10050 10034 127.807 120 1.26844 0.499674
315 16 10085 10069 127.846 140 0.00942776 0.499424
316 16 10115 10099 127.821 120 0.2604 0.49963
317 16 10150 10134 127.86 140 0.554266 0.499397
318 16 10180 10164 127.835 120 0.42475 0.499374
319 16 10214 10198 127.86 136 0.527648 0.49932
2025-09-06T18:26:08.593006+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499472
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 10246 10230 127.861 128 0.204159 0.499472
321 16 10277 10261 127.849 124 0.951069 0.499617
322 16 10312 10296 127.886 140 0.179124 0.49949
323 16 10348 10332 127.936 144 0.84076 0.49935
324 16 10383 10367 127.973 140 0.403396 0.498935
325 16 10417 10401 127.998 136 0.354131 0.498854
326 16 10446 10430 127.961 116 1.58113 0.4989
327 16 10477 10461 127.949 124 1.33911 0.499134
328 16 10507 10491 127.925 120 0.175309 0.499179
329 16 10539 10523 127.925 128 0.488011 0.49916
330 16 10573 10557 127.949 136 0.163388 0.49913
331 16 10608 10592 127.986 140 0.446794 0.499232
332 16 10638 10622 127.962 120 0.197402 0.499172
333 16 10670 10654 127.962 128 0.367944 0.49916
334 16 10708 10692 128.034 152 0.231909 0.49871
335 16 10739 10723 128.022 124 0.141185 0.498821
336 16 10778 10762 128.105 156 0.00940077 0.49852
337 16 10810 10794 128.104 128 0.261648 0.498504
338 16 10847 10831 128.163 148 0.124261 0.498099
339 16 10876 10860 128.127 116 0.164636 0.497995
2025-09-06T18:26:28.595364+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498551
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 15 10910 10895 128.162 140 0.614686 0.498551
341 16 10942 10926 128.15 124 1.04623 0.498454
342 16 10975 10959 128.161 132 0.00949285 0.498393
343 16 11008 10992 128.172 132 0.308079 0.498272
344 16 11038 11022 128.149 120 1.04958 0.498501
345 16 11071 11055 128.16 132 1.19549 0.498422
346 16 11106 11090 128.194 140 0.14117 0.498012
347 16 11135 11119 128.159 116 0.269025 0.498137
348 16 11170 11154 128.193 140 0.233673 0.498077
349 15 11197 11182 128.146 112 1.96743 0.498424
350 16 11233 11217 128.18 140 0.239946 0.498095
351 16 11263 11247 128.157 120 0.141775 0.498139
352 16 11292 11276 128.122 116 0.329235 0.498583
353 16 11324 11308 128.122 128 0.564327 0.498772
354 16 11356 11340 128.121 128 0.245307 0.498601
355 16 11385 11369 128.087 116 0.36941 0.498652
356 16 11418 11402 128.098 132 0.412727 0.498658
357 16 11448 11432 128.075 120 0.912643 0.498826
358 16 11480 11464 128.075 128 0.122428 0.498552
359 16 11511 11495 128.064 124 0.417578 0.498734
2025-09-06T18:26:48.597498+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498608
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 11542 11526 128.052 124 0.00469837 0.498608
361 16 11573 11557 128.041 124 0.277154 0.498769
362 16 11603 11587 128.019 120 0.371347 0.498714
363 16 11634 11618 128.008 124 0.448797 0.498882
364 16 11665 11649 127.997 124 0.297185 0.498912
365 16 11695 11679 127.975 120 0.21292 0.49906
366 16 11727 11711 127.975 128 0.187083 0.499097
367 16 11758 11742 127.964 124 0.297044 0.499262
368 16 11792 11776 127.986 136 0.434425 0.499144
369 16 11822 11806 127.964 120 0.28158 0.499103
370 16 11855 11839 127.975 132 0.547389 0.499218
371 16 11886 11870 127.964 124 0.460247 0.499317
372 16 11917 11901 127.954 124 0.251769 0.499386
373 16 11951 11935 127.975 136 0.224885 0.49912
374 16 11985 11969 127.996 136 0.451716 0.499142
375 16 12014 11998 127.964 116 0.265198 0.499209
376 16 12045 12029 127.954 124 1.11776 0.499352
377 16 12076 12060 127.943 124 0.319043 0.499384
378 16 12108 12092 127.943 128 0.511691 0.499345
379 16 12138 12122 127.922 120 0.683308 0.499455
2025-09-06T18:27:08.599777+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499403
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 12172 12156 127.944 136 0.00950364 0.499403
381 16 12208 12192 127.986 144 0.640739 0.499298
382 16 12238 12222 127.965 120 0.696683 0.499338
383 16 12269 12253 127.954 124 0.743393 0.499362
384 16 12300 12284 127.944 124 0.129444 0.499304
385 16 12331 12315 127.934 124 0.21787 0.49913
386 16 12363 12347 127.934 128 0.00414129 0.499346
387 16 12397 12381 127.955 136 0.00399159 0.499033
388 16 12436 12420 128.027 156 1.67072 0.498726
389 16 12468 12452 128.027 128 0.219336 0.49861
390 16 12500 12484 128.027 128 0.168562 0.498793
391 16 12530 12514 128.006 120 0.211912 0.499008
392 16 12564 12548 128.027 136 1.63693 0.498913
393 16 12594 12578 128.006 120 0.135768 0.498844
394 16 12627 12611 128.016 132 0.00404634 0.498997
395 16 12657 12641 127.996 120 0.383596 0.498998
396 16 12688 12672 127.986 124 0.345043 0.499044
397 16 12722 12706 128.006 136 0.00957307 0.49891
398 16 12752 12736 127.986 120 0.84463 0.499185
399 16 12783 12767 127.976 124 0.13878 0.499246
2025-09-06T18:27:28.601805+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499228
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 12813 12797 127.956 120 0.188823 0.499228
401 16 12846 12830 127.966 132 0.657651 0.499296
402 16 12876 12860 127.946 120 0.163316 0.499298
403 16 12906 12890 127.926 120 0.628978 0.499375
404 16 12935 12919 127.897 116 0.516876 0.499557
405 16 12969 12953 127.917 136 0.0104304 0.499375
406 16 13003 12987 127.937 136 0.297675 0.499425
407 16 13036 13020 127.946 132 0.00389872 0.499382
408 16 13069 13053 127.956 132 0.00997506 0.499302
409 16 13100 13084 127.947 124 1.23969 0.499421
410 16 13136 13120 127.986 144 0.235194 0.499283
411 16 13170 13154 128.005 136 0.132806 0.499113
412 16 13202 13186 128.005 128 0.778281 0.499111
413 16 13234 13218 128.005 128 0.152167 0.499098
414 16 13268 13252 128.024 136 0.294547 0.499115
415 16 13300 13284 128.024 128 0.473965 0.499083
416 16 13330 13314 128.005 120 1.29628 0.499049
417 16 13360 13344 127.986 120 1.59456 0.499016
418 16 13396 13380 128.024 144 0.487929 0.499193
419 16 13423 13407 127.976 108 1.14104 0.499188
2025-09-06T18:27:48.603881+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499253
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 13453 13437 127.957 120 0.00983536 0.499253
421 16 13488 13472 127.986 140 0.0036741 0.499188
422 16 13520 13504 127.986 128 0.71685 0.499059
423 16 13552 13536 127.986 128 1.31663 0.49923
424 16 13582 13566 127.967 120 1.26733 0.49928
425 16 13614 13598 127.967 128 0.251967 0.499233
426 16 13649 13633 127.995 140 0.163331 0.499073
427 16 13676 13660 127.948 108 0.231181 0.498919
428 16 13708 13692 127.948 128 0.304098 0.499214
429 16 13745 13729 127.995 148 0.0096782 0.499198
430 16 13774 13758 127.967 116 0.16486 0.499248
431 16 13803 13787 127.939 116 0.271719 0.499244
432 16 13838 13822 127.967 140 0.240628 0.4991
433 16 13866 13850 127.93 112 0.202561 0.499172
434 16 13899 13883 127.94 132 0.555803 0.499192
435 16 13932 13916 127.949 132 0.341158 0.499307
436 16 13960 13944 127.912 112 0.157427 0.499223
437 16 13993 13977 127.922 132 0.340364 0.49903
438 16 14023 14007 127.904 120 0.186594 0.499304
439 16 14056 14040 127.913 132 0.53627 0.499338
2025-09-06T18:28:08.606171+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499297
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 14095 14079 127.977 156 1.04972 0.499297
441 16 14128 14112 127.986 132 0.250034 0.499277
442 16 14158 14142 127.968 120 1.13538 0.499214
443 16 14192 14176 127.986 136 0.0101212 0.499129
444 16 14223 14207 127.977 124 0.808493 0.499313
445 16 14261 14245 128.031 152 0.910601 0.499082
446 16 14296 14280 128.058 140 1.19419 0.499058
447 16 14326 14310 128.039 120 0.361734 0.499009
448 16 14356 14340 128.022 120 1.47293 0.499104
449 16 14384 14368 127.986 112 0.276342 0.499098
450 16 14419 14403 128.012 140 0.463872 0.499201
451 16 14453 14437 128.03 136 1.03102 0.499199
452 16 14481 14465 127.995 112 0.999764 0.499228
453 16 14512 14496 127.986 124 0.207597 0.499302
454 16 14539 14523 127.942 108 1.20829 0.499473
455 16 14576 14560 127.986 148 0.0915569 0.499287
456 16 14607 14591 127.977 124 0.21471 0.499308
457 16 14640 14624 127.986 132 0.322493 0.49913
458 16 14667 14651 127.942 108 0.207927 0.499287
459 16 14700 14684 127.951 132 0.00461358 0.499235
2025-09-06T18:28:28.608446+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.499353
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 14733 14717 127.96 132 0.48331 0.499353
461 16 14765 14749 127.96 128 0.583011 0.49928
462 16 14801 14785 127.994 144 0.00314487 0.499246
463 16 14834 14818 128.003 132 0.00447125 0.4992
464 16 14868 14852 128.02 136 0.607294 0.499118
465 16 14903 14887 128.046 140 0.265962 0.498998
466 16 14935 14919 128.046 128 0.179967 0.499005
467 16 14968 14952 128.054 132 0.00427903 0.498986
468 16 14998 14982 128.037 120 0.181298 0.499018
469 16 15029 15013 128.028 124 0.132746 0.498882
470 16 15063 15047 128.045 136 1.77387 0.498882
471 16 15096 15080 128.054 132 0.103948 0.498864
472 16 15125 15109 128.028 116 0.158716 0.498938
473 16 15154 15138 128.003 116 0.35774 0.498838
474 16 15184 15168 127.986 120 0.17648 0.499046
475 16 15217 15201 127.994 132 0.240458 0.498823
476 16 15250 15234 128.003 132 1.96903 0.499099
477 16 15283 15267 128.011 132 0.376752 0.498993
478 16 15315 15299 128.011 128 0.338292 0.49895
479 16 15345 15329 127.994 120 0.387207 0.498798
2025-09-06T18:28:48.610480+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498867
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 15377 15361 127.994 128 0.215623 0.498867
481 16 15412 15396 128.019 140 0.36446 0.498951
482 15 15446 15431 128.044 140 0.00948299 0.499021
483 16 15482 15466 128.069 140 0.341337 0.498909
484 16 15512 15496 128.052 120 0.235792 0.499034
485 16 15543 15527 128.044 124 0.69055 0.499051
486 16 15572 15556 128.019 116 0.160791 0.499112
487 16 15605 15589 128.027 132 0.501422 0.499121
488 16 15641 15625 128.06 144 1.01104 0.498938
489 15 15672 15657 128.059 128 1.81597 0.49895
490 16 15702 15686 128.035 116 0.205763 0.499069
491 16 15733 15717 128.027 124 0.284103 0.499165
492 16 15769 15753 128.059 144 0.00945749 0.498934
493 16 15806 15790 128.099 148 0.814693 0.498897
494 16 15839 15823 128.107 132 0.343132 0.498828
495 16 15870 15854 128.099 124 0.00992425 0.498864
496 16 15901 15885 128.091 124 0.884654 0.498903
497 16 15932 15916 128.082 124 0.497856 0.498763
498 16 15966 15950 128.098 136 0.161133 0.49867
499 16 16000 15984 128.114 136 0.25807 0.498577
2025-09-06T18:29:08.612927+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498654
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 16030 16014 128.098 120 1.88328 0.498654
501 16 16063 16047 128.106 132 0.00955982 0.498913
502 16 16092 16076 128.081 116 0.605273 0.498912
503 16 16122 16106 128.065 120 0.00423439 0.49886
504 16 16158 16142 128.097 144 0.245854 0.498815
505 16 16189 16173 128.089 124 0.380209 0.498838
506 16 16224 16208 128.112 140 0.596863 0.498791
507 16 16255 16239 128.104 124 0.159978 0.4988
508 16 16285 16269 128.088 120 0.526997 0.498869
509 16 16320 16304 128.112 140 0.258843 0.498899
510 16 16353 16337 128.119 132 1.15565 0.498675
511 16 16382 16366 128.095 116 0.00555467 0.498588
512 16 16416 16400 128.111 136 0.160095 0.498614
513 16 16445 16429 128.087 116 0.251778 0.498571
514 16 16477 16461 128.087 128 0.479525 0.498749
515 16 16507 16491 128.071 120 0.139536 0.498782
516 15 16537 16522 128.063 124 0.0053727 0.499071
517 16 16578 16562 128.125 160 0.73311 0.498812
518 16 16609 16593 128.117 124 1.10743 0.498809
519 16 16640 16624 128.109 124 0.961275 0.498904
2025-09-06T18:29:28.615254+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.4989
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 16667 16651 128.07 108 1.19303 0.4989
521 16 16701 16685 128.086 136 0.17635 0.498787
522 16 16740 16724 128.139 156 0.336797 0.498659
523 16 16772 16756 128.139 128 0.0039458 0.498658
524 16 16804 16788 128.138 128 0.00398655 0.498685
525 16 16839 16823 128.161 140 1.57302 0.498638
526 16 16867 16851 128.13 112 1.49185 0.498578
527 16 16900 16884 128.138 132 1.93699 0.498574
528 16 16938 16922 128.183 152 0.786089 0.498526
529 16 16971 16955 128.19 132 0.989095 0.498551
530 16 17000 16984 128.167 116 0.237529 0.498528
531 16 17037 17021 128.204 148 0.750842 0.498512
532 16 17071 17055 128.219 136 0.521145 0.498393
533 16 17101 17085 128.203 120 0.721035 0.498481
534 16 17137 17121 128.233 144 0.719929 0.498413
535 16 17171 17155 128.247 136 0.850239 0.498371
536 16 17210 17194 128.299 156 0.354278 0.498126
537 16 17242 17226 128.299 128 0.721432 0.498143
538 16 17272 17256 128.283 120 1.16379 0.498125
539 16 17309 17293 128.32 148 1.17337 0.498027
2025-09-06T18:29:48.617390+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.497789
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 17342 17326 128.326 132 0.416346 0.497789
541 16 17371 17355 128.304 116 1.70174 0.498046
542 16 17407 17391 128.333 144 0.267477 0.498024
543 16 17437 17421 128.317 120 0.410465 0.497982
544 16 17467 17451 128.302 120 0.523432 0.498179
545 16 17498 17482 128.294 124 0.868987 0.498095
546 16 17527 17511 128.271 116 0.286386 0.498133
547 16 17556 17540 128.249 116 0.415641 0.498187
548 16 17586 17570 128.234 120 0.214532 0.498338
549 16 17624 17608 128.277 152 0.185196 0.498253
550 16 17657 17641 128.284 132 0.166757 0.498041
551 16 17687 17671 128.269 120 0.305845 0.498131
552 16 17717 17701 128.254 120 0.252178 0.49813
553 16 17750 17734 128.261 132 0.200912 0.498256
554 16 17783 17767 128.267 132 0.524836 0.498312
555 16 17813 17797 128.252 120 0.155246 0.498354
556 16 17841 17825 128.223 112 0.97922 0.498394
557 16 17874 17858 128.23 132 1.01854 0.498344
558 16 17906 17890 128.23 128 0.609245 0.498393
559 16 17937 17921 128.222 124 1.39075 0.498448
2025-09-06T18:30:08.619595+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498231
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 15 17971 17956 128.243 140 0.51043 0.498231
561 16 18005 17989 128.25 132 0.587466 0.498307
562 16 18036 18020 128.242 124 1.04433 0.498354
563 16 18068 18052 128.242 128 0.389894 0.498337
564 16 18100 18084 128.241 128 1.32011 0.498308
565 16 18131 18115 128.234 124 0.437961 0.498223
566 16 18163 18147 128.233 128 0.205272 0.498271
567 16 18198 18182 128.254 140 0.304688 0.498214
568 16 18226 18210 128.225 112 0.426578 0.498234
569 16 18257 18241 128.218 124 0.241116 0.498225
570 16 18293 18277 128.245 144 0.00453969 0.498237
571 16 18321 18305 128.217 112 1.56114 0.498368
572 16 18350 18334 128.196 116 0.00566492 0.4984
573 16 18384 18368 128.209 136 0.249675 0.498398
574 16 18417 18401 128.216 132 0.194271 0.498352
575 16 18453 18437 128.243 144 0.295329 0.498282
576 16 18487 18471 128.257 136 1.56923 0.49822
577 16 18517 18501 128.242 120 1.73318 0.498215
578 16 18549 18533 128.242 128 0.43951 0.498337
579 16 18579 18563 128.228 120 0.212889 0.498329
2025-09-06T18:30:28.621670+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.498306
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 18612 18596 128.234 132 1.70987 0.498306
581 16 18642 18626 128.22 120 0.00404959 0.498321
582 16 18674 18658 128.219 128 0.260859 0.498406
583 16 18703 18687 128.199 116 0.168831 0.498363
584 16 18733 18717 128.184 120 0.232812 0.498313
585 16 18765 18749 128.184 128 0.257978 0.498367
586 16 18799 18783 128.197 136 1.7413 0.498373
587 16 18836 18820 128.231 148 0.172934 0.498331
588 16 18870 18854 128.244 136 0.222848 0.498373
589 16 18903 18887 128.251 132 0.0107021 0.498279
590 16 18933 18917 128.237 120 0.205191 0.498261
591 16 18973 18957 128.29 160 0.00932829 0.498098
592 16 19005 18989 128.29 128 0.919792 0.498197
593 16 19038 19022 128.296 132 0.193526 0.498153
594 16 19072 19056 128.309 136 0.641516 0.498005
595 16 19105 19089 128.315 132 0.388474 0.497945
596 16 19140 19124 128.335 140 0.265263 0.497798
597 16 19174 19158 128.348 136 0.347531 0.497679
598 16 19205 19189 128.34 124 0.00468298 0.497917
599 16 19234 19218 128.32 116 0.131096 0.497891
2025-09-06T18:30:48.623535+0000 min lat: 0.00154492 max lat: 2.42551 avg lat: 0.497848
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 16 19266 19250 128.319 128 0.20638 0.497848
601 16 19298 19282 128.319 128 0.30307 0.497889
602 16 19328 19312 128.305 120 0.69229 0.498052
603 16 19361 19345 128.311 132 0.966773 0.498166
604 16 19391 19375 128.297 120 0.628106 0.498154
605 16 19424 19408 128.303 132 0.581785 0.498153
606 16 19453 19437 128.283 116 0.245653 0.498085
607 16 19488 19472 128.302 140 0.514374 0.498159
608 16 19516 19500 128.275 112 0.165192 0.498207
609 16 19548 19532 128.275 128 0.18291 0.498116
610 16 19579 19563 128.268 124 0.487579 0.498
611 16 19612 19596 128.274 132 0.350558 0.49813
612 16 19648 19632 128.3 144 1.50344 0.498196
613 15 19680 19665 128.306 132 0.497726 0.498174
614 16 19712 19696 128.299 124 0.387109 0.498147
615 16 19744 19728 128.298 128 0.654766 0.498222
616 16 19777 19761 128.304 132 0.347097 0.498135
617 16 19810 19794 128.31 132 0.00384199 0.498112
618 16 19841 19825 128.303 124 0.00924664 0.49821
619 16 19874 19858 128.309 132 0.317753 0.498164
2025-09-06T18:31:08.625549+0000 min lat: 0.00154492 max lat: 2.50772 avg lat: 0.498133
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
620 16 19904 19888 128.296 120 0.476462 0.498133
621 16 19942 19926 128.334 152 0.742838 0.497973
622 16 19973 19957 128.327 124 1.53709 0.498103
623 16 20003 19987 128.313 120 1.31341 0.498049
624 16 20032 20016 128.294 116 0.370344 0.498175
625 16 20063 20047 128.287 124 0.00488616 0.498207
626 16 20094 20078 128.28 124 0.370828 0.498267
627 16 20124 20108 128.267 120 0.889991 0.498301
628 16 20161 20145 128.298 148 0.407013 0.498236
629 16 20193 20177 128.298 128 0.0045005 0.498117
630 16 20226 20210 128.303 132 0.325955 0.498051
631 16 20257 20241 128.297 124 0.246252 0.498093
632 16 20288 20272 128.29 124 1.5508 0.498095
633 16 20319 20303 128.283 124 0.261586 0.498035
634 16 20347 20331 128.257 112 0.135072 0.498087
635 16 20379 20363 128.257 128 0.224209 0.498247
636 16 20411 20395 128.256 128 0.00320141 0.498235
637 16 20443 20427 128.256 128 0.10677 0.498117
638 16 20475 20459 128.256 128 0.00424576 0.498092
639 16 20513 20497 128.293 152 0.00409179 0.498073
2025-09-06T18:31:28.627678+0000 min lat: 0.00154492 max lat: 2.50772 avg lat: 0.498002
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
640 16 20547 20531 128.305 136 0.217224 0.498002
641 16 20577 20561 128.292 120 0.00941277 0.498134
642 16 20612 20596 128.31 140 0.849566 0.498099
643 15 20643 20628 128.309 128 1.11592 0.49816
644 16 20681 20665 128.34 148 0.303034 0.497779
645 16 20714 20698 128.346 132 0.00453328 0.497771
646 16 20745 20729 128.339 124 0.162846 0.497758
647 16 20776 20760 128.332 124 0.201974 0.497765
648 16 20806 20790 128.319 120 0.270447 0.497867
649 16 20841 20825 128.337 140 0.179079 0.497884
650 16 20871 20855 128.324 120 0.178051 0.497934
651 16 20907 20891 128.348 144 0.707323 0.497984
652 16 20937 20921 128.336 120 0.200974 0.498017
653 16 20969 20953 128.335 128 1.04984 0.497978
654 15 21002 20987 128.347 136 0.332524 0.497963
655 16 21036 21020 128.352 132 0.00292492 0.497902
656 16 21065 21049 128.333 116 0.138455 0.497928
657 16 21094 21078 128.315 116 0.13389 0.498032
658 16 21128 21112 128.326 136 0.350728 0.498019
659 16 21159 21143 128.32 124 0.206449 0.498
2025-09-06T18:31:48.630164+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.498019
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
660 16 21194 21178 128.337 140 0.1939 0.498019
661 16 21228 21212 128.349 136 0.981908 0.498026
662 16 21259 21243 128.342 124 0.60983 0.498078
663 16 21293 21277 128.354 136 0.625971 0.498006
664 16 21328 21312 128.371 140 0.00388638 0.497931
665 16 21359 21343 128.365 124 0.206629 0.497897
666 16 21398 21382 128.406 156 1.15015 0.497804
667 16 21430 21414 128.406 128 0.294225 0.497747
668 16 21461 21445 128.399 124 0.221351 0.497772
669 16 21494 21478 128.404 132 1.10576 0.497805
670 16 21522 21506 128.38 112 1.37844 0.497798
671 16 21560 21544 128.415 152 0.137995 0.497606
672 15 21591 21576 128.414 128 0.00435402 0.497663
673 16 21622 21606 128.402 120 1.67342 0.497771
674 16 21652 21636 128.389 120 0.340688 0.497862
675 16 21689 21673 128.418 148 0.468005 0.497709
676 16 21719 21703 128.406 120 0.985812 0.497749
677 16 21749 21733 128.394 120 0.00462884 0.497805
678 16 21782 21766 128.399 132 0.459022 0.497786
679 16 21817 21801 128.416 140 0.00308075 0.497764
2025-09-06T18:32:08.631990+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497834
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
680 16 21847 21831 128.404 120 0.595554 0.497834
681 16 21883 21867 128.426 144 0.231523 0.497739
682 16 21915 21899 128.426 128 0.26586 0.497685
683 16 21948 21932 128.431 132 1.61398 0.497591
684 16 21978 21962 128.419 120 0.237449 0.497609
685 16 22011 21995 128.424 132 0.317694 0.497474
686 16 22045 22029 128.435 136 2.29904 0.49756
687 16 22078 22062 128.44 132 1.47621 0.497654
688 16 22105 22089 128.41 108 0.333333 0.497719
689 16 22140 22124 128.427 140 0.0102772 0.497606
690 16 22169 22153 128.409 116 0.268968 0.497666
691 16 22202 22186 128.414 132 0.316142 0.497792
692 16 22233 22217 128.408 124 0.531404 0.497859
693 16 22262 22246 128.39 116 0.284596 0.497801
694 16 22300 22284 128.424 152 0.00974954 0.497683
695 16 22331 22315 128.418 124 0.878116 0.497741
696 16 22365 22349 128.428 136 0.00411101 0.497582
697 16 22397 22381 128.428 128 0.67671 0.497591
698 16 22431 22415 128.439 136 0.135415 0.497601
699 16 22466 22450 128.455 140 0.00432351 0.497468
2025-09-06T18:32:28.634440+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.4975
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
700 16 22495 22479 128.437 116 0.00410331 0.4975
701 16 22529 22513 128.448 136 0.00340543 0.497427
702 16 22559 22543 128.436 120 0.584921 0.497398
703 16 22592 22576 128.441 132 0.288112 0.497582
704 16 22624 22608 128.44 128 0.464197 0.497563
705 16 22655 22639 128.434 124 0.249765 0.497639
706 16 22689 22673 128.445 136 1.40698 0.497634
707 16 22723 22707 128.455 136 0.912648 0.497608
708 16 22753 22737 128.443 120 1.14278 0.497607
709 16 22787 22771 128.454 136 0.274615 0.497489
710 16 22820 22804 128.459 132 1.52785 0.497561
711 16 22852 22836 128.458 128 0.143194 0.497524
712 16 22882 22866 128.447 120 0.0108797 0.497514
713 16 22916 22900 128.457 136 0.00969172 0.497626
714 16 22944 22928 128.434 112 0.286761 0.497661
715 16 22981 22965 128.461 148 0.202999 0.497517
716 16 23014 22998 128.466 132 0.985751 0.497543
717 16 23045 23029 128.46 124 0.00401166 0.497519
718 16 23079 23063 128.471 136 1.0441 0.497587
719 16 23109 23093 128.459 120 0.340397 0.497575
2025-09-06T18:32:48.636512+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497535
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
720 16 23140 23124 128.453 124 0.348939 0.497535
721 16 23175 23159 128.469 140 0.00954191 0.497543
722 16 23207 23191 128.468 128 0.141118 0.49741
723 16 23240 23224 128.473 132 0.486684 0.49746
724 16 23272 23256 128.472 128 0.246206 0.497543
725 16 23303 23287 128.466 124 1.46967 0.497569
726 16 23332 23316 128.449 116 0.210593 0.497661
727 16 23361 23345 128.432 116 0.861538 0.497705
728 16 23394 23378 128.436 132 0.247252 0.497672
729 16 23433 23417 128.474 156 0.194794 0.497571
730 16 23463 23447 128.463 120 1.26856 0.497535
731 16 23496 23480 128.467 132 0.348161 0.497462
732 16 23530 23514 128.478 136 0.330198 0.497413
733 16 23561 23545 128.472 124 0.13224 0.497436
734 16 23594 23578 128.476 132 1.69742 0.497318
735 16 23625 23609 128.47 124 0.335757 0.497459
736 16 23654 23638 128.453 116 0.311531 0.497478
737 16 23686 23670 128.453 128 0.334671 0.497574
738 16 23720 23704 128.463 136 0.275606 0.497476
739 16 23753 23737 128.468 132 1.65541 0.497506
2025-09-06T18:33:08.638569+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497496
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
740 16 23783 23767 128.456 120 0.263774 0.497496
741 16 23818 23802 128.472 140 0.110223 0.497351
742 16 23847 23831 128.455 116 0.673877 0.497485
743 16 23878 23862 128.449 124 1.80926 0.497481
744 16 23914 23898 128.47 144 0.1978 0.497421
745 16 23950 23934 128.491 144 0.00333044 0.497357
746 16 23978 23962 128.468 112 1.77292 0.497421
747 16 24011 23995 128.473 132 0.331108 0.497488
748 16 24038 24022 128.446 108 0.245639 0.497612
749 16 24073 24057 128.461 140 0.582493 0.497608
750 16 24102 24086 128.445 116 0.144918 0.497601
751 16 24136 24120 128.455 136 0.00415808 0.497652
752 16 24167 24151 128.449 124 0.291723 0.49767
753 16 24199 24183 128.448 128 1.23694 0.497701
754 16 24228 24212 128.432 116 0.263751 0.497633
755 16 24257 24241 128.415 116 0.52714 0.497729
756 16 24296 24280 128.452 156 0.00388412 0.49759
757 16 24329 24313 128.456 132 0.00396201 0.497566
758 16 24361 24345 128.456 128 0.232989 0.497541
759 16 24395 24379 128.465 136 1.53118 0.49745
2025-09-06T18:33:28.640745+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.49743
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
760 16 24430 24414 128.481 140 0.240701 0.49743
761 16 24465 24449 128.496 140 0.00402037 0.497414
762 16 24492 24476 128.469 108 0.220131 0.497428
763 16 24524 24508 128.468 128 0.00948882 0.497506
764 16 24561 24545 128.494 148 0.438075 0.497412
765 16 24592 24576 128.488 124 0.0106415 0.497388
766 16 24621 24605 128.472 116 0.540332 0.497383
767 16 24650 24634 128.455 116 0.322405 0.497409
768 16 24685 24669 128.47 140 0.671611 0.497507
769 16 24718 24702 128.475 132 0.103512 0.497564
770 16 24750 24734 128.474 128 1.23708 0.497494
771 16 24783 24767 128.479 132 0.00435058 0.497537
772 16 24814 24798 128.473 124 0.188154 0.497412
773 16 24849 24833 128.488 140 0.152624 0.497419
774 16 24882 24866 128.492 132 0.41205 0.497438
775 16 24916 24900 128.502 136 0.545199 0.497474
776 16 24947 24931 128.496 124 0.745055 0.497489
777 16 24980 24964 128.501 132 1.27927 0.497428
778 16 25013 24997 128.505 132 0.375195 0.49743
779 16 25046 25030 128.51 132 0.323529 0.497435
2025-09-06T18:33:48.642586+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497426
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
780 16 25076 25060 128.499 120 0.164765 0.497426
781 16 25107 25091 128.493 124 0.458684 0.497504
782 16 25137 25121 128.482 120 0.0968037 0.497387
783 16 25172 25156 128.497 140 1.36813 0.497426
784 16 25204 25188 128.496 128 0.205059 0.497343
785 16 25234 25218 128.485 120 1.64822 0.497457
786 16 25264 25248 128.475 120 0.570694 0.49751
787 16 25295 25279 128.469 124 0.428104 0.497558
788 16 25329 25313 128.478 136 0.168965 0.497425
789 16 25363 25347 128.488 136 0.236022 0.497381
790 16 25395 25379 128.487 128 1.27283 0.497491
791 16 25426 25410 128.482 124 0.182008 0.497465
792 16 25456 25440 128.471 120 1.41367 0.497491
793 16 25486 25470 128.46 120 1.28479 0.497525
794 16 25522 25506 128.48 144 0.00923601 0.497468
795 16 25550 25534 128.459 112 0.207361 0.497626
796 16 25580 25564 128.448 120 0.553007 0.497641
797 16 25614 25598 128.458 136 0.19253 0.497569
798 16 25644 25628 128.447 120 0.562583 0.497637
799 16 25678 25662 128.457 136 0.294649 0.497624
2025-09-06T18:34:08.644434+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497643
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
800 16 25710 25694 128.456 128 0.154346 0.497643
801 16 25740 25724 128.445 120 0.212227 0.497573
802 16 25768 25752 128.425 112 0.165633 0.497637
803 16 25802 25786 128.434 136 0.429793 0.497596
804 16 25836 25820 128.444 136 0.278389 0.497486
805 16 25867 25851 128.438 124 0.144481 0.49741
806 16 25901 25885 128.448 136 0.196789 0.497476
807 16 25935 25919 128.457 136 0.108006 0.497483
808 16 25967 25951 128.456 128 0.279805 0.497444
809 16 25996 25980 128.441 116 0.314308 0.497563
810 16 26034 26018 128.47 152 0.43413 0.49754
811 16 26067 26051 128.474 132 0.687926 0.497588
812 16 26097 26081 128.464 120 0.625098 0.497609
813 16 26126 26110 128.449 116 0.207972 0.497607
814 16 26159 26143 128.453 132 0.00996344 0.497677
815 16 26189 26173 128.442 120 0.204718 0.497673
816 16 26222 26206 128.447 132 1.44419 0.497655
817 16 26254 26238 128.446 128 1.28227 0.497568
818 16 26284 26268 128.436 120 0.530232 0.497701
819 16 26311 26295 128.411 108 0.229182 0.497704
2025-09-06T18:34:28.646486+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497788
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
820 16 26342 26326 128.406 124 0.010523 0.497788
821 16 26371 26355 128.39 116 0.262661 0.497859
822 16 26402 26386 128.385 124 0.208996 0.497876
823 16 26435 26419 128.389 132 0.588718 0.497848
824 16 26465 26449 128.379 120 0.0042696 0.497911
825 16 26496 26480 128.374 124 0.809936 0.497971
826 16 26532 26516 128.393 144 0.256264 0.497897
827 16 26561 26545 128.378 116 0.37621 0.497934
828 16 26595 26579 128.387 136 0.665159 0.497911
829 16 26623 26607 128.367 112 0.358751 0.497999
830 16 26655 26639 128.367 128 0.320437 0.497905
831 16 26691 26675 128.386 144 0.144223 0.497904
832 16 26725 26709 128.395 136 0.543458 0.497831
833 16 26765 26749 128.433 160 0.00432568 0.49769
834 16 26794 26778 128.418 116 0.720505 0.497772
835 16 26826 26810 128.417 128 0.289933 0.497783
836 16 26857 26841 128.412 124 0.304871 0.497837
837 16 26886 26870 128.397 116 0.137003 0.497788
838 16 26920 26904 128.406 136 0.0095787 0.497795
839 16 26955 26939 128.42 140 0.122133 0.497686
2025-09-06T18:34:48.648507+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497682
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
840 16 26989 26973 128.429 136 0.494781 0.497682
841 16 27020 27004 128.424 124 1.14522 0.497765
842 16 27052 27036 128.423 128 0.121572 0.497678
843 16 27085 27069 128.427 132 1.78955 0.497608
844 16 27117 27101 128.427 128 0.21996 0.497529
845 16 27151 27135 128.436 136 0.152632 0.497576
846 16 27183 27167 128.435 128 0.106062 0.497597
847 16 27214 27198 128.43 124 0.227882 0.497602
848 16 27246 27230 128.429 128 0.379913 0.497665
849 16 27281 27265 128.443 140 0.541272 0.497682
850 16 27312 27296 128.438 124 1.13475 0.497745
851 16 27341 27325 128.423 116 0.147037 0.497718
852 16 27371 27355 128.413 120 0.416081 0.497783
853 16 27401 27385 128.403 120 0.304653 0.497831
854 16 27434 27418 128.408 132 0.91439 0.497827
855 16 27470 27454 128.426 144 0.0043662 0.497798
856 16 27500 27484 128.416 120 0.224426 0.497762
857 16 27531 27515 128.411 124 0.640894 0.497759
858 16 27560 27544 128.396 116 0.509372 0.497834
859 16 27593 27577 128.401 132 0.194444 0.497801
2025-09-06T18:35:08.650646+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497796
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
860 16 27629 27613 128.419 144 1.11625 0.497796
861 16 27659 27643 128.409 120 0.483542 0.497829
862 16 27687 27671 128.39 112 0.124585 0.497871
863 16 27717 27701 128.38 120 0.91981 0.497903
864 16 27749 27733 128.38 128 0.755253 0.497951
865 16 27784 27768 128.393 140 0.345558 0.497904
866 16 27820 27804 128.411 144 0.280595 0.497664
867 16 27853 27837 128.415 132 0.449125 0.497641
868 16 27882 27866 128.401 116 0.276461 0.497804
869 16 27914 27898 128.4 128 0.280504 0.497851
870 16 27945 27929 128.395 124 0.208588 0.497865
871 16 27977 27961 128.395 128 0.322778 0.497902
872 16 28011 27995 128.403 136 0.672454 0.497885
873 16 28040 28024 128.389 116 0.177578 0.497812
874 16 28074 28058 128.398 136 0.399873 0.497765
875 16 28107 28091 128.402 132 0.00451468 0.497827
876 16 28139 28123 128.402 128 1.02589 0.497839
877 16 28173 28157 128.41 136 0.195671 0.497789
878 16 28206 28190 128.414 132 0.497341 0.497751
879 16 28238 28222 128.414 128 0.164813 0.497765
2025-09-06T18:35:28.653034+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497636
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
880 16 28270 28254 128.413 128 0.158231 0.497636
881 16 28300 28284 128.404 120 0.222336 0.497723
882 16 28331 28315 128.399 124 0.485964 0.497786
883 16 28365 28349 128.407 136 0.187136 0.497715
884 16 28398 28382 128.411 132 0.00931851 0.497751
885 16 28431 28415 128.415 132 0.587317 0.497827
886 16 28465 28449 128.424 136 0.263969 0.497762
887 16 28494 28478 128.41 116 0.721625 0.497851
888 16 28530 28514 128.427 144 1.07711 0.497782
889 16 28562 28546 128.427 128 0.187841 0.497687
890 16 28593 28577 128.422 124 0.935895 0.497767
891 16 28627 28611 128.43 136 0.334052 0.497727
892 16 28659 28643 128.43 128 0.358597 0.497715
893 16 28691 28675 128.429 128 0.511986 0.497763
894 16 28722 28706 128.425 124 0.202658 0.497784
895 16 28756 28740 128.433 136 0.00463247 0.497765
896 16 28784 28768 128.415 112 0.27957 0.497785
897 16 28817 28801 128.419 132 0.301228 0.497737
898 16 28850 28834 128.423 132 0.286105 0.497773
899 16 28881 28865 128.418 124 0.36792 0.497791
2025-09-06T18:35:48.655233+0000 min lat: 0.00154492 max lat: 2.54975 avg lat: 0.497846
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
900 11 28903 28892 128.395 108 0.838115 0.497846
Total time run: 900.373
Total reads made: 28903
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 128.405
Average IOPS: 32
Stddev IOPS: 2.54888
Max IOPS: 42
Min IOPS: 25
Average Latency(s): 0.497946
Max latency(s): 2.54975
Min latency(s): 0.00154492
Rados, cleanup
[ceph: root@ms-01-01 /]# time rados -p testbench cleanup
Removed 28947 objects
real 0m8.134s
user 0m0.782s
sys 0m0.628s
RBD, test setup
[ceph: root@ms-01-01 /]# ceph osd pool create rbdbench 32
pool 'rbdbench' created
[ceph: root@ms-01-01 /]# ceph osd pool application enable rbdbench rbd
enabled application 'rbd' on pool 'rbdbench'
[ceph: root@ms-01-01 /]# rbd pool init -p rbdbench
[ceph: root@ms-01-01 /]# rbd create image1 --size 1024 --pool rbdbench
RBD, how to set values
[ceph: root@ms-01-01 /]# rbd help bench
usage: rbd bench [--pool <pool>] [--namespace <namespace>] [--image <image>]
[--io-size <io-size>] [--io-threads <io-threads>]
[--io-total <io-total>] [--io-pattern <io-pattern>]
[--rw-mix-read <rw-mix-read>] --io-type <io-type>
<image-spec>
Simple benchmark.
Positional arguments
<image-spec> image specification
(example: [<pool-name>/[<namespace>/]]<image-name>)
Optional arguments
-p [ --pool ] arg pool name
--namespace arg namespace name
--image arg image name
--io-size arg IO size (in B/K/M/G) (< 4G) [default: 4K]
--io-threads arg ios in flight [default: 16]
--io-total arg total size for IO (in B/K/M/G/T) [default: 1G]
--io-pattern arg IO pattern (rand, seq, or full-seq) [default: seq]
--rw-mix-read arg read proportion in readwrite (<= 100) [default: 50]
--io-type arg IO type (read, write, or readwrite(rw))
RBD, 4k size, random RW, 80% read, 16 parallel IO threads
[ceph: root@ms-01-01 /]# rbd bench \
--pool=rbdbench \
--image image1 \
--io-total 5G \
--io-pattern rand \
--rw-mix-read 80 \
--io-type rw
bench type readwrite read:write=80:20 io_size 4096 io_threads 16 bytes 5368709120 pattern random
[…]
Click tro see full run
[ceph: root@ms-01-01 /]# rbd bench \
> --pool=rbdbench \
> --image image1 \
> --io-total 5G \
> --io-pattern rand \
> --rw-mix-read 80 \
> --io-type rw
bench type readwrite read:write=80:20 io_size 4096 io_threads 16 bytes 5368709120 pattern random
SEC OPS OPS/SEC BYTES/SEC
1 21968 22004.6 86 MiB/s
2 44656 22345.8 87 MiB/s
3 67744 22592.8 88 MiB/s
4 91392 22856.3 89 MiB/s
5 113296 22665.5 89 MiB/s
6 134320 22469 88 MiB/s
7 155408 22149 87 MiB/s
8 177088 21867.4 85 MiB/s
9 198544 21429 84 MiB/s
10 219680 21275.5 83 MiB/s
11 240544 21243.5 83 MiB/s
12 262128 21342.6 83 MiB/s
13 283568 21294.6 83 MiB/s
14 304288 21147.5 83 MiB/s
15 326112 21285.1 83 MiB/s
16 347344 21362.9 83 MiB/s
17 368048 21186.9 83 MiB/s
18 389120 21100.6 82 MiB/s
19 410576 21260.5 83 MiB/s
20 431776 21135.7 83 MiB/s
21 453376 21205.1 83 MiB/s
22 475184 21421.6 84 MiB/s
23 496432 21473.9 84 MiB/s
24 517952 21473.8 84 MiB/s
25 539056 21454.6 84 MiB/s
26 560192 21361.8 83 MiB/s
27 581264 21218.9 83 MiB/s
28 602288 21169.9 83 MiB/s
29 623376 21083.5 82 MiB/s
30 644288 21045.1 82 MiB/s
31 665344 21029.1 82 MiB/s
32 686304 21010.9 82 MiB/s
33 707568 21058.9 82 MiB/s
34 728480 21023.7 82 MiB/s
35 749616 21068.5 82 MiB/s
36 770944 21122.9 83 MiB/s
37 792560 21249.9 83 MiB/s
38 814576 21400.2 84 MiB/s
39 835968 21496.2 84 MiB/s
40 857008 21477 84 MiB/s
41 878208 21451.4 84 MiB/s
42 899392 21365 83 MiB/s
43 920736 21230.7 83 MiB/s
44 942336 21272.3 83 MiB/s
45 964000 21397 84 MiB/s
46 984912 21339.4 83 MiB/s
47 1006160 21352.2 83 MiB/s
48 1027440 21343.7 83 MiB/s
49 1049104 21356.5 83 MiB/s
50 1070288 21256.3 83 MiB/s
51 1091760 21368.2 83 MiB/s
52 1114224 21615.8 84 MiB/s
53 1135280 21566.6 84 MiB/s
54 1156336 21445 84 MiB/s
55 1177696 21484.5 84 MiB/s
56 1198768 21404.5 84 MiB/s
57 1219856 21125.1 83 MiB/s
58 1241360 21214.7 83 MiB/s
59 1262592 21249.9 83 MiB/s
60 1284000 21259.5 83 MiB/s
61 1305008 21246.7 83 MiB/s
elapsed: 61 ops: 1310720 ops/sec: 21388 bytes/sec: 84 MiB/s
read_ops: 1048933 read_ops/sec: 17116.2 read_bytes/sec: 67 MiB/s
write_ops: 261787 write_ops/sec: 4271.78 write_bytes/sec: 17 MiB/s
RBD, celanup
[ceph: root@ms-01-01 /]# rbd remove image1 --pool rbdbench
Removing image: 100% complete...done.
Current state
2025-09-06 late evening
[ceph: root@ms-01-01 /]# ceph health detail
HEALTH_OK
[ceph: root@ms-01-01 /]# ceph -s
cluster:
id: cee8622c-fb45-11ef-9938-5847ca797c06
health: HEALTH_OK
services:
mon: 3 daemons, quorum ms-01-01,ms-01-02,ms-01-03 (age 11d)
mgr: ms-01-01.pnupxa(active, since 11d), standbys: ms-01-02.yuzhvl, ms-01-03.ujufvv
mds: 1/1 daemons up, 1 standby
osd: 10 osds: 10 up (since 8d), 10 in (since 8d)
data:
volumes: 1/1 healthy
pools: 5 pools, 337 pgs
objects: 28 objects, 8.5 MiB
usage: 7.8 GiB used, 18 TiB / 18 TiB avail
pgs: 337 active+clean
[ceph: root@ms-01-01 /]# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 18 TiB 18 TiB 7.8 GiB 7.8 GiB 0.04
TOTAL 18 TiB 18 TiB 7.8 GiB 7.8 GiB 0.04
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
.mgr 1 1 16 MiB 3 48 MiB 0 5.8 TiB
testbench 2 32 0 B 0 0 B 0 5.8 TiB
rbdbench 3 32 9.4 KiB 3 40 KiB 0 5.8 TiB
cephfs.cephfs.meta 4 16 56 KiB 22 260 KiB 0 5.8 TiB
cephfs.cephfs.data 5 256 0 B 0 0 B 0 5.8 TiB
[ceph: root@ms-01-01 /]#
[ceph: root@ms-01-01 /]# ceph orch host ls
HOST ADDR LABELS STATUS
ms-01-01.storage.pcfe.net 192.168.40.151 _admin,mgr,mon,osd
ms-01-02.storage.pcfe.net 192.168.40.152 _admin,mgr,mon,osd
ms-01-03.storage.pcfe.net 192.168.40.153 _admin,mgr,mon,osd
ms-01-04.storage.pcfe.net 192.168.40.154 _admin,osd,mon,mds
ms-01-05.storage.pcfe.net 192.168.40.155 _admin,osd,mon,mds
5 hosts in cluster
and if I add --detail then I also am shown
VENDOR/MODEL CPU RAM HDD SSD NIC
Micro Computer (HK) Tech Limited Venus (Venus Series) 14C/20T 31 GiB - 3/5.…TB 5
Next up
- certs that use a homelab CA or letsencrypt
- RGW with LB
- Simplifying RGW Multi-site Replication with Ceph Dashboard: Introducing the New 4-Step Wizard (Oct 11, 2024)
- Simplyfing Ceph Object Deployments: Production-ready Ceph Object Gateway (RGW) with New Cephadm Features (Dec 28, 2024)
- Automating Ceph Cluster Deployments with Ceph: A Step-by-Step Guide Using Cephadm and Ansible (Part 1) (Dec 27, 2024)
- https://docs.ceph.com/en/squid/cephadm/services/rgw/
- CephFS subvolumes
- move data from old cluster to new one?
- https://www.ibm.com/docs/en/storage-ceph/8.1.0