Hardware changes to my Ceph Nautilus Cluster
Table of Contents
I made some hardware changes to my Ceph Nautilus cluster.
While previously I used one SSD shared between the operating system and Ceph, I did want a clean setup where the OS and Ceph are on separate devices.
Dedicated 120 GB SSD for OS
I bought myself four Crucial BX500 SSD 120GB for about 27€ a piece plus shipping.
Additionally, I recycled 4 HDDs from machines around the lab that I had not booted in a year or more.
Now the storage setup of each F5-422 is as follows;
| bay | type | use |
|---|---|---|
| 1 | 120 GB SATA SSD | Operating System |
| 2 | 500 GB SATA SSD | 3x Ceph BlueStore DB |
| 3 | 1 TB or larger HDD | Ceph BlueStore data |
| 4 | 1 TB or larger HDD | Ceph BlueStore data |
| 5 | 1 TB or larger HDD | Ceph BlueStore data |
The mix of HDD sizes I managed to scrounge in my home-lab plus one 155GiB SSD slice for each of the three OSDs per node show up as follows in the dashboard.
While this wild mix of HDD sizes is not ideal, recycling drives that I already owned is good enough for the home-lab and incurred no addition cost.
Re-install the Nodes
Originally I had the OS (including /boot and /boot/efi) on the 500GB SSD.
While the PV used for the OS I could have moved at runtime, I simply did not want the
hassle of moving the boot bits (cattle, not pets and all that jazz).
Since I have kickstart files in place for the F5-422 nodes, and I do all my configuration with Ansible, it was a lot less hassle for me to re-install each node.
By doing this one machine at at time, I obviously had no downtime on my Ceph.
Steps taken;
- remove OSDs of one node from Cluster, in my case that was 2 per node.
- power node down.
- rearrange storage devices.
- install the node using kickstart
sgdisk --zap-allthe previously used devices (bays 2 through 5) because I also wanted to switch away from me having top set up LVM for Ceph use and that (understandably) only grabs empty devices.- run my configuration playbooks.
- use
ceph-ansible.
repeat with the next node until all nodes done.
Yes, I did wait while the cluster was self healing the removal of OSDs and subsequent addition of more OSDs. But for me that was preferable to interactively spending time on each node to move the OS from one SSD to another.
Used host_vars
Since I already had host specific host_vars, I decided to also test how ceph-ansible deals with a mix of
lvm_volumes: and devices:.
It deals absolutely fine with this.
example of lvm_volumes: (click to expand)
[...]
lvm_volumes:
- data: LV_HDD_bay2_OSD_data
data_vg: VG_HDD_bay2_Ceph
db: LV_SSD_bay1_OSD_db_01
db_vg: VG_SSD_bay1_Ceph
- data: LV_HDD_bay3_OSD_data
data_vg: VG_HDD_bay3_Ceph
db: LV_SSD_bay1_OSD_db_02
db_vg: VG_SSD_bay1_Ceph
[...]
example of devices: (again, click to expand)
[...]
# avoid using half arsed names like sda, sdb, etc
# mapping is:
# bay 1 /dev/disk/by-path/pci-0000:00:12.0-ata-1
# bay 2 /dev/disk/by-path/pci-0000:00:12.0-ata-2
# bay 3 /dev/disk/by-path/pci-0000:03:00.0-ata-1
# bay 4 /dev/disk/by-path/pci-0000:03:00.0-ata-2
# bay 5 /dev/disk/by-path/pci-0000:04:00.0-ata-1
devices:
- /dev/disk/by-path/pci-0000:00:12.0-ata-2
- /dev/disk/by-path/pci-0000:03:00.0-ata-1
- /dev/disk/by-path/pci-0000:03:00.0-ata-2
- /dev/disk/by-path/pci-0000:04:00.0-ata-1
See section OSD Scenario in the ceph-ansible documentation for more information.
This ends up set up by ceph-ansible, as expected, with the 500GB SSD being divided into 3 identical parts (click to expand an lsblk output on one of the nodes).
[root@f5-422-02 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 111.8G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 110.3G 0 part
├─VG_OS_SSD_bay1-LV_root 253:0 0 4G 0 lvm /
├─VG_OS_SSD_bay1-LV_swap 253:1 0 4G 0 lvm [SWAP]
├─VG_OS_SSD_bay1-LV_home 253:2 0 1G 0 lvm /home
├─VG_OS_SSD_bay1-LV_containers 253:3 0 8G 0 lvm /var/lib/containers
├─VG_OS_SSD_bay1-LV_var_crash 253:4 0 15G 0 lvm /var/crash
├─VG_OS_SSD_bay1-LV_var_log 253:5 0 4G 0 lvm /var/log
└─VG_OS_SSD_bay1-LV_var 253:6 0 4G 0 lvm /var
sdb 8:16 0 465.8G 0 disk
├─ceph--block--dbs--a3488d07--c12f[...] 253:8 0 155.3G 0 lvm
│ └─gptTuV-yI5F-CT03-HHIo-GVpi-v5v[...] 253:10 0 155.2G 0 crypt
├─ceph--block--dbs--a3488d07--c12f[...] 253:12 0 155.3G 0 lvm
│ └─lydAIm-olLm-Ml22-p0as-XTe2-5Gp[...] 253:14 0 155.2G 0 crypt
└─ceph--block--dbs--a3488d07--c12f[...] 253:16 0 155.3G 0 lvm
└─qYVMiU-9EHg-nRje-5XoO-rFi2-dtW[...] 253:18 0 155.2G 0 crypt
sdd 8:48 0 1.8T 0 disk
└─ceph--block--fce5b416--a642--48b[...] 253:7 0 1.8T 0 lvm
└─Nsx79i-aRR9-2GeJ-i6jX-6ekE-Far[...] 253:9 0 1.8T 0 crypt
sde 8:64 0 1.8T 0 disk
└─ceph--block--0da07e49--e9d2--49a[...] 253:11 0 1.8T 0 lvm
└─vDb5md-6n6O-4jMj-J5ns-dnFx-NRP[...] 253:13 0 1.8T 0 crypt
sdf 8:80 0 931.5G 0 disk
└─ceph--block--65a4c7bb--0247--471[...] 253:15 0 931.5G 0 lvm
└─NWElgJ-3rQh-iD4b-7x9e-Gmkr-ude[...] 253:17 0 931.5G 0 crypt
First column pruned a bit to increase readability.
Used Ceph Installation Method
It goes without saying that this whole procedure was without any Ceph cluster downtime.
While the manual Ceph method would have been my fallback, I wanted to do this change with ceph-ansible.
I started with node number 4. That runs OSDs and grafana, but no MON nor MGR
and tried infrastructure-playbooks/add-osd.yml with the needed inventory change
(documented, amongst others, in the add-osd.yml itself), because I was unsure
how ceph-ansible would deal with a mix of lvm_volumes: and devices:.
Everything went fine when I added 3 OSDs running on node 4 the add-osds.yml way.
After that I switched to using site-container.yml to give that a try with the mix.
That also went absolutely fine.
In the future, now that I know all is fine with a mix, I would only use
site-container.yml Simply because that ensures that the co-located non-MON containers
(e.g. a MON or a grafana) are put back in place ASAP.
Performance Baseline with rados bench, pool with 16 PGs
Like previously, I gathered a performance baseline based on the instructions at 6.2. Performance baseline of the RHCS 4 Administration Guide.
Since I now had a total of 12 OSDs and with my existing 10 pools, I was below 30 PGs per OSD, I picked 16 PGs for my testbench pool by resizing the existing one from 8 to 16 PGs and waiting for all PGs to be active+clean).
All rados bench tests were run on f5-422-01.
Sequential Write Test, pool with 16 PGs
podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 write --no-cleanup | tee --append $(date +%F)-rados-test-write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_f5-422-01.internal.pcfe.net_1023
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 42 26 103.977 104 0.398447 0.365326
2 16 83 67 133.891 164 0.239482 0.414529
3 16 123 107 142.583 160 0.542042 0.413781
4 16 151 135 134.934 112 0.136264 0.41478
5 16 195 179 143.14 176 0.31506 0.425895
6 16 242 226 150.611 188 0.136594 0.408965
7 16 284 268 153.088 168 0.386108 0.405365
8 16 314 298 148.951 120 0.0920322 0.409887
9 16 358 342 151.953 176 0.413527 0.412807
10 16 398 382 152.754 160 0.320931 0.409935
11 16 439 423 153.772 164 0.225462 0.408766
12 16 470 454 151.29 124 0.182687 0.41236
13 16 502 486 149.497 128 0.270364 0.418863
14 16 531 515 147.103 116 0.120548 0.422944
15 16 577 561 149.559 184 0.261647 0.422054
16 16 618 602 150.46 164 0.237889 0.417494
17 16 653 637 149.844 140 0.111806 0.419136
18 16 689 673 149.518 144 0.538778 0.419554
19 16 726 710 149.437 148 0.237639 0.423924
2020-06-08 11:59:12.552308 min lat: 0.0794906 max lat: 1.44304 avg lat: 0.42214
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 762 746 149.164 144 0.17261 0.42214
21 16 797 781 148.726 140 1.04316 0.42337
22 16 831 815 148.147 136 0.774272 0.426569
23 16 863 847 147.27 128 0.128176 0.425961
24 16 893 877 146.133 120 0.34782 0.43013
25 16 921 905 144.766 112 0.805176 0.431506
26 16 958 942 144.889 148 0.32231 0.435559
27 16 992 976 144.559 136 0.0900235 0.437707
28 16 1035 1019 145.538 172 0.402015 0.436377
29 16 1075 1059 146.035 160 0.362278 0.435442
30 16 1113 1097 146.229 152 0.708746 0.432798
31 16 1150 1134 146.285 148 0.537088 0.433644
32 16 1188 1172 146.462 152 0.111633 0.432276
33 16 1209 1193 144.569 84 0.187649 0.433327
34 16 1243 1227 144.301 136 0.516067 0.43945
35 16 1269 1253 143.149 104 0.218054 0.439513
36 16 1307 1291 143.394 152 0.582905 0.442032
37 16 1348 1332 143.949 164 0.301545 0.440825
38 16 1374 1358 142.897 104 1.56705 0.442952
39 16 1408 1392 142.72 136 0.154026 0.444163
2020-06-08 11:59:32.561544 min lat: 0.0794906 max lat: 2.23993 avg lat: 0.444688
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1441 1425 142.45 132 0.41124 0.444688
41 16 1475 1459 142.292 136 0.154866 0.445385
42 16 1514 1498 142.618 156 0.26497 0.446216
43 16 1551 1535 142.742 148 0.246734 0.444359
44 16 1587 1571 142.762 144 0.0868129 0.444821
45 16 1622 1606 142.7 140 0.414572 0.445094
46 16 1654 1638 142.38 128 0.506686 0.446723
47 16 1693 1677 142.667 156 0.447949 0.445872
48 16 1728 1712 142.607 140 0.274668 0.445592
49 16 1772 1756 143.288 176 0.137756 0.444681
50 16 1811 1795 143.541 156 0.76994 0.444292
51 16 1844 1828 143.314 132 0.384643 0.444032
52 16 1888 1872 143.941 176 0.388899 0.442767
53 16 1923 1907 143.866 140 0.258826 0.441891
54 16 1965 1949 144.299 168 0.466725 0.441493
55 16 2002 1986 144.366 148 0.286507 0.441327
56 16 2036 2020 144.216 136 0.406687 0.440509
57 16 2066 2050 143.791 120 0.478132 0.441727
58 16 2109 2093 144.277 172 0.286874 0.441564
59 16 2152 2136 144.745 172 0.4606 0.440057
2020-06-08 11:59:52.575672 min lat: 0.0794906 max lat: 2.23993 avg lat: 0.440594
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 2180 2164 144.199 112 0.61586 0.440594
61 16 2230 2214 145.113 200 0.339176 0.4399
62 16 2273 2257 145.546 172 0.1729 0.437923
63 16 2311 2295 145.648 152 0.111251 0.437601
64 16 2350 2334 145.809 156 0.115738 0.435915
65 16 2383 2367 145.596 132 0.129876 0.435851
66 16 2417 2401 145.451 136 0.385 0.435713
67 16 2439 2423 144.593 88 0.0921996 0.437128
68 16 2468 2452 144.166 116 2.40533 0.439414
69 16 2495 2479 143.642 108 0.133776 0.440936
70 16 2521 2505 143.075 104 0.162519 0.441913
71 16 2550 2534 142.694 116 0.193849 0.442939
72 16 2578 2562 142.267 112 0.120453 0.442734
73 16 2620 2604 142.619 168 0.180627 0.446909
74 16 2648 2632 142.205 112 0.104623 0.446433
75 16 2688 2672 142.441 160 0.263871 0.446968
76 16 2724 2708 142.462 144 0.282413 0.446169
77 16 2760 2744 142.481 144 0.259337 0.446333
78 16 2802 2786 142.806 168 0.213389 0.446877
79 16 2847 2831 143.277 180 0.316799 0.4456
2020-06-08 12:00:12.583506 min lat: 0.0765629 max lat: 2.92381 avg lat: 0.444675
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 2886 2870 143.435 156 0.120977 0.444675
81 16 2909 2893 142.8 92 0.156527 0.445901
82 16 2942 2926 142.668 132 0.581131 0.446634
83 16 2981 2965 142.829 156 0.550399 0.446677
84 16 3028 3012 143.366 188 0.166803 0.445309
85 16 3061 3045 143.232 132 0.129409 0.444698
86 16 3107 3091 143.706 184 0.260706 0.44461
87 16 3144 3128 143.755 148 0.0969723 0.443777
88 16 3179 3163 143.709 140 0.399811 0.444042
89 16 3218 3202 143.846 156 0.368254 0.443821
90 16 3251 3235 143.715 132 0.679004 0.443631
91 16 3292 3276 143.937 164 0.498053 0.443424
92 16 3330 3314 144.023 152 0.62485 0.443343
93 16 3369 3353 144.151 156 0.104437 0.442384
94 16 3400 3384 143.935 124 0.0818116 0.442889
95 16 3430 3414 143.683 120 0.0899581 0.443341
96 16 3464 3448 143.598 136 0.780032 0.444225
97 16 3501 3485 143.644 148 0.256891 0.44381
98 16 3536 3520 143.606 140 0.198576 0.444417
99 16 3571 3555 143.569 140 0.236379 0.444209
2020-06-08 12:00:32.593968 min lat: 0.0753806 max lat: 2.92381 avg lat: 0.44355
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3615 3599 143.893 176 0.696873 0.44355
101 16 3642 3626 143.538 108 0.626366 0.443691
102 16 3683 3667 143.738 164 0.190347 0.44365
103 16 3717 3701 143.662 136 0.765361 0.443916
104 16 3759 3743 143.896 168 0.276769 0.443753
105 16 3791 3775 143.744 128 0.118761 0.443968
106 16 3830 3814 143.859 156 0.139814 0.443565
107 16 3871 3855 144.042 164 0.371427 0.443257
108 16 3910 3894 144.152 156 0.143957 0.442597
109 16 3951 3935 144.333 164 0.205618 0.442368
110 16 3988 3972 144.366 148 1.04951 0.442508
111 16 4034 4018 144.723 184 0.0987925 0.441425
112 16 4066 4050 144.566 128 0.533442 0.441813
113 16 4108 4092 144.773 168 0.279903 0.440983
114 16 4141 4125 144.66 132 0.657968 0.441282
115 16 4176 4160 144.62 140 0.268767 0.441315
116 16 4208 4192 144.476 128 0.465917 0.442303
117 15 4244 4229 144.506 148 0.318879 0.44183
118 16 4275 4259 144.298 120 0.408339 0.442327
119 16 4314 4298 144.396 156 0.308683 0.441775
2020-06-08 12:00:52.611705 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.442011
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 4353 4337 144.489 156 1.04169 0.442011
121 16 4381 4365 144.221 112 0.899546 0.442104
122 16 4405 4389 143.823 96 1.1861 0.442601
123 16 4439 4423 143.759 136 0.171213 0.443698
124 16 4468 4452 143.535 116 0.123565 0.444382
125 16 4503 4487 143.504 140 0.151779 0.445298
126 16 4540 4524 143.538 148 0.826155 0.444937
127 16 4577 4561 143.573 148 0.296185 0.444777
128 16 4613 4597 143.576 144 0.311355 0.444352
129 16 4640 4624 143.3 108 0.843498 0.444817
130 16 4674 4658 143.244 136 0.122587 0.445197
131 16 4709 4693 143.216 140 0.195114 0.445781
132 16 4751 4735 143.404 168 0.197907 0.445371
133 16 4784 4768 143.318 132 0.256699 0.445172
134 16 4818 4802 143.263 136 0.171877 0.445363
135 16 4860 4844 143.446 168 0.198978 0.445519
136 16 4891 4875 143.303 124 0.305737 0.445381
137 16 4929 4913 143.366 152 0.427361 0.445782
138 16 4964 4948 143.341 140 0.211179 0.445654
139 16 4996 4980 143.23 128 0.241285 0.445271
2020-06-08 12:01:12.624783 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.445793
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 5026 5010 143.064 120 1.03791 0.445793
141 16 5056 5040 142.9 120 0.137966 0.446919
142 16 5080 5064 142.57 96 1.53823 0.446994
143 16 5119 5103 142.664 156 0.51641 0.447947
144 16 5163 5147 142.895 176 0.268755 0.447237
145 16 5205 5189 143.067 168 0.124783 0.446577
146 16 5246 5230 143.21 164 0.317081 0.446294
147 16 5285 5269 143.297 156 0.290308 0.446103
148 16 5332 5316 143.599 188 0.366505 0.445152
149 16 5372 5356 143.709 160 0.214472 0.44444
150 16 5419 5403 144.003 188 0.137074 0.443797
151 16 5459 5443 144.109 160 0.490304 0.443261
152 16 5504 5488 144.344 180 0.434736 0.442887
153 16 5546 5530 144.498 168 0.278205 0.442198
154 16 5588 5572 144.65 168 0.175283 0.441773
155 16 5631 5615 144.826 172 0.242373 0.441296
156 16 5673 5657 144.975 168 0.303418 0.440525
157 16 5717 5701 145.172 176 0.267878 0.440493
158 16 5759 5743 145.316 168 0.218735 0.439766
159 16 5800 5784 145.434 164 0.318679 0.439357
2020-06-08 12:01:32.630641 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.438659
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 5847 5831 145.699 188 0.381478 0.438659
161 16 5892 5876 145.912 180 0.129163 0.437905
162 16 5928 5912 145.9 144 0.412485 0.437835
163 16 5966 5950 145.937 152 0.400137 0.437772
164 16 6006 5990 146.023 160 0.216359 0.437808
165 16 6044 6028 146.059 152 0.175006 0.437535
166 16 6085 6069 146.165 164 0.266665 0.437137
167 16 6124 6108 146.224 156 0.143355 0.436973
168 16 6162 6146 146.258 152 0.264271 0.436783
169 16 6202 6186 146.339 160 0.817793 0.436957
170 16 6235 6219 146.253 132 0.132424 0.436679
171 16 6287 6271 146.614 208 0.34244 0.436127
172 16 6329 6313 146.738 168 0.392364 0.435427
173 16 6379 6363 147.045 200 0.212298 0.434666
174 16 6422 6406 147.188 172 0.3763 0.433958
175 16 6451 6435 147.01 116 0.272379 0.434327
176 16 6491 6475 147.084 160 0.241195 0.434484
177 16 6532 6516 147.179 164 0.0907898 0.433779
178 16 6567 6551 147.138 140 0.114962 0.433723
179 16 6609 6593 147.255 168 0.187057 0.434074
2020-06-08 12:01:52.638675 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.433549
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 6656 6640 147.481 188 0.292246 0.433549
181 16 6703 6687 147.704 188 0.302311 0.432886
182 16 6749 6733 147.904 184 0.37474 0.432322
183 16 6792 6776 148.035 172 0.361231 0.43173
184 16 6838 6822 148.23 184 0.133267 0.431371
185 16 6885 6869 148.445 188 0.29473 0.430569
186 16 6931 6915 148.636 184 0.261117 0.429924
187 16 6965 6949 148.568 136 1.01155 0.429947
188 16 6991 6975 148.331 104 0.410788 0.43045
189 16 7028 7012 148.329 148 0.144746 0.430771
190 16 7068 7052 148.391 160 0.129355 0.43056
191 15 7106 7091 148.43 156 0.516433 0.430598
192 16 7142 7126 148.386 140 0.228968 0.430469
193 16 7182 7166 148.446 160 0.191215 0.430637
194 16 7226 7210 148.588 176 0.265485 0.430225
195 16 7269 7253 148.706 172 0.238815 0.430121
196 16 7319 7303 148.967 200 0.299365 0.42922
197 16 7365 7349 149.145 184 0.135801 0.428615
198 16 7410 7394 149.3 180 0.279442 0.428367
199 16 7455 7439 149.454 180 0.485155 0.427842
2020-06-08 12:02:12.645137 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.427369
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 7495 7479 149.507 160 0.124383 0.427369
201 16 7543 7527 149.718 192 0.307045 0.427205
202 16 7588 7572 149.868 180 0.157108 0.426854
203 16 7624 7608 149.839 144 0.215977 0.426455
204 16 7666 7650 149.927 168 0.103162 0.426126
205 16 7709 7693 150.035 172 0.161674 0.425948
206 16 7759 7743 150.277 200 0.610101 0.425483
207 16 7800 7784 150.342 164 0.401227 0.42529
208 16 7843 7827 150.446 172 0.463575 0.425082
209 16 7894 7878 150.7 204 0.302787 0.424372
210 16 7943 7927 150.915 196 0.309337 0.423608
211 16 7996 7980 151.203 212 0.249553 0.422875
212 16 8038 8022 151.282 168 0.359746 0.422442
213 16 8084 8068 151.436 184 0.429353 0.422246
214 16 8129 8113 151.566 180 0.276603 0.421692
215 16 8176 8160 151.735 188 0.458162 0.421328
216 16 8218 8202 151.811 168 0.241697 0.42092
217 16 8265 8249 151.977 188 0.33623 0.420796
218 16 8303 8287 151.977 152 0.404618 0.420554
219 16 8343 8327 152.013 160 0.616621 0.420138
2020-06-08 12:02:32.662702 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.420531
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 8385 8369 152.084 168 0.311759 0.420531
221 16 8432 8416 152.246 188 0.30745 0.420057
222 16 8482 8466 152.46 200 0.0896688 0.419295
223 16 8523 8507 152.512 164 0.527969 0.419227
224 16 8573 8557 152.724 200 0.243354 0.418652
225 16 8625 8609 152.969 208 0.13521 0.41811
226 16 8667 8651 153.036 168 0.798107 0.41786
227 16 8697 8681 152.89 120 0.121405 0.41768
228 16 8734 8718 152.868 148 1.35738 0.418116
229 16 8784 8768 153.074 200 0.459701 0.417857
230 16 8823 8807 153.086 156 0.56197 0.417524
231 16 8871 8855 153.255 192 0.433494 0.417145
232 16 8921 8905 153.456 200 0.607592 0.416747
233 16 8965 8949 153.553 176 0.111828 0.416236
234 16 9000 8984 153.495 140 0.804002 0.416245
235 16 9035 9019 153.437 140 0.966972 0.416357
236 16 9071 9055 153.396 144 0.253087 0.41683
237 16 9120 9104 153.576 196 0.157833 0.416397
238 16 9152 9136 153.468 128 0.129639 0.416469
239 16 9198 9182 153.596 184 0.165864 0.416347
2020-06-08 12:02:52.669071 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.416047
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 9243 9227 153.705 180 0.103726 0.416047
241 16 9294 9278 153.914 204 0.11835 0.415359
242 16 9340 9324 154.038 184 0.363781 0.415079
243 16 9376 9360 153.997 144 0.141759 0.414889
244 16 9410 9394 153.923 136 0.264552 0.415212
245 16 9438 9422 153.75 112 1.19538 0.41518
246 16 9469 9453 153.629 124 0.188027 0.415619
247 16 9490 9474 153.347 84 0.093494 0.416
248 16 9518 9502 153.18 112 0.0887564 0.416562
249 16 9544 9528 152.983 104 0.195166 0.417013
250 16 9568 9552 152.754 96 0.181726 0.41786
251 16 9582 9566 152.369 56 0.137892 0.418165
252 16 9608 9592 152.177 104 0.195848 0.419046
253 16 9633 9617 151.971 100 0.145371 0.420078
254 16 9661 9645 151.813 112 0.293228 0.420621
255 16 9695 9679 151.751 136 0.159858 0.42052
256 16 9723 9707 151.595 112 0.166974 0.42118
257 16 9745 9729 151.348 88 0.0940269 0.421655
258 16 9763 9747 151.04 72 0.118476 0.42253
259 16 9784 9768 150.781 84 0.134013 0.42298
2020-06-08 12:03:12.678681 min lat: 0.0722708 max lat: 3.23771 avg lat: 0.423818
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 9799 9783 150.432 60 0.123094 0.423818
261 16 9826 9810 150.269 108 0.695075 0.425154
262 16 9866 9850 150.306 160 0.348291 0.425046
263 16 9882 9866 149.977 64 1.42374 0.425621
264 16 9911 9895 149.848 116 2.4134 0.425976
265 16 9929 9913 149.555 72 0.0979419 0.426662
266 16 9962 9946 149.488 132 0.209607 0.427224
267 16 9991 9975 149.362 116 0.225419 0.427444
268 16 10018 10002 149.208 108 1.18886 0.427815
269 16 10053 10037 149.174 140 0.216152 0.42824
270 16 10068 10052 148.843 60 0.210188 0.428728
271 16 10101 10085 148.781 132 0.19895 0.428999
272 16 10114 10098 148.425 52 0.122905 0.429609
273 16 10132 10116 148.145 72 0.0935088 0.43025
274 16 10160 10144 148.013 112 0.147505 0.430672
275 16 10176 10160 147.708 64 1.49305 0.431699
276 16 10194 10178 147.434 72 0.724972 0.432737
277 16 10212 10196 147.161 72 2.36926 0.433687
278 16 10233 10217 146.934 84 0.143773 0.434311
279 16 10253 10237 146.694 80 1.52916 0.435166
2020-06-08 12:03:32.687564 min lat: 0.0722708 max lat: 3.23771 avg lat: 0.435724
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 10271 10255 146.427 72 0.161734 0.435724
281 16 10290 10274 146.176 76 0.125682 0.436299
282 16 10312 10296 145.97 88 0.129635 0.436866
283 16 10342 10326 145.878 120 0.15821 0.437504
284 16 10377 10361 145.857 140 0.0928508 0.437496
285 16 10406 10390 145.752 116 0.518276 0.438129
286 16 10422 10406 145.467 64 1.07445 0.43872
287 16 10440 10424 145.211 72 0.776983 0.439414
288 16 10469 10453 145.109 116 0.111772 0.439781
289 16 10486 10470 144.842 68 1.64157 0.440535
290 16 10502 10486 144.563 64 0.136105 0.440971
291 16 10540 10524 144.588 152 0.246552 0.442327
292 16 10576 10560 144.586 144 0.312766 0.44208
293 16 10610 10594 144.557 136 0.149812 0.441935
294 16 10627 10611 144.296 68 1.81328 0.442429
295 16 10645 10629 144.051 72 2.88195 0.443139
296 16 10672 10656 143.929 108 0.09646 0.443584
297 16 10698 10682 143.795 104 0.169175 0.443832
298 16 10723 10707 143.648 100 0.0914411 0.444542
299 16 10744 10728 143.448 84 0.187771 0.444777
2020-06-08 12:03:52.693662 min lat: 0.0722708 max lat: 3.378 avg lat: 0.445679
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 10775 10759 143.383 124 0.133518 0.445679
301 16 10808 10792 143.346 132 0.185371 0.445421
302 16 10831 10815 143.175 92 1.07333 0.446164
303 16 10853 10837 142.993 88 1.42305 0.446246
304 16 10878 10862 142.852 100 0.886884 0.446938
305 16 10920 10904 142.934 168 0.101993 0.447436
306 16 10956 10940 142.937 144 0.122725 0.44727
307 16 10989 10973 142.901 132 0.169972 0.447055
308 16 11013 10997 142.748 96 0.139724 0.447154
309 16 11037 11021 142.597 96 0.12336 0.447716
310 16 11060 11044 142.434 92 0.289583 0.448132
311 16 11091 11075 142.375 124 0.0838391 0.448213
312 16 11104 11088 142.085 52 0.152349 0.448905
313 16 11128 11112 141.938 96 0.168615 0.449489
314 16 11151 11135 141.779 92 0.181976 0.450049
315 16 11170 11154 141.57 76 0.0941329 0.450461
316 16 11202 11186 141.526 128 0.117752 0.451235
317 16 11230 11214 141.433 112 0.178836 0.451296
318 16 11241 11225 141.126 44 1.07644 0.451933
319 16 11261 11245 140.935 80 0.124781 0.452948
2020-06-08 12:04:12.702569 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.453411
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 11294 11278 140.907 132 0.0905571 0.453411
321 16 11318 11302 140.767 96 1.35392 0.453578
322 16 11347 11331 140.69 116 0.0921295 0.453859
323 16 11372 11356 140.564 100 1.87001 0.454322
324 16 11386 11370 140.303 56 0.166954 0.454588
325 16 11421 11405 140.302 140 0.102514 0.455089
326 16 11446 11430 140.178 100 0.14884 0.455715
327 16 11458 11442 139.896 48 1.09856 0.456165
328 16 11485 11469 139.799 108 0.163835 0.456908
329 16 11509 11493 139.666 96 0.112958 0.457008
330 16 11523 11507 139.412 56 0.109407 0.457764
331 16 11553 11537 139.353 120 0.148788 0.458412
332 16 11572 11556 139.163 76 0.0888086 0.458897
333 16 11590 11574 138.961 72 1.99026 0.459604
334 16 11611 11595 138.796 84 0.159798 0.46014
335 16 11622 11606 138.513 44 0.126609 0.460395
336 16 11652 11636 138.458 120 1.67195 0.461612
337 16 11679 11663 138.368 108 0.18286 0.461683
338 16 11710 11694 138.325 124 0.0989499 0.46173
339 16 11740 11724 138.271 120 0.128565 0.462053
2020-06-08 12:04:32.707720 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.462351
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 11760 11744 138.1 80 2.54891 0.462351
341 16 11808 11792 138.258 192 0.334997 0.462584
342 16 11865 11849 138.52 228 0.376299 0.461876
343 16 11902 11886 138.547 148 0.686378 0.461656
344 16 11948 11932 138.679 184 0.296212 0.461112
345 16 11996 11980 138.834 192 0.264821 0.46061
346 16 12046 12030 139.01 200 0.370486 0.460037
347 16 12095 12079 139.174 196 0.209926 0.459683
348 16 12146 12130 139.361 204 0.254113 0.459082
349 16 12201 12185 139.591 220 0.355664 0.458304
350 16 12250 12234 139.752 196 0.220042 0.457672
351 16 12300 12284 139.924 200 0.432428 0.457187
352 16 12352 12336 140.117 208 0.258861 0.45658
353 16 12398 12382 140.241 184 0.275295 0.456105
354 16 12445 12429 140.376 188 0.233265 0.455581
355 16 12496 12480 140.554 204 0.160331 0.455161
356 16 12537 12521 140.619 164 0.336082 0.454759
357 16 12581 12565 140.718 176 0.310762 0.454478
358 16 12637 12621 140.951 224 0.274925 0.453871
359 16 12685 12669 141.093 192 0.232541 0.453355
2020-06-08 12:04:52.715753 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.452734
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 12739 12723 141.301 216 0.225254 0.452734
361 16 12784 12768 141.407 180 0.148216 0.452396
362 16 12825 12809 141.47 164 0.258651 0.452135
363 16 12867 12851 141.543 168 0.269325 0.451757
364 16 12913 12897 141.659 184 0.238692 0.451572
365 16 12964 12948 141.83 204 0.252388 0.451034
366 16 13012 12996 141.966 192 0.131394 0.450524
367 16 13058 13042 142.08 184 0.272433 0.45026
368 16 13105 13089 142.205 188 0.268804 0.44981
369 16 13141 13125 142.209 144 0.311773 0.44929
370 16 13179 13163 142.234 152 2.03279 0.449493
371 16 13223 13207 142.325 176 0.227359 0.44935
372 16 13269 13253 142.437 184 0.179598 0.448779
373 16 13307 13291 142.462 152 0.318101 0.448404
374 16 13351 13335 142.552 176 0.66818 0.448633
375 16 13390 13374 142.587 156 0.223599 0.44857
376 16 13435 13419 142.687 180 0.233914 0.448252
377 16 13482 13466 142.807 188 0.277388 0.447959
378 16 13529 13513 142.926 188 0.602233 0.447456
379 16 13576 13560 143.045 188 0.20597 0.447141
2020-06-08 12:05:12.729810 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.446799
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 13625 13609 143.184 196 0.350819 0.446799
381 16 13676 13660 143.343 204 0.377371 0.446217
382 16 13719 13703 143.418 172 0.376005 0.445957
383 16 13774 13758 143.618 220 0.202868 0.44556
384 16 13812 13796 143.64 152 0.335153 0.445335
385 16 13853 13837 143.692 164 0.195561 0.444999
386 16 13899 13883 143.794 184 0.198264 0.444808
387 16 13956 13940 144.012 228 0.127742 0.444229
388 16 13997 13981 144.063 164 0.677126 0.444023
389 16 14040 14024 144.135 172 0.708282 0.443725
390 16 14091 14075 144.288 204 0.30224 0.443349
391 16 14141 14125 144.431 200 0.215943 0.442911
392 16 14179 14163 144.45 152 0.196804 0.442711
393 16 14211 14195 144.408 128 0.0901027 0.442863
394 16 14254 14238 144.478 172 0.610329 0.442763
395 16 14293 14277 144.507 156 0.584145 0.442646
396 16 14339 14323 144.606 184 0.410331 0.442333
397 16 14382 14366 144.675 172 0.347563 0.442056
398 16 14436 14420 144.854 216 0.148742 0.44166
399 16 14485 14469 144.982 196 0.504047 0.441184
2020-06-08 12:05:32.741202 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.440707
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 14536 14520 145.13 204 0.354395 0.440707
401 16 14585 14569 145.256 196 0.349962 0.440394
402 16 14630 14614 145.342 180 0.797574 0.440063
403 16 14681 14665 145.487 204 0.426952 0.439734
404 16 14723 14707 145.543 168 0.428204 0.43952
405 16 14775 14759 145.697 208 0.239253 0.439079
406 16 14826 14810 145.841 204 0.480806 0.438641
407 16 14881 14865 146.022 220 0.317419 0.43815
408 16 14931 14915 146.154 200 0.210176 0.437732
409 16 14982 14966 146.295 204 0.293815 0.437273
410 16 15023 15007 146.339 164 0.611661 0.437085
411 16 15075 15059 146.488 208 0.128427 0.436685
412 16 15127 15111 146.638 208 0.132936 0.436226
413 16 15178 15162 146.776 204 0.290908 0.435812
414 16 15231 15215 146.933 212 0.358994 0.435405
415 16 15272 15256 146.974 164 0.365472 0.435181
416 16 15314 15298 147.024 168 0.335647 0.435094
417 16 15366 15350 147.171 208 0.33835 0.434725
418 16 15414 15398 147.278 192 0.462802 0.434373
419 16 15462 15446 147.384 192 0.453619 0.433979
2020-06-08 12:05:52.751372 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.433664
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 15513 15497 147.519 204 0.391907 0.433664
421 16 15564 15548 147.653 204 0.384983 0.433258
422 16 15611 15595 147.748 188 0.246389 0.432973
423 16 15656 15640 147.824 180 0.093805 0.432728
424 16 15702 15686 147.91 184 0.179644 0.432476
425 16 15752 15736 148.031 200 0.319919 0.432128
426 16 15805 15789 148.18 212 0.175002 0.43168
427 16 15851 15835 148.263 184 0.290861 0.431402
428 16 15903 15887 148.403 208 0.318378 0.431115
429 16 15946 15930 148.458 172 0.206864 0.43086
430 16 16000 15984 148.615 216 0.417185 0.430481
431 16 16044 16028 148.678 176 0.348322 0.430173
432 16 16097 16081 148.824 212 0.255571 0.429839
433 16 16150 16134 148.97 212 0.488645 0.429347
434 16 16200 16184 149.088 200 0.229537 0.429113
435 16 16254 16238 149.239 216 0.429418 0.428716
436 16 16300 16284 149.319 184 0.364776 0.428328
437 16 16350 16334 149.435 200 0.243105 0.428091
438 16 16400 16384 149.55 200 0.267639 0.427729
439 16 16447 16431 149.638 188 0.590509 0.427413
2020-06-08 12:06:12.768851 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.427493
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 16483 16467 149.625 144 0.468702 0.427493
441 16 16524 16508 149.657 164 0.251888 0.427409
442 16 16570 16554 149.735 184 0.205337 0.427203
443 16 16623 16607 149.875 212 0.162406 0.426895
444 16 16669 16653 149.951 184 0.286025 0.426638
445 15 16718 16703 150.064 200 0.417447 0.426309
446 16 16773 16757 150.211 216 0.189652 0.425987
447 16 16818 16802 150.278 180 0.153314 0.425695
448 16 16864 16848 150.353 184 0.297534 0.425524
449 16 16904 16888 150.374 160 0.14118 0.425322
450 16 16960 16944 150.538 224 0.0969851 0.424962
451 16 16995 16979 150.514 140 0.146842 0.424915
452 16 17047 17031 150.641 208 0.310798 0.424657
453 16 17092 17076 150.706 180 0.417551 0.424498
454 16 17132 17116 150.726 160 0.465241 0.424403
455 16 17190 17174 150.905 232 0.245149 0.423994
456 16 17236 17220 150.977 184 0.291126 0.423743
457 16 17281 17265 151.037 180 0.454404 0.42348
458 16 17326 17310 151.1 180 0.178635 0.423288
459 16 17375 17359 151.197 196 0.223157 0.423067
2020-06-08 12:06:32.788643 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.422829
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 17425 17409 151.303 200 0.219531 0.422829
461 16 17477 17461 151.426 208 0.339728 0.422485
462 16 17532 17516 151.574 220 0.367407 0.422087
463 16 17577 17561 151.635 180 0.267091 0.421897
464 16 17626 17610 151.73 196 0.291679 0.421667
465 16 17673 17657 151.808 188 0.334067 0.421415
466 16 17710 17694 151.8 148 0.163953 0.421337
467 16 17765 17749 151.946 220 0.222007 0.421052
468 16 17812 17796 152.023 188 0.553244 0.420795
469 16 17862 17846 152.125 200 0.224579 0.420571
470 16 17918 17902 152.278 224 0.327208 0.420123
471 16 17968 17952 152.379 200 0.261327 0.419853
472 16 18020 18004 152.497 208 0.164646 0.419458
473 16 18069 18053 152.589 196 0.258609 0.419252
474 16 18127 18111 152.756 232 0.280153 0.418806
475 16 18176 18160 152.847 196 0.468567 0.418589
476 16 18211 18195 152.82 140 0.634876 0.418565
477 16 18257 18241 152.885 184 0.135715 0.418466
478 16 18312 18296 153.024 220 0.351938 0.41815
479 16 18362 18346 153.121 200 0.383693 0.417802
2020-06-08 12:06:52.806381 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.417531
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 18411 18395 153.209 196 0.0827548 0.417531
481 16 18456 18440 153.265 180 0.280516 0.417473
482 16 18504 18488 153.345 192 0.223016 0.417164
483 16 18552 18536 153.425 192 0.2243 0.41698
484 16 18601 18585 153.513 196 0.403077 0.416713
485 16 18649 18633 153.592 192 0.164898 0.416485
486 16 18702 18686 153.712 212 0.454405 0.416235
487 16 18753 18737 153.815 204 0.55738 0.415955
488 16 18800 18784 153.885 188 0.326559 0.41571
489 16 18845 18829 153.938 180 0.307519 0.415588
490 16 18885 18869 153.951 160 0.163752 0.415423
491 16 18939 18923 154.077 216 0.206451 0.415218
492 16 18994 18978 154.211 220 0.448214 0.414921
493 16 19033 19017 154.213 156 0.326946 0.414701
494 16 19082 19066 154.296 196 0.337582 0.414674
495 16 19135 19119 154.413 212 0.249238 0.414306
496 16 19180 19164 154.464 180 0.105208 0.414089
497 16 19227 19211 154.532 188 0.101409 0.413962
498 16 19276 19260 154.614 196 0.294092 0.413753
499 16 19328 19312 154.721 208 0.213221 0.413482
2020-06-08 12:07:12.820907 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.413285
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 19376 19360 154.795 192 0.531453 0.413285
501 16 19422 19406 154.854 184 0.248493 0.413083
502 16 19477 19461 154.983 220 0.657029 0.412738
503 16 19524 19508 155.048 188 0.413768 0.412669
504 16 19573 19557 155.13 196 0.313754 0.412422
505 16 19616 19600 155.163 172 0.553231 0.412282
506 16 19662 19646 155.22 184 0.142738 0.412156
507 16 19709 19693 155.282 188 0.230748 0.412015
508 16 19759 19743 155.369 200 0.0998474 0.411784
509 16 19804 19788 155.417 180 0.378392 0.411674
510 16 19849 19833 155.465 180 0.112066 0.411457
511 16 19896 19880 155.529 188 0.237953 0.411342
512 16 19947 19931 155.624 204 0.352576 0.411094
513 16 19987 19971 155.632 160 0.257366 0.411005
514 16 20040 20024 155.742 212 0.296339 0.410809
515 16 20090 20074 155.827 200 0.384746 0.410599
516 16 20144 20128 155.944 216 0.308672 0.41026
517 16 20193 20177 156.021 196 0.236316 0.410056
518 16 20245 20229 156.121 208 0.340904 0.409771
519 16 20292 20276 156.182 188 0.806175 0.409652
2020-06-08 12:07:32.840163 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.409371
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 20344 20328 156.281 208 0.243567 0.409371
521 16 20399 20383 156.403 220 0.328701 0.409084
522 16 20445 20429 156.456 184 0.230827 0.408857
523 16 20488 20472 156.485 172 0.454214 0.408771
524 16 20540 20524 156.584 208 0.230132 0.408558
525 16 20585 20569 156.628 180 0.141949 0.408427
526 16 20626 20610 156.642 164 0.211831 0.408297
527 16 20675 20659 156.717 196 0.386993 0.408298
528 16 20723 20707 156.783 192 0.322823 0.408065
529 16 20765 20749 156.804 168 0.238553 0.407967
530 16 20816 20800 156.893 204 0.176812 0.407788
531 16 20866 20850 156.974 200 0.363665 0.407535
532 16 20911 20895 157.017 180 0.369616 0.407365
533 16 20959 20943 157.083 192 0.400751 0.407259
534 16 21008 20992 157.154 196 0.455853 0.407108
535 16 21048 21032 157.16 160 0.764347 0.407024
536 16 21082 21066 157.12 136 0.171308 0.407108
537 16 21132 21116 157.2 200 0.130863 0.40698
538 16 21179 21163 157.257 188 0.101887 0.406867
539 16 21228 21212 157.329 196 0.168948 0.406631
2020-06-08 12:07:52.854944 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.406513
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 21274 21258 157.377 184 0.442846 0.406513
541 16 21322 21306 157.441 192 0.332705 0.40635
542 16 21375 21359 157.541 212 0.440169 0.40609
543 16 21422 21406 157.598 188 0.385983 0.405945
544 16 21475 21459 157.697 212 0.305759 0.405751
545 16 21527 21511 157.79 208 0.176825 0.405499
546 16 21570 21554 157.816 172 0.53299 0.405342
547 16 21622 21606 157.907 208 0.321296 0.405171
548 16 21675 21659 158.006 212 0.548881 0.404927
549 16 21721 21705 158.053 184 0.828538 0.404788
550 15 21774 21759 158.158 216 0.509562 0.40455
551 16 21835 21819 158.307 240 0.363137 0.404145
552 16 21888 21872 158.402 212 0.310273 0.403918
553 16 21929 21913 158.412 164 0.225615 0.403842
554 16 21984 21968 158.523 220 0.237284 0.403607
555 16 22030 22014 158.569 184 0.218398 0.403441
556 16 22083 22067 158.665 212 0.376688 0.403269
557 16 22132 22116 158.732 196 0.153964 0.403055
558 16 22185 22169 158.828 212 0.136437 0.40283
559 16 22238 22222 158.923 212 0.474256 0.402602
2020-06-08 12:08:12.864001 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.402452
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 22283 22267 158.96 180 0.462368 0.402452
561 16 22335 22319 159.047 208 0.33058 0.402278
562 16 22376 22360 159.056 164 0.670445 0.402127
563 16 22423 22407 159.107 188 0.102584 0.402078
564 16 22467 22451 159.137 176 0.334776 0.401971
565 16 22517 22501 159.209 200 0.287939 0.401841
566 16 22570 22554 159.303 212 0.336027 0.401625
567 16 22611 22595 159.311 164 0.282087 0.401573
568 16 22664 22648 159.403 212 0.316354 0.401382
569 16 22711 22695 159.454 188 0.36714 0.401212
570 16 22752 22736 159.462 164 0.64866 0.401213
571 16 22796 22780 159.49 176 0.0893633 0.401081
572 16 22845 22829 159.554 196 0.217594 0.40101
573 16 22897 22881 159.639 208 0.395147 0.400811
574 16 22950 22934 159.73 212 0.421703 0.400534
575 16 22996 22980 159.772 184 0.230524 0.400469
576 16 23044 23028 159.828 192 0.106761 0.400302
577 16 23092 23076 159.883 192 0.136746 0.400093
578 16 23138 23122 159.925 184 0.260966 0.400044
579 16 23184 23168 159.967 184 0.48356 0.39993
2020-06-08 12:08:32.868653 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.399762
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 23233 23217 160.029 196 0.100009 0.399762
581 16 23285 23269 160.111 208 0.158893 0.399638
582 16 23328 23312 160.131 172 0.252311 0.399499
583 16 23384 23368 160.241 224 0.116848 0.39925
584 16 23438 23422 160.336 216 0.258759 0.399074
585 16 23482 23466 160.363 176 0.312108 0.398946
586 16 23532 23516 160.43 200 0.107516 0.398786
587 16 23589 23573 160.545 228 0.221537 0.398577
588 16 23632 23616 160.564 172 0.535554 0.398405
589 16 23678 23662 160.604 184 0.647506 0.398369
590 16 23725 23709 160.65 188 0.0864281 0.398182
591 16 23776 23760 160.724 204 0.532072 0.39804
592 16 23831 23815 160.824 220 0.234719 0.397821
593 16 23880 23864 160.882 196 0.262867 0.397658
594 16 23935 23919 160.982 220 0.416923 0.397486
595 16 23980 23964 161.014 180 0.184396 0.397362
596 16 24019 24003 161.005 156 0.216402 0.397268
597 16 24069 24053 161.071 200 0.373137 0.39722
598 16 24123 24107 161.162 216 0.325188 0.396974
599 16 24172 24156 161.22 196 0.238673 0.396852
2020-06-08 12:08:52.876561 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.396759
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 16 24206 24190 161.178 136 0.263172 0.396759
Total time run: 600.336
Total writes made: 24206
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 161.283
Stddev Bandwidth: 40.5339
Max bandwidth (MB/sec): 240
Min bandwidth (MB/sec): 44
Average IOPS: 40
Stddev IOPS: 10.1335
Max IOPS: 60
Min IOPS: 11
Average Latency(s): 0.396815
Stddev Latency(s): 0.332784
Max latency(s): 4.09319
Min latency(s): 0.0703938
Total time run: 600.336
Total writes made: 24206
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 161.283
Stddev Bandwidth: 40.5339
Max bandwidth (MB/sec): 240
Min bandwidth (MB/sec): 44
Average IOPS: 40
Stddev IOPS: 10.1335
Max IOPS: 60
Min IOPS: 11
Average Latency(s): 0.396815
Stddev Latency(s): 0.332784
Max latency(s): 4.09319
Min latency(s): 0.0703938
Sequential Read Test, pool with 16 PGs
podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 seq | tee --append $(date +%F)-rados-test-read-seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 16 16 0 0 0 - 0
1 16 61 45 179.725 180 0.653734 0.265785
2 16 102 86 171.736 164 0.26818 0.304166
3 16 141 125 166.455 156 0.62637 0.346581
4 16 191 175 174.807 200 0.653689 0.333282
5 16 236 220 175.837 180 0.0471615 0.341651
6 16 281 265 176.454 180 0.68578 0.338637
7 16 326 310 176.947 180 0.327634 0.340565
8 16 377 361 180.322 204 0.0924538 0.335444
9 15 424 409 181.595 192 0.0621164 0.338367
10 16 475 459 183.404 200 0.0580188 0.330934
11 16 517 501 182.002 168 0.305636 0.334249
12 16 560 544 181.168 172 0.0857937 0.338028
13 16 597 581 178.603 148 0.328407 0.345288
14 16 642 626 178.701 180 0.0691258 0.344505
15 16 685 669 178.253 172 0.0807547 0.348703
16 16 723 707 176.612 152 0.200282 0.349929
17 16 766 750 176.326 172 0.154271 0.35287
18 16 818 802 178.045 208 0.0766331 0.349761
19 16 862 846 177.936 176 0.44736 0.350352
2020-06-08 12:21:19.424208 min lat: 0.0249483 max lat: 1.60795 avg lat: 0.350601
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 908 892 178.182 184 0.249175 0.350601
21 16 953 937 178.267 180 0.292387 0.348578
22 16 994 978 177.61 164 0.059223 0.350062
23 16 1032 1016 176.497 152 0.87098 0.352958
24 16 1070 1054 175.477 152 0.114091 0.354365
25 15 1103 1088 173.859 136 0.100794 0.357305
26 16 1147 1131 173.775 172 0.296744 0.358744
27 16 1188 1172 173.4 164 0.0501677 0.359326
28 15 1223 1208 172.349 144 0.662945 0.362405
29 16 1265 1249 171.949 164 1.23023 0.361509
30 16 1312 1296 172.467 188 0.338225 0.360919
31 16 1352 1336 172.065 160 0.138675 0.362541
32 16 1396 1380 172.187 176 0.229331 0.363456
33 16 1436 1420 171.818 160 0.0771103 0.363231
34 16 1477 1461 171.587 164 0.382998 0.363294
35 16 1524 1508 172.055 188 0.106015 0.362061
36 16 1564 1548 171.714 160 0.289089 0.362784
37 15 1604 1589 171.496 164 0.0598599 0.363486
38 16 1642 1626 170.878 148 1.15266 0.366261
39 16 1675 1659 169.882 132 0.255769 0.368106
2020-06-08 12:21:39.463237 min lat: 0.0248126 max lat: 1.69083 avg lat: 0.367704
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 15 1726 1711 170.829 208 0.545817 0.367704
41 16 1769 1753 170.608 168 0.0535616 0.368004
42 15 1810 1795 170.541 168 0.373448 0.367582
43 16 1855 1839 170.667 176 0.553561 0.36781
44 16 1897 1881 170.601 168 0.612187 0.368522
45 16 1940 1924 170.631 172 0.270757 0.368138
46 16 1981 1965 170.475 164 0.748291 0.368665
47 16 2022 2006 170.335 164 0.800102 0.367622
48 16 2059 2043 169.87 148 0.166018 0.368026
49 16 2101 2085 169.831 168 0.131844 0.369179
50 16 2149 2133 170.272 192 0.0691953 0.369083
51 16 2193 2177 170.384 176 1.1028 0.368662
52 16 2224 2208 169.493 124 0.0501345 0.368457
53 16 2254 2238 168.552 120 1.97205 0.368903
54 16 2300 2284 168.837 184 0.780315 0.372975
55 16 2339 2323 168.604 156 0.0482763 0.372959
56 16 2386 2370 168.947 188 0.1946 0.37259
57 16 2426 2410 168.789 160 0.885615 0.372047
58 16 2467 2451 168.707 164 0.176191 0.372209
59 16 2510 2494 168.761 172 0.0975932 0.372901
2020-06-08 12:21:59.513225 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.372011
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 2557 2541 169.08 188 0.258079 0.372011
61 16 2599 2583 169.062 168 0.283663 0.371939
62 16 2641 2625 169.045 168 0.154265 0.371796
63 16 2682 2666 168.964 164 0.121294 0.371245
64 16 2719 2703 168.635 148 0.210276 0.372871
65 16 2766 2750 168.932 188 0.301155 0.372831
66 16 2814 2798 169.28 192 0.0746804 0.371501
67 16 2857 2841 169.32 172 0.196694 0.372003
68 15 2905 2890 169.7 196 0.186451 0.370624
69 16 2946 2930 169.56 160 0.282274 0.371114
70 16 2992 2976 169.765 184 1.14253 0.370176
71 16 3031 3015 169.571 156 0.228478 0.371012
72 16 3078 3062 169.826 188 0.209222 0.370815
73 15 3130 3115 170.398 212 0.179196 0.369642
74 16 3173 3157 170.366 168 0.958304 0.369307
75 16 3208 3192 169.961 140 1.01501 0.36968
76 16 3251 3235 169.988 172 0.0769253 0.370113
77 16 3290 3274 169.806 156 0.40686 0.370771
78 16 3341 3325 170.243 204 0.143182 0.369888
79 16 3378 3362 169.962 148 0.944513 0.370147
2020-06-08 12:22:19.524533 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.369985
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 3420 3404 169.935 168 0.807407 0.369985
81 16 3463 3447 169.96 172 0.493126 0.370948
82 16 3504 3488 169.887 164 1.13408 0.370608
83 15 3545 3530 169.853 168 0.24172 0.370816
84 16 3589 3573 169.875 172 0.521379 0.370599
85 15 3632 3617 169.946 176 0.0960671 0.370405
86 16 3676 3660 169.962 172 0.102477 0.370182
87 16 3721 3705 170.077 180 0.516006 0.37001
88 16 3764 3748 170.099 172 0.180662 0.370361
89 16 3811 3795 170.299 188 0.353325 0.369694
90 16 3861 3845 170.629 200 0.6795 0.369402
91 16 3903 3887 170.599 168 0.474503 0.369444
92 16 3946 3930 170.614 172 0.407425 0.369108
93 16 3988 3972 170.586 168 0.106546 0.368901
94 15 4030 4015 170.592 172 0.334292 0.369065
95 16 4079 4063 170.817 192 0.0831617 0.368644
96 16 4116 4100 170.579 148 0.531145 0.3694
97 16 4155 4139 170.421 156 0.888093 0.36943
98 16 4202 4186 170.6 188 0.429109 0.36934
99 16 4239 4223 170.372 148 0.597653 0.369292
2020-06-08 12:22:39.549519 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.36881
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 4292 4276 170.784 212 0.382936 0.36881
101 16 4332 4316 170.677 160 0.0284035 0.36907
102 16 4376 4360 170.729 176 0.57246 0.369122
103 16 4419 4403 170.741 172 0.280385 0.369219
104 16 4462 4446 170.753 172 0.401163 0.369185
105 16 4504 4488 170.726 168 0.779487 0.368732
106 16 4540 4524 170.474 144 1.48542 0.368711
107 16 4587 4571 170.638 188 0.189971 0.369469
108 16 4630 4614 170.65 172 0.0716492 0.369319
109 16 4672 4656 170.626 168 0.169196 0.36955
110 16 4716 4700 170.674 176 0.0684362 0.369269
111 16 4760 4744 170.722 176 0.833629 0.369114
112 16 4799 4783 170.59 156 0.668671 0.369513
113 16 4841 4825 170.567 168 0.168062 0.369518
114 16 4886 4870 170.65 180 0.316925 0.369232
115 15 4932 4917 170.8 188 0.137072 0.369261
116 16 4973 4957 170.705 160 0.0846937 0.369009
117 16 5016 5000 170.71 172 0.52219 0.368954
118 16 5062 5046 170.821 184 0.398768 0.369099
119 15 5092 5077 170.425 124 0.0661291 0.368983
2020-06-08 12:22:59.560617 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.369774
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 5132 5116 170.305 156 0.0683798 0.369774
121 16 5174 5158 170.285 168 0.154458 0.370022
122 16 5208 5192 170.004 136 1.02461 0.370329
123 16 5235 5219 169.5 108 0.129999 0.371054
124 16 5280 5264 169.58 180 0.225555 0.371786
125 15 5316 5301 169.377 148 0.241433 0.371444
126 16 5356 5340 169.27 156 0.22375 0.371694
127 16 5394 5378 169.134 152 1.0247 0.37258
128 16 5433 5417 169.03 156 1.55902 0.37207
129 16 5478 5462 169.112 180 0.043229 0.372815
130 16 5526 5510 169.279 192 0.23312 0.372621
131 16 5566 5550 169.207 160 0.0453516 0.372683
132 16 5608 5592 169.198 168 0.0749937 0.372598
133 16 5651 5635 169.219 172 0.908214 0.372527
134 16 5692 5676 169.172 164 0.12826 0.37275
135 16 5739 5723 169.302 188 0.0424444 0.372397
136 16 5786 5770 169.438 188 0.207119 0.372037
137 16 5828 5812 169.42 168 0.0628769 0.37193
138 16 5864 5848 169.236 144 1.41336 0.37205
139 16 5909 5893 169.313 180 1.58894 0.372159
2020-06-08 12:23:19.621113 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.37275
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 5943 5927 169.076 136 0.0688151 0.37275
141 16 5975 5959 168.784 128 0.254856 0.37341
142 16 6015 5999 168.721 160 0.13998 0.373737
143 15 6059 6044 168.797 180 0.324826 0.373979
144 16 6111 6095 169.038 204 0.0586259 0.373278
145 16 6151 6135 168.973 160 0.176567 0.37326
146 15 6195 6180 169.044 180 0.960333 0.373222
147 16 6237 6221 169.01 164 0.0696781 0.373076
148 16 6277 6261 168.946 160 1.31551 0.373172
149 16 6319 6303 168.939 168 1.06384 0.373341
150 16 6360 6344 168.905 164 1.07922 0.373454
151 16 6397 6381 168.766 148 0.0775141 0.373405
152 16 6437 6421 168.707 160 0.888969 0.37367
153 16 6478 6462 168.676 164 0.0954357 0.373791
154 16 6521 6505 168.679 172 1.08875 0.373622
155 16 6564 6548 168.696 172 0.106057 0.373667
156 16 6613 6597 168.871 196 0.247057 0.373366
157 16 6655 6639 168.865 168 0.152871 0.373457
158 16 6695 6679 168.809 160 0.32451 0.373523
159 16 6736 6720 168.74 164 0.797271 0.373767
2020-06-08 12:23:39.698033 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.373872
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 6776 6760 168.686 160 0.175902 0.373872
161 16 6820 6804 168.726 176 0.430652 0.373903
162 15 6870 6855 168.941 204 0.345304 0.373512
163 15 6916 6901 169.033 184 0.0927639 0.373159
164 15 6961 6946 169.09 180 0.301668 0.373161
165 16 7014 6998 169.324 208 0.39952 0.372602
166 16 7059 7043 169.388 180 0.272027 0.37244
167 16 7106 7090 169.498 188 0.346263 0.372241
168 15 7153 7138 169.63 192 0.303535 0.372102
169 16 7200 7184 169.715 184 0.792886 0.371806
170 16 7237 7221 169.587 148 0.397384 0.371751
171 16 7280 7264 169.601 172 0.100315 0.371644
172 16 7317 7301 169.475 148 0.642609 0.371243
173 16 7354 7338 169.351 148 0.151381 0.37173
174 16 7397 7381 169.366 172 0.299631 0.372604
175 16 7443 7427 169.449 184 0.908767 0.372409
176 16 7483 7467 169.396 160 0.385455 0.372517
177 16 7525 7509 169.385 168 0.407758 0.37254
178 15 7567 7552 169.398 172 0.273753 0.372301
179 16 7617 7601 169.54 196 0.110554 0.372117
2020-06-08 12:23:59.734115 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.371832
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 7662 7646 169.596 180 0.2817 0.371832
181 16 7705 7689 169.609 172 0.269791 0.37186
182 16 7745 7729 169.556 160 0.0923176 0.372007
183 16 7784 7768 169.482 156 0.631796 0.372254
184 16 7820 7804 169.343 144 0.713916 0.372468
185 16 7861 7845 169.314 164 0.319438 0.372665
186 15 7907 7892 169.412 188 0.298928 0.372459
187 16 7949 7933 169.383 164 0.111257 0.372334
188 16 7992 7976 169.397 172 0.111834 0.372168
189 16 8032 8016 169.347 160 0.0932603 0.372001
190 16 8070 8054 169.256 152 1.72287 0.372328
191 16 8092 8076 168.831 88 0.049547 0.372895
192 16 8119 8103 168.514 108 0.0920323 0.373235
193 16 8156 8140 168.408 148 0.0856176 0.374086
194 16 8199 8183 168.426 172 0.116559 0.374232
195 16 8237 8221 168.337 152 0.215625 0.374518
196 16 8278 8262 168.315 164 0.0801257 0.374161
197 16 8320 8304 168.312 168 0.233133 0.374362
198 16 8358 8342 168.23 152 0.211434 0.374398
199 16 8392 8376 168.068 136 0.0805567 0.374965
2020-06-08 12:24:19.749046 min lat: 0.0230592 max lat: 2.31143 avg lat: 0.375312
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 8440 8424 168.186 192 0.893368 0.375312
201 16 8484 8468 168.221 176 0.306854 0.375168
202 16 8524 8508 168.144 160 0.13247 0.374973
203 16 8560 8544 168.025 144 0.148057 0.375034
204 16 8591 8575 167.81 124 0.10823 0.375202
205 16 8624 8608 167.635 132 0.0907727 0.375628
206 16 8660 8644 167.519 144 0.0740024 0.375771
207 16 8703 8687 167.541 172 0.292113 0.37628
208 16 8750 8734 167.639 188 0.120207 0.376575
209 16 8790 8774 167.601 160 0.300057 0.376465
210 15 8838 8823 167.733 196 0.259945 0.376447
211 16 8889 8873 167.885 200 0.297212 0.37591
212 15 8936 8921 167.989 192 0.0924405 0.375739
213 15 8981 8966 168.044 180 0.756622 0.375544
214 16 9024 9008 168.043 168 0.742799 0.375558
215 16 9069 9053 168.099 180 0.0969489 0.375406
216 15 9114 9099 168.172 184 0.507433 0.37537
217 16 9159 9143 168.208 176 0.604682 0.375247
218 16 9195 9179 168.097 144 0.075566 0.37535
219 16 9233 9217 168.02 152 0.459101 0.375446
2020-06-08 12:24:39.826809 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.375683
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 9274 9258 168.001 164 0.709525 0.375683
221 16 9316 9300 168.001 168 0.105815 0.375641
222 15 9356 9341 167.982 164 0.103179 0.375719
223 16 9402 9386 168.036 180 0.0976973 0.375358
224 15 9446 9431 168.087 180 0.220347 0.375707
225 15 9492 9477 168.157 184 1.01907 0.375363
226 16 9531 9515 168.086 152 0.376302 0.375362
227 16 9574 9558 168.103 172 0.937817 0.375517
228 16 9613 9597 168.05 156 0.660098 0.375498
229 15 9661 9646 168.163 196 0.818207 0.375485
230 16 9705 9689 168.179 172 0.0882443 0.375519
231 16 9755 9739 168.315 200 0.171719 0.375227
232 16 9802 9786 168.4 188 0.136964 0.374891
233 16 9844 9828 168.398 168 0.237665 0.374939
234 16 9887 9871 168.413 172 0.295812 0.374974
235 16 9932 9916 168.462 180 0.0674042 0.374769
236 16 9976 9960 168.494 176 0.132963 0.374399
237 16 10019 10003 168.509 172 0.0382068 0.374833
238 16 10062 10046 168.523 172 0.753091 0.374687
239 16 10106 10090 168.546 176 0.428319 0.374695
2020-06-08 12:24:59.860167 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.374344
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 10155 10139 168.66 196 0.0943221 0.374344
241 15 10197 10182 168.647 172 0.20105 0.374382
242 16 10234 10218 168.545 144 1.05015 0.374511
243 15 10279 10264 168.608 184 0.0601367 0.374424
244 16 10322 10306 168.605 168 0.367138 0.374263
245 16 10363 10347 168.586 164 0.0847016 0.374395
246 16 10401 10385 168.518 152 0.124863 0.374697
247 16 10437 10421 168.417 144 0.127071 0.374713
248 16 10486 10470 168.528 196 0.0701429 0.374565
249 16 10529 10513 168.541 172 0.159608 0.374656
250 16 10567 10551 168.474 152 0.450094 0.37483
251 16 10606 10590 168.424 156 0.646874 0.374853
252 16 10650 10634 168.454 176 1.07673 0.374889
253 16 10682 10666 168.293 128 0.872949 0.375043
254 16 10727 10711 168.339 180 0.0961278 0.374819
255 16 10771 10755 168.369 176 0.0948567 0.374893
256 16 10812 10796 168.352 164 0.0991479 0.374972
257 16 10848 10832 168.257 144 1.24371 0.3752
258 16 10893 10877 168.303 180 0.657484 0.37519
259 16 10935 10919 168.301 168 0.100677 0.375137
2020-06-08 12:25:19.917240 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.375227
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 10981 10965 168.357 184 0.0463872 0.375227
261 15 11025 11010 168.401 180 0.3973 0.375052
262 16 11072 11056 168.459 184 0.338258 0.374912
263 16 11117 11101 168.502 180 0.595439 0.374865
264 16 11154 11138 168.424 148 0.29759 0.374847
265 16 11191 11175 168.346 148 0.157838 0.375024
266 16 11232 11216 168.33 164 0.0774951 0.375107
267 16 11284 11268 168.475 208 0.0569242 0.374752
268 16 11319 11303 168.369 140 0.111172 0.374761
269 16 11355 11339 168.277 144 0.500632 0.375068
270 16 11397 11381 168.276 168 0.218784 0.375335
271 16 11443 11427 168.334 184 0.0822048 0.37517
272 16 11490 11474 168.406 188 0.361562 0.374994
273 16 11536 11520 168.463 184 0.499848 0.374917
274 16 11584 11568 168.548 192 0.405607 0.374757
275 16 11632 11616 168.633 192 0.921353 0.37453
276 15 11672 11657 168.615 164 0.471223 0.374392
277 16 11713 11697 168.583 160 0.920939 0.374607
278 16 11755 11739 168.581 168 0.10323 0.374535
279 16 11796 11780 168.564 164 0.0690183 0.374355
2020-06-08 12:25:39.937611 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.374501
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 11839 11823 168.576 172 0.0729308 0.374501
281 16 11878 11862 168.531 156 0.62186 0.374631
282 16 11921 11905 168.543 172 0.173199 0.374627
283 16 11962 11946 168.521 164 0.613569 0.374548
284 15 12002 11987 168.505 164 0.409955 0.37471
285 16 12045 12029 168.503 168 0.507504 0.374697
286 16 12085 12069 168.46 160 0.142108 0.374831
287 16 12131 12115 168.499 184 0.423421 0.374676
288 16 12181 12165 168.608 200 0.349587 0.374549
289 16 12234 12218 168.757 212 0.434835 0.374189
290 16 12276 12260 168.755 168 0.0687152 0.37401
291 16 12320 12304 168.78 176 0.0388994 0.374132
292 16 12361 12345 168.763 164 0.926596 0.374063
293 15 12410 12395 168.869 200 0.491036 0.373951
294 16 12449 12433 168.811 152 0.35548 0.373944
295 16 12494 12478 168.846 180 0.14995 0.373931
296 16 12537 12521 168.856 172 0.100294 0.373855
297 16 12580 12564 168.867 172 0.0870731 0.373873
298 16 12624 12608 168.89 176 0.179845 0.373846
299 16 12672 12656 168.966 192 0.158471 0.373711
2020-06-08 12:26:00.010689 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373596
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 12721 12705 169.056 196 0.090572 0.373596
301 16 12767 12751 169.092 184 0.058073 0.373406
302 16 12810 12794 169.095 172 1.22799 0.373267
303 16 12854 12838 169.118 176 0.280225 0.37343
304 16 12897 12881 169.127 172 0.271161 0.373415
305 15 12939 12924 169.135 172 0.788942 0.373355
306 15 12977 12962 169.075 152 0.732941 0.373458
307 15 13019 13004 169.048 168 0.0439113 0.37342
308 16 13066 13050 169.095 184 1.24987 0.373496
309 16 13110 13094 169.118 176 0.284408 0.373332
310 16 13148 13132 169.062 152 0.228812 0.373455
311 16 13191 13175 169.072 172 0.499808 0.37354
312 16 13240 13224 169.158 196 0.165439 0.373414
313 16 13284 13268 169.18 176 0.0820455 0.373294
314 16 13332 13316 169.252 192 0.525569 0.373007
315 16 13372 13356 169.222 160 0.0941317 0.37301
316 16 13416 13400 169.243 176 0.19094 0.372979
317 16 13455 13439 169.201 156 0.368214 0.373151
318 16 13494 13478 169.159 156 1.40004 0.373129
319 15 13537 13522 169.178 176 0.0604933 0.373176
2020-06-08 12:26:20.110826 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373125
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 15 13578 13563 169.162 164 0.149702 0.373125
321 16 13624 13608 169.195 180 0.371208 0.373004
322 16 13672 13656 169.259 192 0.197198 0.372899
323 16 13716 13700 169.279 176 0.701214 0.372868
324 16 13755 13739 169.238 156 0.854059 0.373033
325 16 13799 13783 169.259 176 0.0956063 0.372985
326 16 13839 13823 169.23 160 1.27962 0.372944
327 15 13879 13864 169.213 164 0.026284 0.372841
328 16 13927 13911 169.271 188 0.0643772 0.372945
329 16 13962 13946 169.18 140 0.302032 0.373101
330 16 14001 13985 169.136 156 1.51292 0.373174
331 16 14037 14021 169.059 144 0.470913 0.373425
332 16 14085 14069 169.128 192 0.921336 0.373352
333 16 14123 14107 169.076 152 0.534653 0.373415
334 15 14162 14147 169.047 160 1.42114 0.373338
335 16 14206 14190 169.056 172 0.99175 0.373491
336 16 14248 14232 169.053 168 1.03147 0.373408
337 16 14297 14281 169.132 196 0.194408 0.37337
338 15 14335 14320 169.092 156 0.104599 0.373343
339 16 14375 14359 169.053 156 1.12634 0.373467
2020-06-08 12:26:40.153864 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373552
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 15 14416 14401 169.049 168 0.315563 0.373552
341 16 14465 14449 169.115 192 0.284339 0.373345
342 15 14503 14488 169.072 156 0.190026 0.3734
343 16 14548 14532 169.093 176 0.329976 0.373403
344 16 14588 14572 169.048 160 0.868697 0.373524
345 16 14626 14610 168.998 152 0.197223 0.373709
346 16 14671 14655 169.029 180 0.0527044 0.373635
347 15 14714 14699 169.048 176 0.264045 0.373598
348 16 14759 14743 169.066 176 0.287995 0.373565
349 16 14808 14792 169.143 196 0.0734263 0.373333
350 16 14850 14834 169.139 168 0.845664 0.373225
351 16 14892 14876 169.136 168 0.347596 0.373362
352 16 14933 14917 169.121 164 0.553563 0.373443
353 16 14975 14959 169.118 168 0.388754 0.373438
354 16 15011 14995 169.046 144 1.08782 0.37342
355 16 15052 15036 169.023 164 0.230196 0.373354
356 16 15101 15085 169.081 196 0.331586 0.373464
357 16 15149 15133 169.131 192 0.155034 0.373396
358 16 15191 15175 169.128 168 0.950143 0.373327
359 16 15239 15223 169.191 192 0.254462 0.373304
2020-06-08 12:27:00.300879 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373238
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 15281 15265 169.188 168 0.222089 0.373238
361 16 15322 15306 169.172 164 0.165509 0.373221
362 16 15366 15350 169.189 176 0.630044 0.373239
363 16 15414 15398 169.251 192 1.1549 0.373111
364 16 15454 15438 169.222 160 0.33455 0.373049
365 16 15494 15478 169.197 160 0.0976162 0.373173
366 16 15531 15515 169.139 148 0.400027 0.373355
367 16 15575 15559 169.158 176 0.890982 0.373261
368 16 15610 15594 169.079 140 0.334188 0.373346
369 16 15648 15632 169.033 152 0.696061 0.373407
370 16 15690 15674 169.03 168 0.301545 0.373403
371 16 15734 15718 169.048 176 0.297896 0.37346
372 16 15773 15757 169.013 156 0.102745 0.373609
373 16 15811 15795 168.959 152 1.2442 0.373519
374 16 15850 15834 168.924 156 0.266913 0.373633
375 16 15887 15871 168.868 148 0.154025 0.373804
376 16 15923 15907 168.801 144 1.56844 0.373869
377 16 15967 15951 168.82 176 0.438185 0.374101
378 16 16009 15993 168.818 168 0.434633 0.374083
379 16 16058 16042 168.889 196 0.288303 0.373905
2020-06-08 12:27:20.341162 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373764
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 16106 16090 168.95 192 0.102851 0.373764
381 16 16143 16127 168.895 148 0.59253 0.373856
382 16 16187 16171 168.913 176 0.0711049 0.373791
383 16 16233 16217 168.953 184 0.307945 0.373758
384 16 16274 16258 168.94 164 0.997311 0.373781
385 16 16315 16299 168.927 164 0.0534854 0.37376
386 15 16357 16342 168.934 172 1.30778 0.373766
387 16 16402 16386 168.952 176 0.0273739 0.373742
388 16 16446 16430 168.97 176 0.0823523 0.373643
389 16 16486 16470 168.947 160 0.236351 0.373601
390 16 16529 16513 168.954 172 0.0866406 0.373601
391 16 16573 16557 168.971 176 0.0804077 0.373588
392 16 16605 16589 168.867 128 0.659212 0.373787
393 16 16649 16633 168.882 176 0.324744 0.373891
394 16 16691 16675 168.879 168 0.109391 0.373982
395 16 16735 16719 168.88 176 0.371198 0.373865
396 16 16781 16765 168.918 184 0.250705 0.37384
397 16 16823 16807 168.915 168 0.218924 0.37385
398 15 16866 16851 168.932 176 0.284068 0.373831
399 16 16911 16895 168.95 176 0.371854 0.373711
2020-06-08 12:27:40.400625 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373788
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 16948 16932 168.897 148 0.0891069 0.373788
401 16 16980 16964 168.796 128 0.062622 0.374021
402 16 17024 17008 168.813 176 0.0428555 0.373992
403 16 17069 17053 168.84 180 0.357778 0.37399
404 16 17111 17095 168.836 168 0.156089 0.373952
405 15 17156 17141 168.873 184 0.0988409 0.373801
406 16 17194 17178 168.822 148 0.101354 0.373972
407 16 17240 17224 168.859 184 0.480111 0.373999
408 16 17280 17264 168.837 160 0.334585 0.374053
409 16 17329 17313 168.904 196 0.45556 0.373887
410 16 17368 17352 168.871 156 1.04888 0.373839
411 16 17411 17395 168.877 172 0.0679636 0.373787
412 16 17450 17434 168.846 156 1.07292 0.373792
413 16 17487 17471 168.796 148 1.17226 0.373957
414 16 17530 17514 168.803 172 1.16831 0.374032
415 16 17575 17559 168.83 180 0.0991521 0.373938
416 16 17602 17586 168.683 108 1.48129 0.373972
417 16 17634 17618 168.586 128 0.0961447 0.374313
418 16 17677 17661 168.593 172 0.285313 0.374612
419 15 17719 17704 168.6 172 0.807376 0.374583
2020-06-08 12:28:00.423428 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.374479
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 17765 17749 168.627 180 0.11501 0.374479
421 16 17805 17789 168.607 160 1.00096 0.374543
422 16 17847 17831 168.605 168 0.112573 0.37464
423 16 17893 17877 168.641 184 0.460584 0.374563
424 16 17940 17924 168.687 188 0.0542812 0.374299
425 16 17982 17966 168.683 168 0.147957 0.374244
426 16 18022 18006 168.663 160 0.0830238 0.374335
427 16 18071 18055 168.727 196 0.094898 0.374253
428 16 18115 18099 168.743 176 1.00264 0.374132
429 16 18161 18145 168.779 184 0.260795 0.374049
430 16 18190 18174 168.656 116 0.439204 0.374323
431 15 18236 18221 168.701 188 0.0688787 0.37434
432 16 18283 18267 168.736 184 0.10751 0.374236
433 16 18331 18315 168.789 192 0.304596 0.374142
434 15 18372 18357 168.786 168 0.0558009 0.374132
435 16 18413 18397 168.766 160 1.01258 0.374165
436 16 18456 18440 168.766 172 0.443794 0.374169
437 16 18491 18475 168.698 140 0.382729 0.374223
438 16 18534 18518 168.691 172 0.462145 0.374213
439 15 18582 18567 168.753 196 0.271437 0.374163
2020-06-08 12:28:20.501147 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.374149
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 15 18623 18608 168.741 164 0.090205 0.374149
441 16 18665 18649 168.73 164 0.593713 0.374186
442 16 18709 18693 168.746 176 0.379424 0.374092
443 16 18750 18734 168.733 164 0.0899103 0.374166
444 15 18786 18771 168.678 148 0.611534 0.374258
445 16 18832 18816 168.703 180 0.158367 0.374262
446 16 18885 18869 168.8 212 0.80647 0.374105
447 15 18923 18908 168.771 156 0.099704 0.37403
448 15 18967 18952 168.784 176 0.0231707 0.374053
449 16 19014 18998 168.818 184 0.0553393 0.373923
450 16 19046 19030 168.721 128 0.09116 0.374001
451 16 19096 19080 168.79 200 0.031907 0.373984
452 16 19134 19118 168.753 152 0.0739179 0.374068
453 16 19169 19153 168.689 140 0.232571 0.374115
454 16 19213 19197 168.705 176 0.418458 0.374043
455 16 19254 19238 168.694 164 0.277625 0.374144
456 16 19302 19286 168.743 192 0.200266 0.37418
457 16 19358 19342 168.863 224 0.0717209 0.373924
458 16 19402 19386 168.878 176 0.126194 0.373794
459 16 19442 19426 168.859 160 0.315356 0.37374
2020-06-08 12:28:40.572353 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373873
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 19485 19469 168.865 172 1.10045 0.373873
461 16 19525 19509 168.837 160 0.447995 0.373842
462 16 19564 19548 168.809 156 0.439457 0.373913
463 16 19600 19584 168.755 144 0.0422886 0.374122
464 16 19642 19626 168.753 168 0.275161 0.374125
465 16 19690 19674 168.802 192 0.0649647 0.374009
466 16 19741 19725 168.876 204 0.101011 0.373934
467 16 19794 19778 168.968 212 0.100933 0.373578
468 15 19831 19816 168.931 152 0.070962 0.373754
469 16 19876 19860 168.945 176 0.0472632 0.373631
470 16 19918 19902 168.939 168 0.272227 0.373537
471 16 19957 19941 168.911 156 1.47847 0.373679
472 16 20003 19987 168.943 184 0.0821059 0.373639
473 16 20044 20028 168.932 164 1.12177 0.37375
474 16 20088 20072 168.942 176 0.0653919 0.373706
475 16 20129 20113 168.932 164 0.803102 0.373695
476 16 20171 20155 168.929 168 0.424077 0.37381
477 16 20217 20201 168.96 184 0.35976 0.373763
478 16 20255 20239 168.925 152 0.0528211 0.373753
479 16 20300 20284 168.948 180 0.0582426 0.373685
2020-06-08 12:29:00.652451 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373749
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 20341 20325 168.934 164 1.11283 0.373749
481 16 20380 20364 168.907 156 0.585802 0.373812
482 16 20422 20406 168.905 168 0.0464579 0.373784
483 16 20465 20449 168.912 172 0.626965 0.373749
484 16 20504 20488 168.869 156 0.123267 0.373827
485 16 20553 20537 168.925 196 0.315675 0.373704
486 16 20599 20583 168.955 184 1.24376 0.373673
487 15 20640 20625 168.953 168 0.0696686 0.373654
488 16 20689 20673 169 192 0.0469364 0.37358
489 16 20728 20712 168.973 156 0.0309123 0.373647
490 16 20769 20753 168.963 164 0.113886 0.37361
491 16 20816 20800 169.002 188 1.5052 0.373531
492 16 20856 20840 168.983 160 0.0744769 0.373628
493 16 20896 20880 168.964 160 0.0434707 0.373596
494 16 20936 20920 168.946 160 0.089939 0.373704
495 16 20972 20956 168.895 144 0.318724 0.373779
496 15 21009 20994 168.859 152 0.0729679 0.37393
497 16 21049 21033 168.834 156 0.0760474 0.373911
498 16 21089 21073 168.816 160 0.0967915 0.373925
499 16 21125 21109 168.766 144 0.235737 0.373987
2020-06-08 12:29:20.713789 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.374068
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 21170 21154 168.788 180 0.70099 0.374068
501 16 21209 21193 168.763 156 0.0473979 0.374112
502 16 21249 21233 168.745 160 0.0695431 0.374115
503 16 21287 21271 168.706 152 0.175888 0.374247
504 16 21337 21321 168.768 200 0.275681 0.374105
505 16 21371 21355 168.699 136 0.820066 0.374221
506 16 21414 21398 168.705 172 0.395828 0.374157
507 16 21452 21436 168.673 152 0.0561993 0.374271
508 16 21505 21489 168.74 212 0.109654 0.37416
509 15 21549 21534 168.762 180 0.0586498 0.374031
510 16 21590 21574 168.744 160 0.472577 0.374101
511 16 21630 21614 168.727 160 0.479988 0.374137
512 16 21669 21653 168.702 156 0.250771 0.374201
513 16 21701 21685 168.623 128 0.0358874 0.374358
514 16 21750 21734 168.676 196 0.947301 0.374273
515 16 21795 21779 168.696 180 0.0787879 0.374205
516 15 21837 21822 168.702 172 0.0434811 0.374192
517 16 21888 21872 168.762 200 0.390348 0.374104
518 16 21932 21916 168.775 176 0.0632455 0.373985
519 16 21977 21961 168.797 180 0.0785645 0.37388
2020-06-08 12:29:40.812520 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373921
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 22028 22012 168.864 204 0.0770314 0.373921
521 15 22077 22062 168.923 200 0.0860754 0.373745
522 16 22127 22111 168.974 196 0.0544964 0.373598
523 16 22172 22156 168.995 180 0.103899 0.373551
524 16 22211 22195 168.971 156 0.266032 0.373505
525 16 22260 22244 169.021 196 0.598512 0.373534
526 16 22311 22295 169.087 204 0.13551 0.373386
527 16 22355 22339 169.1 176 1.12599 0.373324
528 16 22398 22382 169.105 172 0.103698 0.373311
529 16 22436 22420 169.072 152 0.0599454 0.373317
530 16 22471 22455 169.018 140 0.652483 0.373402
531 16 22516 22500 169.038 180 0.0859822 0.373477
532 15 22553 22538 169.005 152 0.12833 0.373531
533 16 22590 22574 168.957 144 0.270461 0.373631
534 16 22636 22620 168.985 184 0.101993 0.373594
535 16 22676 22660 168.968 160 0.720174 0.373609
536 16 22711 22695 168.914 140 0.0571462 0.373761
537 16 22751 22735 168.897 160 0.0430687 0.37376
538 16 22788 22772 168.858 148 0.0985609 0.373844
539 16 22837 22821 168.907 196 0.0666636 0.373742
2020-06-08 12:30:00.866806 min lat: 0.0230414 max lat: 2.43145 avg lat: 0.373761
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 15 22880 22865 168.911 176 0.728356 0.373761
541 16 22931 22915 168.969 200 0.425171 0.373605
542 16 22968 22952 168.93 148 1.00427 0.373727
543 15 23012 22997 168.95 180 0.623734 0.373699
544 16 23059 23043 168.977 184 0.559206 0.373701
545 16 23105 23089 169.005 184 0.0540311 0.373539
546 16 23140 23124 168.952 140 0.406238 0.373631
547 16 23187 23171 168.978 188 0.208259 0.373622
548 16 23225 23209 168.947 152 0.310856 0.373648
549 16 23267 23251 168.945 168 0.0596425 0.373595
550 16 23314 23298 168.98 188 0.360297 0.373586
551 15 23364 23349 169.041 204 0.394656 0.373417
552 16 23412 23396 169.075 188 0.392099 0.373443
553 16 23449 23433 169.037 148 0.316882 0.373461
554 16 23487 23471 169.003 152 1.37468 0.373478
555 16 23526 23510 168.979 156 0.952755 0.373527
556 16 23570 23554 168.992 176 0.699079 0.373596
557 16 23617 23601 169.026 188 0.105293 0.373513
558 16 23655 23639 168.995 152 0.046516 0.373511
559 16 23691 23675 168.951 144 0.490041 0.373625
2020-06-08 12:30:20.919319 min lat: 0.0230414 max lat: 2.43145 avg lat: 0.373597
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 23736 23720 168.97 180 1.10394 0.373597
561 16 23783 23767 169.004 188 0.100392 0.373506
562 16 23825 23809 169.002 168 0.149469 0.373554
563 16 23871 23855 169.029 184 0.0914753 0.373522
564 16 23915 23899 169.041 176 0.854179 0.373463
565 15 23952 23937 169.011 152 0.581261 0.373546
566 16 23997 23981 169.023 176 0.157163 0.373489
567 16 24039 24023 169.021 168 0.0542004 0.373521
568 15 24082 24067 169.031 176 0.350725 0.373455
569 16 24117 24101 168.973 136 0.544972 0.373547
570 15 24161 24146 168.992 180 0.0691027 0.373577
571 15 24203 24188 168.99 168 0.365866 0.373523
Total time run: 573.097
Total reads made: 24206
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 168.949
Average IOPS: 42
Stddev IOPS: 4.60581
Max IOPS: 56
Min IOPS: 22
Average Latency(s): 0.373816
Max latency(s): 2.43145
Min latency(s): 0.0230414
Total time run: 573.097
Total reads made: 24206
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 168.949
Average IOPS: 42
Stddev IOPS: 4.60581
Max IOPS: 56
Min IOPS: 22
Average Latency(s): 0.373816
Max latency(s): 2.43145
Min latency(s): 0.0230414
Random Read Test, pool with 16 PGs
podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 rand | tee --append $(date +%F)-rados-test-read-rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 54 38 151.941 152 0.0560878 0.301558
2 15 93 78 155.961 160 0.0508687 0.334311
3 16 139 123 163.965 180 0.0970996 0.344061
4 16 182 166 165.969 172 0.0827236 0.351308
5 16 228 212 169.57 184 0.104305 0.347256
6 16 265 249 165.922 148 0.793498 0.354993
7 16 310 294 167.929 180 0.877642 0.361335
8 16 347 331 165.436 148 0.606384 0.3668
9 16 395 379 168.384 192 0.0695003 0.359143
10 16 440 424 169.542 180 0.127387 0.352231
11 16 486 470 170.854 184 0.208886 0.364528
12 16 532 516 171.947 184 0.100045 0.358566
13 16 579 563 173.18 188 0.707554 0.359704
14 15 620 605 172.809 168 0.294718 0.358
15 15 664 649 173.02 176 0.0703784 0.354072
16 16 710 694 173.455 180 0.0691815 0.355399
17 16 749 733 172.427 156 0.631627 0.35975
18 16 787 771 171.291 152 0.102011 0.362699
19 16 831 815 171.538 176 0.194053 0.363742
2020-06-08 12:34:24.728113 min lat: 0.00855444 max lat: 1.74713 avg lat: 0.362494
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 877 861 172.16 184 0.778364 0.362494
21 16 921 905 172.337 176 0.0567624 0.362528
22 16 974 958 174.132 212 0.153594 0.359249
23 16 1016 1000 173.856 168 0.256497 0.358638
24 16 1054 1038 172.945 152 0.212232 0.361255
25 16 1093 1077 172.266 156 0.0840251 0.361163
26 15 1134 1119 172.101 168 0.410222 0.362375
27 16 1175 1159 171.652 160 0.886293 0.362714
28 16 1213 1197 170.949 152 0.15387 0.36495
29 16 1261 1245 171.674 192 0.656081 0.364763
30 16 1298 1282 170.885 148 1.39393 0.364236
31 16 1348 1332 171.823 200 0.385381 0.362314
32 16 1385 1369 171.078 148 0.0715221 0.365919
33 16 1430 1414 171.345 180 0.216946 0.365676
34 16 1471 1455 171.126 164 0.0916466 0.365462
35 16 1513 1497 171.035 168 0.170297 0.365562
36 15 1553 1538 170.839 164 0.568043 0.367149
37 16 1595 1579 170.618 164 0.203232 0.367024
38 16 1637 1621 170.549 168 0.10333 0.366906
39 15 1673 1658 169.966 148 1.01188 0.368589
2020-06-08 12:34:44.743051 min lat: 0.00855444 max lat: 1.74713 avg lat: 0.369317
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 15 1717 1702 170.117 176 0.237889 0.369317
41 15 1761 1746 170.258 176 0.21963 0.368777
42 16 1807 1791 170.484 180 0.716421 0.368889
43 16 1851 1835 170.611 176 0.492626 0.368246
44 16 1893 1877 170.55 168 0.868193 0.368495
45 16 1933 1917 170.315 160 0.658381 0.368697
46 16 1971 1955 169.916 152 0.884089 0.369789
47 16 2016 2000 170.13 180 0.0755125 0.368881
48 16 2060 2044 170.251 176 0.105925 0.36842
49 15 2105 2090 170.531 184 1.4859 0.368157
50 15 2146 2131 170.397 164 0.295478 0.368033
51 16 2191 2175 170.507 176 0.13361 0.368038
52 16 2236 2220 170.689 180 0.0560121 0.36767
53 16 2281 2265 170.859 180 0.0925873 0.368093
54 16 2324 2308 170.879 172 0.165669 0.367542
55 16 2374 2358 171.408 200 1.51535 0.367335
56 15 2414 2399 171.276 164 0.437569 0.366908
57 15 2454 2439 171.078 160 0.682976 0.367406
58 16 2499 2483 171.16 176 0.475668 0.367909
59 16 2545 2529 171.377 184 0.390914 0.367125
2020-06-08 12:35:04.751358 min lat: 0.00855444 max lat: 1.78207 avg lat: 0.366872
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 2589 2573 171.454 176 0.172929 0.366872
61 16 2629 2613 171.265 160 0.0765162 0.366603
62 16 2670 2654 171.148 164 0.21768 0.366755
63 15 2707 2692 170.844 152 1.07602 0.36782
64 16 2748 2732 170.674 160 0.288796 0.368158
65 16 2785 2769 170.325 148 0.392669 0.368874
66 16 2828 2812 170.35 172 0.14012 0.369346
67 16 2874 2858 170.553 184 0.0623601 0.368788
68 16 2923 2907 170.927 196 0.078374 0.368413
69 15 2966 2951 171 176 0.0939366 0.367736
70 15 3004 2989 170.728 152 0.785511 0.368216
71 16 3050 3034 170.858 180 0.322062 0.36865
72 15 3088 3073 170.647 156 0.0633324 0.368374
73 16 3130 3114 170.556 164 0.318614 0.368543
74 16 3175 3159 170.683 180 0.0571033 0.368064
75 15 3215 3200 170.594 164 0.450791 0.368223
76 16 3264 3248 170.873 192 0.28818 0.368458
77 16 3311 3295 171.093 188 0.539617 0.368093
78 16 3352 3336 171.002 164 0.943884 0.367812
79 16 3392 3376 170.862 160 0.0803677 0.367359
2020-06-08 12:35:24.757886 min lat: 0.00855444 max lat: 1.78207 avg lat: 0.367158
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 3438 3422 171.026 184 0.153218 0.367158
81 16 3489 3473 171.433 204 0.0991558 0.367021
82 16 3537 3521 171.683 192 0.417421 0.367092
83 15 3582 3567 171.831 184 0.13382 0.366716
84 15 3618 3603 171.498 144 0.211952 0.367122
85 16 3663 3647 171.551 176 0.190954 0.366645
86 16 3701 3685 171.323 152 1.22431 0.367533
87 15 3747 3732 171.514 188 0.326937 0.367436
88 16 3795 3779 171.701 188 0.242663 0.366849
89 15 3839 3824 171.794 180 0.822329 0.366739
90 16 3878 3862 171.574 152 0.225071 0.366788
91 16 3919 3903 171.491 164 1.09846 0.36668
92 16 3959 3943 171.366 160 0.220888 0.367126
93 16 4003 3987 171.415 176 0.635968 0.367697
94 16 4051 4035 171.634 192 0.22924 0.367152
95 16 4093 4077 171.595 168 0.228047 0.367052
96 15 4138 4123 171.724 184 0.151202 0.366646
97 15 4177 4162 171.562 156 0.393589 0.366684
98 15 4221 4206 171.607 176 0.606391 0.367675
99 15 4268 4253 171.709 188 0.531056 0.367072
2020-06-08 12:35:44.798371 min lat: 0.00855444 max lat: 1.84405 avg lat: 0.366413
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 4312 4296 171.711 172 0.1059 0.366413
101 16 4356 4340 171.751 176 1.02397 0.366625
102 16 4406 4390 172.028 200 0.160712 0.366399
103 16 4444 4428 171.826 152 1.1547 0.366429
104 16 4485 4469 171.751 164 0.101778 0.366449
105 16 4530 4514 171.829 180 0.280699 0.366633
106 15 4572 4557 171.83 172 0.282829 0.366736
107 15 4617 4602 171.906 180 0.706571 0.366768
108 15 4668 4653 172.201 204 0.105208 0.366122
109 15 4706 4691 172.013 152 0.210905 0.36567
110 16 4742 4726 171.722 140 0.129857 0.366329
111 16 4785 4769 171.722 172 0.228014 0.366765
112 16 4831 4815 171.83 184 0.146259 0.366835
113 15 4875 4860 171.898 180 0.235777 0.366896
114 16 4916 4900 171.794 160 0.706492 0.366817
115 16 4960 4944 171.83 176 0.580098 0.366866
116 16 5005 4989 171.9 180 0.151648 0.366682
117 16 5051 5035 172.003 184 0.158664 0.366572
118 16 5091 5075 171.901 160 0.198732 0.366619
119 16 5129 5113 171.734 152 0.659346 0.366775
2020-06-08 12:36:04.814978 min lat: 0.00855444 max lat: 1.84405 avg lat: 0.367176
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 5167 5151 171.569 152 0.199474 0.367176
121 15 5208 5193 171.539 168 0.466834 0.367381
122 16 5251 5235 171.51 168 0.599882 0.367319
123 15 5291 5276 171.449 164 0.102313 0.367412
124 16 5332 5316 171.357 160 0.0881955 0.367465
125 16 5370 5354 171.202 152 0.101228 0.367561
126 16 5412 5396 171.176 168 0.348982 0.367732
127 16 5449 5433 170.993 148 0.0733555 0.36795
128 15 5491 5476 170.997 172 0.516966 0.3681
129 15 5530 5515 170.881 156 0.0965617 0.368581
130 15 5570 5555 170.797 160 0.133512 0.368795
131 16 5607 5591 170.592 144 0.0667129 0.369345
132 16 5641 5625 170.33 136 0.111097 0.370059
133 15 5675 5660 170.102 140 0.0685105 0.370463
134 16 5720 5704 170.146 176 0.0961603 0.370605
135 16 5767 5751 170.278 188 0.660571 0.370415
136 15 5813 5798 170.408 188 0.4856 0.370251
137 15 5854 5839 170.361 164 0.0978587 0.370041
138 16 5892 5876 170.199 148 0.0902359 0.370212
139 16 5933 5917 170.154 164 0.188452 0.370492
2020-06-08 12:36:24.821230 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.370804
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 5974 5958 170.11 164 0.503391 0.370804
141 16 6017 6001 170.123 172 0.322609 0.370546
142 16 6059 6043 170.108 168 0.136321 0.370721
143 16 6110 6094 170.345 204 0.49758 0.3703
144 16 6156 6140 170.439 184 0.105002 0.369923
145 15 6203 6188 170.588 192 0.0709188 0.369495
146 16 6249 6233 170.652 180 0.0909963 0.369342
147 15 6293 6278 170.715 180 0.104871 0.369187
148 16 6333 6317 170.616 156 0.0888819 0.369408
149 15 6376 6361 170.652 176 0.357895 0.369484
150 16 6422 6406 170.714 180 0.0842686 0.369438
151 16 6468 6452 170.801 184 0.088993 0.369056
152 16 6514 6498 170.888 184 0.106771 0.368836
153 16 6552 6536 170.764 152 0.274565 0.368838
154 16 6595 6579 170.77 172 0.0927322 0.36896
155 16 6634 6618 170.675 156 0.0970653 0.36905
156 16 6673 6657 170.581 156 0.0825291 0.369279
157 16 6716 6700 170.59 172 0.281331 0.369147
158 16 6758 6742 170.573 168 0.178415 0.36942
159 15 6798 6783 170.532 164 0.0685451 0.369779
2020-06-08 12:36:44.826489 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.369803
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 6837 6821 170.415 152 1.03978 0.369803
161 16 6877 6861 170.35 160 0.299515 0.370076
162 15 6921 6906 170.41 180 0.479073 0.370047
163 16 6955 6939 170.174 132 0.970325 0.370485
164 16 6991 6975 170.015 144 1.25103 0.37083
165 15 7038 7023 170.147 192 0.412588 0.37058
166 16 7084 7068 170.206 180 0.166015 0.370487
167 16 7136 7120 170.432 208 0.0960646 0.370014
168 16 7176 7160 170.37 160 1.20631 0.370032
169 15 7217 7202 170.355 168 0.0725855 0.370052
170 16 7261 7245 170.365 172 0.0880765 0.369988
171 16 7301 7285 170.304 160 0.0939977 0.370118
172 16 7338 7322 170.175 148 0.0937779 0.370366
173 16 7373 7357 170 140 0.285505 0.370597
174 16 7414 7398 169.964 164 0.509076 0.370912
175 16 7462 7446 170.09 192 0.123406 0.370835
176 16 7504 7488 170.077 168 0.0673462 0.370791
177 15 7537 7522 169.885 136 0.359524 0.370841
178 16 7582 7566 169.916 176 0.0573193 0.371038
179 15 7617 7602 169.77 144 0.161006 0.371041
2020-06-08 12:37:04.836065 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.371552
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 7653 7637 169.605 140 1.48986 0.371552
181 16 7689 7673 169.463 144 0.165271 0.371909
182 16 7736 7720 169.565 188 0.087052 0.371983
183 16 7779 7763 169.578 172 0.755047 0.372088
184 16 7833 7817 169.829 216 0.175646 0.371405
185 16 7881 7865 169.949 192 0.0892331 0.37119
186 16 7921 7905 169.896 160 0.299585 0.371403
187 16 7967 7951 169.971 184 0.0684797 0.371202
188 16 8009 7993 169.96 168 0.292215 0.371086
189 16 8053 8037 169.992 176 0.614826 0.371233
190 15 8086 8071 169.813 136 0.0720735 0.37129
191 16 8126 8110 169.741 156 0.320248 0.371545
192 16 8166 8150 169.69 160 0.549814 0.371717
193 15 8211 8196 169.764 184 0.425214 0.371778
194 16 8254 8238 169.755 168 0.0870834 0.371445
195 15 8300 8285 169.848 188 0.337432 0.371548
196 16 8346 8330 169.9 180 0.19846 0.370976
197 16 8384 8368 169.809 152 1.16759 0.371254
198 16 8425 8409 169.77 164 0.328002 0.371196
199 16 8471 8455 169.842 184 0.522 0.371561
2020-06-08 12:37:24.851318 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.371114
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 15 8515 8500 169.891 180 0.369488 0.371114
201 16 8556 8540 169.841 160 0.229863 0.371443
202 16 8599 8583 169.852 172 0.378308 0.371379
203 16 8648 8632 169.98 196 0.321516 0.371269
204 16 8692 8676 170.01 176 0.106048 0.371112
205 16 8731 8715 169.941 156 0.821938 0.371155
206 16 8775 8759 169.97 176 0.128704 0.370892
207 16 8820 8804 170.018 180 1.33029 0.371066
208 16 8861 8845 169.986 164 0.706841 0.371111
209 16 8909 8893 170.092 192 0.500026 0.370966
210 15 8947 8932 170.024 156 0.460491 0.370982
211 16 8992 8976 170.051 176 0.0817317 0.370814
212 16 9034 9018 170.041 168 0.121517 0.370766
213 16 9072 9056 169.956 152 0.0494781 0.370849
214 16 9116 9100 169.984 176 0.337138 0.370895
215 16 9155 9139 169.919 156 0.250411 0.371265
216 16 9198 9182 169.928 172 0.0511419 0.370894
217 16 9245 9229 170.011 188 0.0505669 0.371062
218 16 9289 9273 170.038 176 0.241485 0.371116
219 16 9332 9316 170.047 172 0.420921 0.371013
2020-06-08 12:37:44.869174 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.370747
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 15 9380 9365 170.16 196 0.123874 0.370747
221 16 9430 9414 170.269 196 0.727918 0.370537
222 16 9478 9462 170.353 192 0.151264 0.370298
223 16 9517 9501 170.288 156 1.33769 0.37031
224 16 9561 9545 170.314 176 0.0647897 0.37037
225 16 9613 9597 170.481 208 0.0624257 0.36997
226 16 9658 9642 170.522 180 0.517904 0.370019
227 16 9697 9681 170.458 156 0.0834134 0.370062
228 16 9738 9722 170.43 164 0.33671 0.370219
229 15 9785 9770 170.524 192 0.589027 0.37014
230 15 9828 9813 170.53 172 0.0972365 0.370032
231 15 9876 9861 170.623 192 0.161094 0.369796
232 16 9918 9902 170.594 164 0.934149 0.369721
233 15 9957 9942 170.546 160 0.537636 0.369896
234 16 10003 9987 170.587 180 0.842218 0.369962
235 16 10042 10026 170.524 156 0.136153 0.36994
236 16 10085 10069 170.531 172 0.110852 0.36995
237 15 10127 10112 170.537 172 0.585616 0.36997
238 16 10168 10152 170.492 160 0.165433 0.370122
239 16 10215 10199 170.565 188 0.491289 0.369969
2020-06-08 12:38:04.905117 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.369971
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 15 10252 10237 170.488 152 0.450304 0.369971
241 15 10296 10281 170.509 176 0.074617 0.370017
242 16 10340 10324 170.514 172 0.0748047 0.369973
243 15 10384 10369 170.553 180 0.190594 0.36987
244 16 10435 10419 170.67 200 0.687906 0.369674
245 15 10486 10471 170.821 208 0.346122 0.369289
246 16 10518 10502 170.63 124 0.0958972 0.369579
247 16 10561 10545 170.635 172 0.477452 0.369753
248 16 10600 10584 170.575 156 0.149365 0.369825
249 15 10647 10632 170.66 192 0.587592 0.369778
250 15 10691 10676 170.681 176 0.0887679 0.369776
251 16 10738 10722 170.734 184 0.43434 0.369488
252 15 10775 10760 170.659 152 0.289411 0.369328
253 16 10821 10805 170.695 180 1.10484 0.369712
254 15 10864 10849 170.71 176 0.614384 0.36973
255 16 10906 10890 170.684 164 0.360487 0.369725
256 16 10946 10930 170.642 160 0.691751 0.369749
257 16 10990 10974 170.663 176 0.102485 0.369666
258 16 11033 11017 170.666 172 0.078748 0.369672
259 16 11077 11061 170.687 176 0.205638 0.369744
2020-06-08 12:38:24.935612 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.369638
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 11118 11102 170.661 164 0.280352 0.369638
261 15 11157 11142 170.62 160 1.23471 0.369661
262 16 11199 11183 170.594 164 0.0719021 0.369878
263 16 11238 11222 170.538 156 0.0719583 0.369864
264 16 11285 11269 170.604 188 0.372989 0.369713
265 16 11329 11313 170.624 176 0.205712 0.369717
266 16 11377 11361 170.704 192 0.217287 0.369723
267 16 11425 11409 170.783 192 0.104386 0.369514
268 16 11468 11452 170.785 172 0.0754446 0.369326
269 16 11509 11493 170.76 164 0.201825 0.369348
270 16 11547 11531 170.69 152 1.57529 0.36947
271 16 11597 11581 170.798 200 0.405929 0.369466
272 15 11645 11630 170.89 196 0.0776648 0.369341
273 16 11691 11675 170.923 180 0.0870909 0.369237
274 15 11733 11718 170.926 172 0.103877 0.369162
275 16 11778 11762 170.944 176 0.340553 0.369209
276 16 11825 11809 171.005 188 0.498878 0.369047
277 15 11873 11858 171.095 196 0.389793 0.368885
278 16 11910 11894 170.998 144 0.184223 0.369059
279 16 11952 11936 170.987 168 0.880617 0.368982
2020-06-08 12:38:44.950293 min lat: 0.00803674 max lat: 2.04466 avg lat: 0.368991
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 11997 11981 171.019 180 0.124486 0.368991
281 16 12048 12032 171.136 204 0.241506 0.368816
282 15 12087 12072 171.096 160 0.268891 0.36891
283 16 12129 12113 171.07 164 1.00581 0.368959
284 16 12173 12157 171.087 176 0.531792 0.36892
285 16 12212 12196 171.034 156 1.16607 0.368898
286 16 12254 12238 171.024 168 0.0710417 0.368941
287 16 12295 12279 170.999 164 0.0871388 0.368994
288 16 12338 12322 171.003 172 0.591573 0.369082
289 16 12375 12359 170.923 148 0.495751 0.36923
290 15 12417 12402 170.926 172 0.0411501 0.36914
291 16 12460 12444 170.916 168 0.101774 0.369164
292 16 12498 12482 170.851 152 0.199923 0.369323
293 16 12546 12530 170.923 192 0.443601 0.369198
294 16 12585 12569 170.872 156 0.32418 0.369049
295 16 12625 12609 170.835 160 0.205517 0.369366
296 16 12671 12655 170.88 184 0.147168 0.369341
297 16 12722 12706 170.991 204 0.120022 0.368844
298 16 12768 12752 171.034 184 0.97924 0.368894
299 15 12812 12797 171.064 180 0.116044 0.368897
2020-06-08 12:39:04.956275 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.36888
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 12856 12840 171.067 172 0.519146 0.36888
301 16 12901 12885 171.097 180 0.402145 0.368828
302 16 12943 12927 171.086 168 0.0975264 0.368752
303 16 12990 12974 171.142 188 0.0440586 0.368661
304 16 13036 13020 171.184 184 0.186594 0.36858
305 15 13081 13066 171.226 184 0.816621 0.36862
306 16 13129 13113 171.281 188 0.0939368 0.368523
307 16 13167 13151 171.218 152 0.071984 0.368553
308 16 13205 13189 171.156 152 0.0115672 0.368622
309 16 13242 13226 171.081 148 0.0681405 0.36879
310 16 13285 13269 171.084 172 0.296127 0.368752
311 15 13328 13313 171.099 176 1.01723 0.368844
312 16 13367 13351 171.038 152 0.12673 0.368946
313 15 13406 13391 171.003 160 0.544955 0.36898
314 15 13449 13434 171.006 172 0.120477 0.368944
315 16 13497 13481 171.06 188 0.119845 0.368942
316 16 13541 13525 171.075 176 0.0955662 0.368919
317 16 13586 13570 171.103 180 0.828479 0.368862
318 16 13634 13618 171.169 192 0.135091 0.368797
319 16 13674 13658 171.134 160 0.261829 0.368743
2020-06-08 12:39:24.959038 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368694
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 13724 13708 171.224 200 0.443875 0.368694
321 15 13763 13748 171.189 160 0.318633 0.368698
322 16 13807 13791 171.191 172 0.0734651 0.368665
323 16 13854 13838 171.243 188 0.329117 0.368621
324 16 13900 13884 171.282 184 0.346575 0.368538
325 15 13941 13926 171.272 168 0.10977 0.368579
326 16 13984 13968 171.262 168 0.461086 0.368565
327 16 14033 14017 171.337 196 0.617285 0.368431
328 16 14081 14065 171.4 192 0.234995 0.368327
329 15 14122 14107 171.371 168 0.261684 0.368319
330 16 14167 14151 171.385 176 0.264592 0.368219
331 16 14208 14192 171.362 164 0.326276 0.368272
332 16 14254 14238 171.399 184 0.106693 0.368265
333 16 14298 14282 171.413 176 0.526763 0.368131
334 16 14337 14321 171.366 156 0.121332 0.368159
335 16 14375 14359 171.307 152 0.652309 0.368182
336 16 14414 14398 171.261 156 0.457002 0.368495
337 16 14458 14442 171.274 176 0.0491926 0.368411
338 16 14503 14487 171.299 180 0.0676485 0.368358
339 16 14539 14523 171.219 144 0.0824609 0.368449
2020-06-08 12:39:45.008426 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368397
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 14579 14563 171.186 160 0.0587123 0.368397
341 16 14620 14604 171.165 164 0.164267 0.368729
342 16 14664 14648 171.178 176 1.46672 0.368628
343 15 14701 14686 171.122 152 0.19675 0.368726
344 16 14744 14728 171.113 168 0.579119 0.368835
345 16 14786 14770 171.103 168 0.0777964 0.368897
346 16 14835 14819 171.175 196 0.145912 0.368732
347 16 14882 14866 171.222 188 0.108431 0.368658
348 16 14924 14908 171.213 168 0.344849 0.368635
349 16 14968 14952 171.226 176 0.170138 0.368617
350 16 15013 14997 171.25 180 0.577516 0.3686
351 16 15047 15031 171.148 136 0.279342 0.368771
352 15 15088 15073 171.139 168 0.216839 0.368739
353 16 15129 15113 171.107 160 0.26602 0.368819
354 16 15169 15153 171.076 160 0.0670473 0.368949
355 16 15206 15190 171.011 148 0.10684 0.369033
356 16 15250 15234 171.025 176 0.0900062 0.369047
357 16 15293 15277 171.026 172 0.496779 0.369027
358 16 15338 15322 171.051 180 0.0877171 0.36904
359 16 15375 15359 170.987 148 0.081768 0.369064
2020-06-08 12:40:05.025289 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.369087
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 15419 15403 171.001 176 1.18147 0.369087
361 16 15454 15438 170.915 140 0.304616 0.369171
362 16 15494 15478 170.885 160 0.085328 0.369151
363 16 15543 15527 170.953 196 0.156983 0.369239
364 15 15582 15567 170.923 160 0.647775 0.369236
365 16 15627 15611 170.937 176 0.085923 0.369237
366 16 15674 15658 170.983 188 0.112202 0.369109
367 16 15727 15711 171.095 212 0.12493 0.368917
368 16 15773 15757 171.13 184 0.200318 0.368847
369 15 15818 15803 171.165 184 0.194294 0.368766
370 16 15862 15846 171.167 172 0.18507 0.368782
371 16 15910 15894 171.223 192 0.510724 0.368781
372 16 15951 15935 171.204 164 0.321037 0.368631
373 16 15991 15975 171.174 160 0.04023 0.3685
374 16 16031 16015 171.144 160 0.0743827 0.368522
375 16 16071 16055 171.114 160 0.100989 0.368641
376 16 16111 16095 171.084 160 0.0698934 0.368633
377 16 16148 16132 171.023 148 1.56425 0.368877
378 16 16193 16177 171.047 180 0.076014 0.369106
379 15 16229 16214 170.986 148 0.104242 0.369005
2020-06-08 12:40:25.030381 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.369126
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 15 16273 16258 170.999 176 0.266939 0.369126
381 16 16316 16300 170.991 168 0.651709 0.369243
382 15 16366 16351 171.077 204 0.244598 0.369095
383 15 16413 16398 171.121 188 0.661911 0.368918
384 16 16453 16437 171.082 156 0.1239 0.368912
385 15 16493 16478 171.062 164 0.205343 0.369012
386 16 16537 16521 171.065 172 0.92265 0.36905
387 16 16578 16562 171.046 164 0.0943781 0.369027
388 15 16629 16614 171.141 208 0.288907 0.368991
389 16 16671 16655 171.123 164 0.283482 0.368878
390 16 16714 16698 171.125 172 0.846183 0.369016
391 16 16751 16735 171.066 148 0.0405144 0.368932
392 15 16794 16779 171.078 176 0.0665434 0.368977
393 16 16837 16821 171.069 168 0.109843 0.368986
394 16 16882 16866 171.091 180 0.274312 0.369024
395 16 16929 16913 171.133 188 0.813605 0.368956
396 16 16974 16958 171.155 180 0.196726 0.368841
397 15 17017 17002 171.167 176 0.828185 0.368902
398 16 17066 17050 171.219 192 0.621207 0.368727
399 16 17109 17093 171.221 172 0.264891 0.368679
2020-06-08 12:40:45.044451 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368777
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 15 17147 17132 171.183 156 0.0894283 0.368777
401 16 17193 17177 171.205 180 0.632196 0.36874
402 16 17240 17224 171.246 188 0.48601 0.368732
403 16 17284 17268 171.258 176 0.303091 0.368619
404 16 17327 17311 171.259 172 1.13456 0.368504
405 16 17371 17355 171.271 176 0.196824 0.36845
406 15 17415 17400 171.292 180 0.484843 0.368536
407 16 17461 17445 171.314 180 0.0772191 0.368493
408 16 17505 17489 171.325 176 1.28317 0.368537
409 16 17548 17532 171.326 172 0.45095 0.368494
410 16 17588 17572 171.298 160 0.929039 0.368493
411 16 17632 17616 171.31 176 1.12283 0.368465
412 16 17671 17655 171.273 156 0.0630831 0.368423
413 16 17717 17701 171.303 184 0.311614 0.368483
414 16 17760 17744 171.305 172 1.12869 0.368439
415 15 17809 17794 171.374 200 0.202476 0.368366
416 15 17854 17839 171.395 180 1.43189 0.368289
417 16 17895 17879 171.367 160 0.071285 0.368411
418 16 17942 17926 171.407 188 0.334598 0.368354
419 16 17991 17975 171.465 196 0.238768 0.368236
2020-06-08 12:41:05.050839 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368204
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 18034 18018 171.466 172 0.986549 0.368204
421 15 18078 18063 171.487 180 1.46089 0.368135
422 16 18127 18111 171.535 192 0.386874 0.367998
423 15 18165 18150 171.498 156 0.300668 0.368097
424 16 18211 18195 171.518 180 0.351673 0.368059
425 16 18254 18238 171.519 172 0.738602 0.368047
426 16 18298 18282 171.529 176 0.100675 0.368054
427 16 18341 18325 171.529 172 0.291243 0.368056
428 16 18381 18365 171.502 160 0.705053 0.368095
429 16 18426 18410 171.522 180 0.238712 0.368041
430 15 18468 18453 171.523 172 0.479224 0.368042
431 16 18513 18497 171.533 176 0.121838 0.367902
432 15 18555 18540 171.534 172 0.143614 0.367975
433 16 18602 18586 171.562 184 0.130385 0.368027
434 16 18643 18627 171.545 164 0.340189 0.368091
435 16 18687 18671 171.554 176 0.651568 0.368042
436 15 18728 18713 171.546 168 0.24549 0.368076
437 16 18778 18762 171.602 196 0.132161 0.367919
438 16 18819 18803 171.584 164 0.0591813 0.367831
439 16 18860 18844 171.566 164 0.297181 0.367922
2020-06-08 12:41:25.064246 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368035
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 18902 18886 171.558 168 0.743189 0.368035
441 16 18948 18932 171.586 184 0.219826 0.368001
442 16 18995 18979 171.623 188 0.47599 0.367938
443 16 19040 19024 171.641 180 0.535307 0.367893
444 16 19082 19066 171.633 168 0.685933 0.367873
445 16 19130 19114 171.677 192 0.0607425 0.367814
446 16 19178 19162 171.712 192 0.256125 0.367747
447 16 19220 19204 171.704 168 0.723128 0.367719
448 15 19253 19238 171.624 136 0.0871409 0.367759
449 16 19299 19283 171.643 180 0.0977956 0.367739
450 15 19345 19330 171.679 188 0.350577 0.367832
451 16 19393 19377 171.715 188 0.443752 0.367754
452 16 19440 19424 171.751 188 0.055133 0.367597
453 15 19484 19469 171.769 180 0.332569 0.367572
454 16 19526 19510 171.752 164 0.336732 0.367549
455 16 19567 19551 171.734 164 0.0777378 0.367588
456 15 19610 19595 171.744 176 0.684707 0.367607
457 16 19653 19637 171.735 168 0.595498 0.367618
458 16 19698 19682 171.752 180 0.546961 0.367603
459 16 19742 19726 171.76 176 0.154392 0.367519
2020-06-08 12:41:45.107477 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.367502
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 19788 19772 171.787 184 0.0965505 0.367502
461 15 19830 19815 171.787 172 0.69698 0.367524
462 16 19878 19862 171.822 188 0.293698 0.367446
463 16 19926 19910 171.866 192 0.721074 0.367413
464 16 19971 19955 171.883 180 0.0823879 0.367297
465 16 20012 19996 171.866 164 0.193197 0.36731
466 16 20051 20035 171.831 156 0.427847 0.367267
467 16 20092 20076 171.814 164 0.334002 0.367453
468 16 20133 20117 171.797 164 0.0732855 0.367419
469 16 20177 20161 171.806 176 0.250304 0.367468
470 16 20214 20198 171.755 148 0.0578775 0.367529
471 16 20252 20236 171.713 152 0.0606962 0.367566
472 15 20296 20281 171.731 180 0.0706551 0.367523
473 16 20340 20324 171.731 172 0.0995366 0.367534
474 16 20377 20361 171.681 148 0.105159 0.367619
475 16 20420 20404 171.682 172 0.193995 0.367719
476 16 20462 20446 171.674 168 0.745976 0.367802
477 16 20500 20484 171.633 152 0.100294 0.367803
478 16 20541 20525 171.616 164 1.04113 0.367899
479 16 20583 20567 171.609 168 0.300737 0.367753
2020-06-08 12:42:05.116361 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.367749
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 20626 20610 171.61 172 0.0737056 0.367749
481 16 20671 20655 171.627 180 0.161571 0.367761
482 16 20712 20696 171.611 164 0.250212 0.367869
483 16 20754 20738 171.604 168 0.0801082 0.367947
484 16 20797 20781 171.604 172 0.080388 0.367923
485 16 20841 20825 171.613 176 0.0886142 0.367931
486 16 20881 20865 171.589 160 0.354113 0.367922
487 16 20923 20907 171.582 168 0.259697 0.367951
488 16 20968 20952 171.599 180 0.312084 0.367889
489 16 21013 20997 171.616 180 0.0996561 0.367802
490 16 21053 21037 171.592 160 0.0765918 0.367855
491 16 21098 21082 171.61 180 0.461261 0.367967
492 16 21142 21126 171.618 176 0.298625 0.367939
493 16 21182 21166 171.595 160 0.0972208 0.367882
494 16 21223 21207 171.579 164 0.194985 0.36794
495 16 21262 21246 171.548 156 0.100899 0.368001
496 16 21300 21284 171.508 152 0.684771 0.368061
497 16 21342 21326 171.501 168 0.0776132 0.368097
498 16 21383 21367 171.486 164 0.971859 0.368162
499 16 21428 21412 171.503 180 0.0725294 0.368133
2020-06-08 12:42:25.120038 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368206
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 15 21463 21448 171.448 144 0.0502814 0.368206
501 16 21507 21491 171.449 172 0.283424 0.368146
502 15 21543 21528 171.402 148 0.507856 0.368198
503 15 21587 21572 171.411 176 0.950412 0.368274
504 16 21627 21611 171.381 156 1.53861 0.36832
505 16 21670 21654 171.382 172 0.138185 0.368333
506 16 21711 21695 171.367 164 0.0794395 0.368323
507 16 21751 21735 171.344 160 0.197224 0.368437
508 15 21790 21775 171.322 160 0.285314 0.368379
509 16 21833 21817 171.315 168 0.272573 0.368378
510 16 21875 21859 171.309 168 1.69653 0.368503
511 16 21920 21904 171.326 180 0.11077 0.368537
512 16 21972 21956 171.397 208 0.278276 0.36838
513 16 22016 22000 171.406 176 0.250833 0.368378
514 15 22059 22044 171.414 176 0.304413 0.368413
515 16 22102 22086 171.408 168 0.0509173 0.368366
516 16 22138 22122 171.355 144 0.0969523 0.36837
517 16 22180 22164 171.348 168 1.59564 0.368439
518 16 22223 22207 171.349 172 0.71206 0.36854
519 15 22266 22251 171.357 176 0.607223 0.3685
2020-06-08 12:42:45.129420 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368498
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 22308 22292 171.343 164 0.801967 0.368498
521 15 22352 22337 171.357 180 0.313468 0.368539
522 16 22401 22385 171.396 192 0.155938 0.368349
523 16 22435 22419 171.328 136 1.59721 0.368336
524 16 22464 22448 171.223 116 1.73026 0.368489
525 16 22501 22485 171.178 148 1.8747 0.368573
526 16 22543 22527 171.172 168 0.18821 0.368636
527 16 22586 22570 171.173 172 0.0725106 0.368654
528 16 22625 22609 171.145 156 0.444407 0.368929
529 16 22672 22656 171.176 188 0.0658022 0.368885
530 16 22713 22697 171.162 164 0.0653625 0.368855
531 16 22751 22735 171.126 152 1.05794 0.368961
532 15 22797 22782 171.158 188 1.01297 0.368923
533 16 22843 22827 171.175 180 0.331542 0.368947
534 16 22890 22874 171.206 188 0.112522 0.368779
535 16 22934 22918 171.215 176 0.295483 0.368704
536 15 22975 22960 171.209 168 0.330568 0.368747
537 16 23013 22997 171.165 148 0.399563 0.368815
538 16 23050 23034 171.122 148 0.0617131 0.369026
539 16 23095 23079 171.139 180 0.224041 0.368934
2020-06-08 12:43:05.147687 min lat: 0.00736974 max lat: 2.12878 avg lat: 0.368984
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 15 23133 23118 171.11 156 0.323369 0.368984
541 16 23177 23161 171.112 172 0.256919 0.369023
542 16 23222 23206 171.128 180 0.0765498 0.369039
543 16 23259 23243 171.085 148 0.754259 0.369051
544 15 23300 23285 171.08 168 0.323543 0.369055
545 16 23347 23331 171.103 184 0.432705 0.369011
546 15 23388 23373 171.098 168 1.00259 0.368974
547 16 23428 23412 171.07 156 1.37922 0.369058
548 16 23470 23454 171.064 168 0.91654 0.369091
549 15 23506 23491 171.022 148 0.825622 0.369162
550 16 23544 23528 170.98 148 0.0766564 0.369265
551 15 23588 23573 170.996 180 1.33703 0.369289
552 16 23632 23616 170.997 172 0.301987 0.369187
553 16 23674 23658 170.992 168 0.108623 0.369226
554 16 23713 23697 170.963 156 0.234901 0.369327
555 16 23762 23746 171.009 196 0.238057 0.369289
556 16 23793 23777 170.924 124 0.0873343 0.36924
557 16 23829 23813 170.876 144 1.57494 0.369429
558 16 23866 23850 170.835 148 0.182317 0.36952
559 16 23914 23898 170.872 192 0.187263 0.369548
2020-06-08 12:43:25.160659 min lat: 0.00736974 max lat: 2.12878 avg lat: 0.369533
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 23952 23936 170.838 152 0.364742 0.369533
561 16 23996 23980 170.847 176 0.214645 0.36954
562 15 24035 24020 170.828 160 0.340947 0.369606
563 16 24076 24060 170.808 160 0.654762 0.369707
564 16 24120 24104 170.817 176 0.367582 0.369676
565 16 24163 24147 170.819 172 0.387184 0.369667
566 16 24204 24188 170.807 164 0.279247 0.369648
567 16 24245 24229 170.795 164 1.05249 0.36973
568 16 24288 24272 170.797 172 0.198967 0.369741
569 16 24332 24316 170.806 176 0.136252 0.369715
570 15 24366 24351 170.752 140 0.0584609 0.369714
571 16 24409 24393 170.747 168 0.740716 0.369819
572 16 24452 24436 170.749 172 0.101809 0.369823
573 16 24496 24480 170.758 176 0.0578064 0.369746
574 16 24537 24521 170.746 164 0.998901 0.369775
575 16 24576 24560 170.72 156 0.207606 0.36985
576 16 24618 24602 170.715 168 0.212732 0.369869
577 16 24665 24649 170.745 188 0.949637 0.369771
578 15 24712 24697 170.773 192 0.923029 0.369766
579 16 24753 24737 170.755 160 0.651137 0.369768
2020-06-08 12:43:45.198519 min lat: 0.00736974 max lat: 2.12878 avg lat: 0.369777
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 24796 24780 170.757 172 0.0584573 0.369777
581 15 24845 24830 170.806 200 0.364453 0.369687
582 15 24891 24876 170.827 184 0.205766 0.369647
583 16 24932 24916 170.809 160 0.0839604 0.36966
584 16 24974 24958 170.804 168 0.0906737 0.369679
585 15 25009 24994 170.758 144 0.0812008 0.369732
586 15 25054 25039 170.774 180 0.0987196 0.369754
587 15 25091 25076 170.735 148 0.0988818 0.369777
588 15 25131 25116 170.716 160 0.14241 0.36976
589 16 25178 25162 170.738 184 0.0643666 0.369812
590 16 25221 25205 170.741 172 0.18708 0.369836
591 16 25269 25253 170.776 192 0.743014 0.369722
592 16 25314 25298 170.791 180 0.263785 0.369716
593 16 25361 25345 170.82 188 0.376313 0.369688
594 16 25401 25385 170.802 160 0.125557 0.369671
595 16 25446 25430 170.817 180 0.087672 0.369677
596 15 25495 25480 170.866 200 0.060978 0.369554
597 16 25544 25528 170.902 192 0.22937 0.369446
598 15 25578 25563 170.85 140 1.47206 0.369474
599 16 25622 25606 170.852 172 0.964434 0.369568
2020-06-08 12:44:05.214016 min lat: 0.00736974 max lat: 2.12878 avg lat: 0.369611
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 15 25640 25625 170.694 76 0.988185 0.369611
Total time run: 600.544
Total reads made: 25640
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 170.779
Average IOPS: 42
Stddev IOPS: 3.97991
Max IOPS: 54
Min IOPS: 19
Average Latency(s): 0.369894
Max latency(s): 2.12878
Min latency(s): 0.00736974
Total time run: 600.544
Total reads made: 25640
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 170.779
Average IOPS: 42
Stddev IOPS: 3.97991
Max IOPS: 54
Min IOPS: 19
Average Latency(s): 0.369894
Max latency(s): 2.12878
Min latency(s): 0.00736974
Cleanup, pool with 16 PGs
podman exec ceph-mon-f5-422-01 rados -p testbench cleanup
Performance Baseline with rados bench, pool with 256 PGs
Since in an earlier blog post I tested with 8 and 256, I wanted to have at least one test run with the same number of PGs. Long term though I will align myself to the numbers suggested by Ceph Nautilus’ PG autoscaler.
I went on to create a single, replica 3, pool testbench2 with 256 PGs.
I set the application type to rgw.
All rados bench tests were run on f5-422-01.
Sequential Write Test, pool with 256 PGs
podman exec ceph-mon-f5-422-01 rados bench -p testbench2 600 write --no-cleanup | tee --append $(date +%F)-256PGs-rados-test-write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_f5-422-01.internal.pcfe.net_1349
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 36 20 79.8409 80 0.321672 0.474287
2 16 53 37 73.919 68 0.623514 0.598543
3 16 72 56 74.6075 76 2.21294 0.721355
4 16 91 75 74.9514 76 0.85585 0.710295
5 16 106 90 71.9595 60 0.656597 0.728313
6 16 125 109 72.6089 76 0.290908 0.763383
7 16 141 125 71.3779 64 0.901804 0.800385
8 16 158 142 70.9531 68 0.518742 0.831229
9 16 179 163 72.4008 84 0.704042 0.832311
10 16 194 178 71.156 60 1.54382 0.839197
11 16 210 194 70.5041 64 1.97506 0.854411
12 16 232 216 71.9601 88 0.983213 0.84574
13 16 252 236 72.5772 80 0.6913 0.842717
14 16 271 255 72.8205 76 0.791158 0.840037
15 16 291 275 73.298 80 0.812229 0.837232
16 16 310 294 73.4662 76 0.921979 0.839894
17 16 326 310 72.9056 64 0.529278 0.839648
18 16 348 332 73.7432 88 0.598223 0.834106
19 16 363 347 73.0193 60 0.667751 0.833339
2020-06-08 20:11:17.984346 min lat: 0.0802415 max lat: 3.10168 avg lat: 0.839968
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 378 362 72.368 60 0.450386 0.839968
21 16 396 380 72.3482 72 0.139175 0.86046
22 16 408 392 71.2415 48 0.418451 0.86067
23 16 423 407 70.7523 60 2.20081 0.874599
24 16 445 429 71.4697 88 1.2708 0.866653
25 16 471 455 72.7696 104 0.285421 0.860298
26 16 486 470 72.2781 60 0.389528 0.861504
27 16 499 483 71.5249 52 0.410962 0.866005
28 16 515 499 71.2558 64 0.429008 0.877484
29 16 529 513 70.7292 56 0.285012 0.879612
30 16 541 525 69.9714 48 0.457178 0.883154
31 16 552 536 69.1307 44 0.545492 0.890488
32 16 560 544 67.9705 32 2.8067 0.896042
33 16 572 556 67.3653 48 3.29071 0.905777
34 16 586 570 67.0308 56 0.294248 0.914702
35 16 603 587 67.0582 68 1.75803 0.938793
36 16 621 605 67.1951 72 0.532758 0.936639
37 16 645 629 67.973 96 0.604943 0.925118
38 16 667 651 68.4918 88 0.558411 0.924037
39 16 678 662 67.8639 44 2.005 0.924458
2020-06-08 20:11:37.994972 min lat: 0.0802415 max lat: 4.29548 avg lat: 0.9253
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 697 681 68.0669 76 1.08916 0.9253
41 16 719 703 68.5525 88 0.570717 0.922517
42 16 737 721 68.6338 72 1.31116 0.922207
43 16 757 741 68.8978 80 0.526462 0.916848
44 16 774 758 68.8771 68 1.93632 0.916822
45 16 797 781 69.39 92 0.134226 0.91297
46 16 810 794 69.0119 52 1.02052 0.912372
47 16 831 815 69.3304 84 0.15731 0.909389
48 16 848 832 69.3021 68 0.360588 0.910789
49 16 863 847 69.1121 60 0.659684 0.913781
50 16 887 871 69.6491 96 0.133689 0.912219
51 16 905 889 69.6949 72 0.855513 0.909605
52 16 921 905 69.585 64 1.17666 0.908347
53 16 938 922 69.5548 68 1.38649 0.91053
54 16 951 935 69.2281 52 0.55436 0.911495
55 16 971 955 69.4206 80 0.219227 0.908762
56 16 987 971 69.3236 64 0.301743 0.907939
57 16 1005 989 69.3701 72 0.501604 0.911605
58 16 1026 1010 69.6175 84 0.986777 0.912436
59 16 1047 1031 69.8609 84 0.559188 0.908551
2020-06-08 20:11:58.007277 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.905304
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 1063 1047 69.7631 64 0.758972 0.905304
61 16 1079 1063 69.6684 64 1.48431 0.907042
62 16 1093 1077 69.4478 56 1.04793 0.909227
63 16 1109 1093 69.3592 64 0.517822 0.908047
64 16 1126 1110 69.3379 68 0.138691 0.908729
65 16 1145 1129 69.439 76 0.81336 0.914216
66 15 1163 1148 69.538 76 0.927726 0.911847
67 16 1186 1170 69.8131 88 1.01725 0.909855
68 16 1205 1189 69.9038 76 0.662465 0.907191
69 16 1219 1203 69.7022 56 0.609877 0.908125
70 16 1240 1224 69.9062 84 0.708919 0.909375
71 16 1261 1245 70.1043 84 1.41813 0.909731
72 16 1280 1264 70.1859 76 0.693473 0.905951
73 16 1298 1282 70.2106 72 0.135294 0.903533
74 16 1316 1300 70.2346 72 0.96985 0.90298
75 16 1337 1321 70.4179 84 1.08314 0.902772
76 16 1360 1344 70.7016 92 0.328514 0.899776
77 16 1379 1363 70.7701 76 0.242521 0.897653
78 16 1401 1385 70.9906 88 0.458992 0.895658
79 16 1418 1402 70.9526 68 0.263757 0.892196
2020-06-08 20:12:18.014418 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.892981
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 1435 1419 70.9155 68 1.08344 0.892981
81 16 1450 1434 70.7806 60 0.30134 0.893305
82 16 1469 1453 70.8442 76 0.848352 0.893995
83 16 1491 1475 71.0499 88 0.38473 0.894693
84 16 1504 1488 70.8231 52 0.0967975 0.893998
85 16 1521 1505 70.7897 68 0.873757 0.895943
86 16 1536 1520 70.6642 60 0.529404 0.897625
87 16 1555 1539 70.7254 76 0.529974 0.899846
88 16 1573 1557 70.7395 72 0.366885 0.899557
89 16 1589 1573 70.6637 64 0.154215 0.898277
90 16 1611 1595 70.8561 88 0.150982 0.895677
91 16 1632 1616 71.0002 84 0.54568 0.894606
92 16 1647 1631 70.8806 60 0.504008 0.89485
93 16 1667 1651 70.9784 80 1.06263 0.894491
94 16 1680 1664 70.7765 52 0.454223 0.894676
95 16 1706 1690 71.126 104 0.19763 0.894489
96 16 1723 1707 71.0933 68 0.403824 0.893514
97 16 1744 1728 71.2261 84 0.573909 0.893502
98 16 1759 1743 71.1106 60 0.968286 0.89342
99 16 1775 1759 71.0387 64 2.57467 0.89396
2020-06-08 20:12:38.020321 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.8947
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 1790 1774 70.9282 60 0.271104 0.8947
101 16 1803 1787 70.7407 52 0.526812 0.896171
102 16 1821 1805 70.7529 72 0.45184 0.89595
103 16 1830 1814 70.4154 36 0.507727 0.896412
104 16 1846 1830 70.3536 64 0.13647 0.899743
105 16 1861 1845 70.2549 60 0.520945 0.899306
106 16 1877 1861 70.1958 64 3.94324 0.901647
107 16 1893 1877 70.1378 64 0.835055 0.904566
108 16 1914 1898 70.266 84 0.620073 0.904139
109 16 1930 1914 70.2085 64 0.653299 0.9056
110 16 1944 1928 70.0791 56 1.85829 0.907273
111 16 1963 1947 70.1324 76 1.55756 0.907764
112 16 1985 1969 70.2917 88 0.645438 0.906213
113 16 2006 1990 70.4129 84 0.524514 0.904945
114 16 2026 2010 70.4968 80 0.813488 0.904036
115 16 2053 2037 70.8227 108 0.959682 0.900399
116 16 2069 2053 70.7637 64 0.459798 0.898856
117 16 2099 2083 71.1842 120 0.371983 0.897921
118 16 2121 2105 71.3266 88 0.594193 0.894294
119 16 2139 2123 71.332 72 1.12007 0.893198
2020-06-08 20:12:58.024585 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.892195
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 2158 2142 71.3708 76 1.58431 0.892195
121 16 2177 2161 71.4088 76 0.477455 0.892128
122 16 2196 2180 71.4462 76 0.571804 0.892119
123 16 2210 2194 71.3199 56 0.689722 0.891802
124 16 2226 2210 71.2585 64 0.603326 0.893514
125 16 2240 2224 71.1364 56 1.53439 0.894229
126 16 2260 2244 71.2066 80 0.426871 0.89444
127 16 2281 2265 71.3072 84 0.357074 0.892664
128 16 2303 2287 71.4374 88 0.597109 0.891852
129 16 2322 2306 71.4727 76 0.460509 0.889652
130 16 2345 2329 71.6303 92 0.615189 0.890176
131 16 2366 2350 71.7227 84 2.0324 0.888586
132 16 2388 2372 71.8458 88 1.24972 0.88716
133 16 2408 2392 71.907 80 0.488024 0.88656
134 16 2428 2412 71.9672 80 0.930692 0.885833
135 16 2447 2431 71.997 76 0.323277 0.884211
136 16 2473 2457 72.2321 104 0.394564 0.88333
137 16 2499 2483 72.4638 104 0.195229 0.879484
138 16 2532 2516 72.8949 132 0.521735 0.87598
139 16 2552 2536 72.9459 80 0.613578 0.874899
2020-06-08 20:13:18.037765 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.870502
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 2584 2568 73.3388 128 0.810856 0.870502
141 16 2611 2595 73.5836 108 0.0896614 0.866676
142 16 2631 2615 73.6287 80 0.152923 0.865296
143 16 2656 2640 73.8129 100 0.569885 0.86478
144 16 2677 2661 73.8836 84 0.816704 0.862674
145 16 2693 2677 73.8154 64 0.710956 0.862682
146 16 2714 2698 73.885 84 0.534859 0.862624
147 16 2730 2714 73.8177 64 2.11317 0.862827
148 16 2750 2734 73.8573 80 0.140814 0.862759
149 16 2779 2763 74.1361 116 0.597553 0.861577
150 16 2800 2784 74.2017 84 0.294405 0.859901
151 16 2813 2797 74.0547 52 0.551044 0.859506
152 16 2827 2811 73.9359 56 0.179322 0.86001
153 16 2841 2825 73.8186 56 0.537953 0.862982
154 16 2851 2835 73.599 40 0.0894164 0.862833
155 16 2870 2854 73.6144 76 0.146518 0.863878
156 16 2893 2877 73.7321 92 0.667938 0.864718
157 16 2909 2893 73.6694 64 0.35911 0.864521
158 16 2931 2915 73.7593 88 1.99247 0.864606
159 16 2954 2938 73.8738 92 0.470108 0.864151
2020-06-08 20:13:38.059453 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.863755
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 2971 2955 73.8363 68 0.913875 0.863755
161 16 2985 2969 73.7255 56 0.122776 0.86388
162 16 3015 2999 74.0109 120 0.292877 0.862976
163 16 3034 3018 74.0229 76 1.14578 0.862273
164 16 3052 3036 74.0105 72 0.194916 0.861431
165 16 3068 3052 73.9484 64 2.15639 0.861781
166 16 3083 3067 73.8643 60 2.59581 0.861478
167 16 3097 3081 73.7573 56 0.217251 0.861189
168 16 3112 3096 73.6754 60 0.744343 0.862415
169 16 3131 3115 73.689 76 0.432271 0.862675
170 16 3149 3133 73.679 72 0.132586 0.865311
171 16 3163 3147 73.5756 56 0.311371 0.864491
172 16 3185 3169 73.6593 88 2.54587 0.866844
173 16 3203 3187 73.6497 72 0.597465 0.86665
174 16 3213 3197 73.4563 40 1.66117 0.86732
175 16 3230 3214 73.425 68 0.525784 0.868341
176 16 3251 3235 73.485 84 0.195678 0.868587
177 16 3273 3257 73.5669 88 0.318552 0.866793
178 16 3294 3278 73.6254 84 0.456156 0.866243
179 16 3316 3300 73.7056 88 0.90386 0.86562
2020-06-08 20:13:58.066526 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.866679
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 3330 3314 73.6072 56 0.772631 0.866679
181 16 3349 3333 73.6204 76 0.506055 0.867013
182 16 3370 3354 73.6773 84 0.862603 0.866022
183 16 3389 3373 73.6899 76 0.126265 0.865657
184 16 3413 3397 73.811 96 0.472542 0.864694
185 16 3438 3422 73.9525 100 0.548133 0.863556
186 16 3459 3443 74.0064 84 0.142547 0.862001
187 16 3482 3466 74.1025 92 0.197975 0.86119
188 16 3504 3488 74.1763 88 1.21617 0.860449
189 16 3520 3504 74.1224 64 0.617482 0.860191
190 16 3538 3522 74.1111 72 1.42981 0.860781
191 16 3555 3539 74.0791 68 0.67103 0.860066
192 16 3575 3559 74.1098 80 0.209799 0.859894
193 16 3595 3579 74.1403 80 0.102169 0.860217
194 16 3608 3592 74.0256 52 0.551272 0.860418
195 16 3623 3607 73.9537 60 0.549342 0.860765
196 16 3641 3625 73.9436 72 0.284283 0.862124
197 16 3658 3642 73.9134 68 1.3556 0.863595
198 16 3672 3656 73.8229 56 0.118254 0.863518
199 16 3685 3669 73.7133 52 0.442957 0.86486
2020-06-08 20:14:18.071479 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.865218
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 3709 3693 73.8246 96 0.764446 0.865218
201 16 3729 3713 73.8552 80 0.10531 0.86423
202 16 3753 3737 73.9646 96 1.16193 0.863313
203 16 3776 3760 74.0533 92 1.35557 0.862287
204 16 3793 3777 74.0236 68 0.362237 0.86228
205 16 3818 3802 74.1502 100 1.95675 0.860927
206 16 3843 3827 74.2756 100 0.699881 0.859939
207 16 3862 3846 74.2838 76 0.373804 0.858611
208 16 3887 3871 74.4073 100 0.468576 0.858524
209 16 3904 3888 74.3767 68 0.850498 0.858089
210 16 3921 3905 74.3459 68 0.122704 0.85862
211 16 3946 3930 74.4671 100 0.688795 0.857776
212 16 3965 3949 74.4743 76 0.887289 0.85747
213 16 3986 3970 74.5189 84 0.10099 0.857015
214 16 4007 3991 74.5631 84 0.652765 0.85672
215 16 4023 4007 74.5131 64 1.24319 0.856411
216 16 4031 4015 74.3162 32 0.123267 0.856949
217 16 4051 4035 74.342 80 1.76581 0.85828
218 16 4069 4053 74.3312 72 1.24225 0.858736
219 16 4087 4071 74.3205 72 0.36971 0.8589
2020-06-08 20:14:38.080841 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.858769
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 4109 4093 74.3826 88 0.421784 0.858769
221 16 4131 4115 74.4441 88 0.643271 0.858154
222 16 4150 4134 74.451 76 1.15955 0.857912
223 16 4166 4150 74.4041 64 0.364356 0.857335
224 16 4184 4168 74.3933 72 0.556197 0.857229
225 16 4201 4185 74.3648 68 2.87592 0.857676
226 16 4220 4204 74.372 76 0.11732 0.857482
227 16 4234 4218 74.291 56 0.542402 0.857934
228 16 4250 4234 74.2458 64 0.994756 0.857719
229 16 4260 4244 74.0962 40 0.460497 0.859127
230 16 4276 4260 74.0523 64 0.166571 0.859328
231 16 4290 4274 73.9742 56 0.633513 0.859575
232 16 4302 4286 73.8622 48 4.20751 0.8606
233 16 4319 4303 73.837 68 0.704034 0.862291
234 16 4337 4321 73.8289 72 0.767082 0.863067
235 16 4355 4339 73.8211 72 0.307592 0.864738
236 16 4369 4353 73.7455 56 1.06873 0.865095
237 16 4393 4377 73.8393 96 0.640935 0.865275
238 16 4410 4394 73.8148 68 0.217787 0.86458
239 16 4430 4414 73.8406 80 0.112997 0.864407
2020-06-08 20:14:58.085368 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.86433
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 4448 4432 73.8329 72 0.750785 0.86433
241 16 4466 4450 73.8252 72 0.980823 0.86425
242 16 4485 4469 73.8327 76 0.330765 0.863928
243 16 4496 4480 73.7099 44 2.53958 0.863934
244 16 4517 4501 73.752 84 0.300727 0.865453
245 16 4528 4512 73.6304 44 1.32415 0.866626
246 16 4550 4534 73.6887 88 0.683801 0.867046
247 16 4565 4549 73.6333 60 0.559875 0.866727
248 16 4586 4570 73.675 84 1.58628 0.867045
249 16 4610 4594 73.7646 96 0.619608 0.865653
250 16 4631 4615 73.8054 84 0.893729 0.865241
251 16 4643 4627 73.7026 48 1.36248 0.865516
252 16 4668 4652 73.8068 100 0.111955 0.865293
253 16 4682 4666 73.7364 56 1.03799 0.865374
254 16 4698 4682 73.698 64 0.118631 0.865129
255 16 4712 4696 73.6286 56 2.77725 0.865131
256 16 4734 4718 73.6846 88 0.200313 0.866035
257 16 4747 4731 73.6002 52 0.377505 0.867007
258 16 4767 4751 73.6249 80 0.333156 0.866838
259 16 4782 4766 73.5723 60 0.391211 0.867606
2020-06-08 20:15:18.095119 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.867525
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 4801 4785 73.5815 76 0.801623 0.867525
261 16 4828 4812 73.7132 108 1.46074 0.867074
262 15 4849 4834 73.7675 88 0.0946217 0.8662
263 16 4869 4853 73.776 76 0.485207 0.865938
264 16 4887 4871 73.7692 72 1.02176 0.865994
265 16 4896 4880 73.6267 36 1.02482 0.866309
266 16 4916 4900 73.6506 80 1.75639 0.867135
267 16 4930 4914 73.5844 56 0.793195 0.866968
268 16 4949 4933 73.5934 76 0.269814 0.866848
269 16 4965 4949 73.5576 64 0.47214 0.867088
270 16 4984 4968 73.5666 76 0.0818159 0.867103
271 16 5003 4987 73.5755 76 0.355008 0.867918
272 16 5022 5006 73.5843 76 1.37336 0.86837
273 16 5034 5018 73.4906 48 1.08197 0.868069
274 16 5050 5034 73.4558 64 0.574166 0.868039
275 16 5063 5047 73.3775 52 0.410347 0.868049
276 16 5080 5064 73.3579 68 0.462378 0.868981
277 16 5097 5081 73.3386 68 0.595854 0.869527
278 16 5117 5101 73.3624 80 0.144244 0.869842
279 16 5133 5117 73.3289 64 0.152577 0.869983
2020-06-08 20:15:38.101770 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.869712
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 5153 5137 73.3526 80 0.265636 0.869712
281 16 5181 5165 73.49 112 0.100195 0.86907
282 16 5200 5184 73.4989 76 0.531119 0.868996
283 16 5210 5194 73.3806 40 0.164295 0.869019
284 16 5230 5214 73.4038 80 0.501973 0.869291
285 16 5248 5232 73.3979 72 0.139468 0.869976
286 16 5265 5249 73.379 68 0.805504 0.869721
287 16 5280 5264 73.3324 60 0.837478 0.870432
288 16 5289 5273 73.2027 36 1.06794 0.870923
289 16 5305 5289 73.1708 64 2.13368 0.872454
290 16 5324 5308 73.1804 76 0.284596 0.872336
291 16 5340 5324 73.1487 64 0.72313 0.872466
292 16 5358 5342 73.1447 72 0.31993 0.872824
293 16 5375 5359 73.1271 68 0.252697 0.873302
294 16 5391 5375 73.096 64 0.106871 0.873439
295 16 5409 5393 73.0922 72 0.59772 0.872936
296 16 5422 5406 73.0209 52 0.109273 0.873116
297 16 5439 5423 73.0033 68 0.444732 0.873416
298 16 5454 5438 72.9597 60 0.32053 0.874756
299 16 5476 5460 73.0099 88 1.80855 0.875165
2020-06-08 20:15:58.113911 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.8745
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 5493 5477 72.993 68 0.442914 0.8745
301 16 5514 5498 73.0292 84 0.454339 0.874582
302 16 5532 5516 73.0258 72 2.05633 0.874407
303 16 5549 5533 73.0092 68 1.9747 0.874562
304 16 5565 5549 72.9795 64 0.556228 0.874017
305 16 5583 5567 72.9763 72 0.584978 0.874186
306 16 5605 5589 73.0252 88 2.75141 0.874373
307 16 5625 5609 73.0478 80 0.394376 0.873885
308 16 5642 5626 73.0314 68 0.687721 0.87404
309 16 5660 5644 73.0279 72 0.835038 0.873733
310 16 5682 5666 73.0762 88 0.481862 0.874148
311 16 5700 5684 73.0726 72 0.0981046 0.873689
312 16 5717 5701 73.0563 68 0.173722 0.873277
313 16 5734 5718 73.0397 68 0.454973 0.873389
314 16 5751 5735 73.0236 68 0.584949 0.874248
315 16 5769 5753 73.0203 72 0.451557 0.874648
316 16 5784 5768 72.9791 60 0.379275 0.87474
317 16 5801 5785 72.9633 68 0.2345 0.875049
318 16 5826 5810 73.0483 100 0.633942 0.875274
319 16 5842 5826 73.0199 64 0.526023 0.875175
2020-06-08 20:16:18.121757 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.875051
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 5857 5841 72.9791 60 0.869822 0.875051
321 16 5877 5861 73.001 80 2.01968 0.875335
322 16 5890 5874 72.9357 52 0.500115 0.87541
323 16 5912 5896 72.9823 88 0.476156 0.875176
324 16 5930 5914 72.9793 72 3.12918 0.87563
325 16 5950 5934 73.0008 80 0.417863 0.875449
326 16 5971 5955 73.0345 84 0.387265 0.874447
327 16 5994 5978 73.0924 92 0.912747 0.873835
328 16 6011 5995 73.0769 68 0.340952 0.873547
329 16 6035 6019 73.1465 96 1.76545 0.873406
330 16 6055 6039 73.1672 80 0.439994 0.872812
331 16 6073 6057 73.1636 72 0.182007 0.872341
332 16 6092 6076 73.1721 76 0.539628 0.872316
333 16 6111 6095 73.1806 76 0.397292 0.872843
334 16 6127 6111 73.153 64 0.686426 0.872999
335 16 6150 6134 73.2092 92 1.55672 0.873199
336 16 6167 6151 73.1937 68 0.745112 0.872449
337 16 6188 6172 73.2257 84 0.416634 0.872576
338 16 6213 6197 73.3048 100 0.221025 0.871792
339 16 6237 6221 73.3717 96 0.380544 0.87119
2020-06-08 20:16:38.125389 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.870645
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 6255 6239 73.3677 72 0.237674 0.870645
341 16 6273 6257 73.3636 72 1.91148 0.870875
342 16 6285 6269 73.2894 48 0.625473 0.870964
343 16 6304 6288 73.2972 76 0.684181 0.871232
344 16 6320 6304 73.2702 64 1.31585 0.871307
345 16 6337 6321 73.2548 68 0.533962 0.871754
346 16 6354 6338 73.2396 68 0.567822 0.871976
347 16 6370 6354 73.213 64 0.282414 0.872345
348 16 6392 6376 73.2553 88 1.0019 0.872403
349 16 6413 6397 73.2861 84 0.834793 0.87182
350 16 6434 6418 73.3166 84 0.690901 0.871459
351 16 6456 6440 73.3584 88 1.84694 0.871126
352 16 6470 6454 73.3091 56 0.764582 0.871519
353 16 6484 6468 73.26 56 0.369115 0.871565
354 16 6500 6484 73.2338 64 2.27152 0.871695
355 16 6518 6502 73.2303 72 0.420901 0.872285
356 16 6536 6520 73.2268 72 0.72114 0.872575
357 16 6556 6540 73.2434 80 2.14448 0.872497
358 16 6573 6557 73.2288 68 1.11625 0.872674
359 16 6588 6572 73.1919 60 0.364302 0.872606
2020-06-08 20:16:58.141106 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.873117
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 6604 6588 73.1663 64 0.174063 0.873117
361 16 6623 6607 73.1742 76 0.862532 0.873303
362 16 6646 6630 73.2261 92 0.89393 0.872871
363 16 6665 6649 73.2335 76 0.837989 0.872608
364 16 6685 6669 73.252 80 0.781318 0.872458
365 16 6706 6690 73.2814 84 1.0914 0.872143
366 16 6723 6707 73.2669 68 0.451284 0.871841
367 16 6738 6722 73.2307 60 0.506314 0.872117
368 16 6759 6743 73.2599 84 0.625485 0.872373
369 16 6775 6759 73.2348 64 0.687533 0.87251
370 16 6790 6774 73.1986 60 0.67488 0.872351
371 16 6807 6791 73.1846 68 0.488116 0.872876
372 16 6826 6810 73.1921 76 1.86775 0.87267
373 16 6849 6833 73.2425 92 0.102388 0.872601
374 16 6870 6854 73.2712 84 0.963751 0.872613
375 16 6892 6876 73.3104 88 0.738477 0.871927
376 16 6911 6895 73.3176 76 0.712372 0.871598
377 16 6929 6913 73.314 72 0.500957 0.871591
378 16 6947 6931 73.3105 72 0.308952 0.871554
379 16 6968 6952 73.3387 84 0.469417 0.871198
2020-06-08 20:17:18.147871 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.870984
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 6989 6973 73.3667 84 0.438869 0.870984
381 16 7008 6992 73.3736 76 0.980196 0.871126
382 16 7025 7009 73.3595 68 0.302758 0.870969
383 16 7048 7032 73.4078 92 0.579218 0.871022
384 16 7065 7049 73.3937 68 0.215526 0.87051
385 16 7081 7065 73.3693 64 0.15136 0.870739
386 16 7108 7092 73.4589 108 0.933536 0.870274
387 16 7128 7112 73.4758 80 1.34933 0.870011
388 16 7147 7131 73.4822 76 0.278081 0.869447
389 16 7165 7149 73.4784 72 0.0909752 0.869213
390 16 7184 7168 73.4848 76 1.68477 0.86937
391 16 7196 7180 73.4196 48 2.09545 0.869853
392 16 7212 7196 73.3956 64 1.25778 0.870143
393 16 7224 7208 73.3304 48 1.51674 0.870445
394 16 7243 7227 73.3372 76 0.511055 0.870731
395 16 7259 7243 73.3135 64 1.0424 0.870905
396 16 7278 7262 73.3203 76 0.144077 0.871276
397 16 7297 7281 73.327 76 0.53167 0.870933
398 16 7316 7300 73.3325 76 0.261177 0.870826
399 16 7329 7313 73.279 52 0.800692 0.871398
2020-06-08 20:17:38.162475 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.871241
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 7347 7331 73.2758 72 0.440267 0.871241
401 16 7365 7349 73.2721 72 0.41204 0.871827
402 16 7381 7365 73.249 64 0.34712 0.871989
403 16 7401 7385 73.2657 80 0.446505 0.87246
404 16 7415 7399 73.2229 56 1.57592 0.872653
405 16 7434 7418 73.2298 76 1.14035 0.872554
406 16 7458 7442 73.2858 96 0.937192 0.872384
407 16 7477 7461 73.2923 76 1.24865 0.871886
408 16 7491 7475 73.2499 56 0.689652 0.872057
409 16 7519 7503 73.3446 112 0.105469 0.871771
410 16 7535 7519 73.3218 64 0.592535 0.871449
411 16 7558 7542 73.3659 92 0.490002 0.870978
412 16 7572 7556 73.3237 56 0.613341 0.871017
413 16 7586 7570 73.2818 56 0.0930081 0.870963
414 16 7598 7582 73.2207 48 0.336964 0.871426
415 16 7616 7600 73.2177 72 0.630183 0.872075
416 16 7635 7619 73.2244 76 0.362077 0.872057
417 16 7651 7635 73.2022 64 1.79705 0.872563
418 16 7665 7649 73.1611 56 0.524832 0.872746
419 16 7674 7658 73.0724 36 2.70833 0.873783
2020-06-08 20:17:58.176349 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.874201
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 7692 7676 73.0698 72 0.606834 0.874201
421 16 7707 7691 73.0388 60 0.530377 0.874531
422 16 7729 7713 73.0742 88 1.5095 0.874335
423 16 7753 7737 73.1283 96 0.729968 0.874409
424 16 7771 7755 73.1256 72 0.899083 0.874033
425 16 7788 7772 73.1135 68 1.56701 0.873923
426 16 7811 7795 73.1578 92 0.146697 0.87367
427 16 7826 7810 73.127 60 0.525777 0.87344
428 16 7843 7827 73.115 68 0.833314 0.873993
429 16 7858 7842 73.0844 60 0.749745 0.873882
430 16 7877 7861 73.091 76 2.57347 0.873901
431 16 7895 7879 73.0884 72 0.480756 0.87408
432 16 7905 7889 73.0119 40 0.667566 0.874432
433 16 7921 7905 72.9908 64 0.271584 0.874489
434 16 7938 7922 72.9793 68 0.57887 0.874595
435 16 7956 7940 72.977 72 0.788886 0.875303
436 16 7971 7955 72.9472 60 0.991915 0.875371
437 16 7991 7975 72.9633 80 0.781064 0.875913
438 16 8014 7998 73.0068 92 0.531112 0.875517
439 16 8031 8015 72.9953 68 1.0933 0.875622
2020-06-08 20:18:18.181905 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.875376
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 8053 8037 73.0294 88 1.48891 0.875376
441 16 8069 8053 73.0089 64 0.667919 0.875279
442 16 8090 8074 73.0337 84 0.515497 0.875288
443 16 8111 8095 73.0584 84 0.969861 0.875129
444 16 8131 8115 73.074 80 0.586855 0.874649
445 16 8150 8134 73.0806 76 0.911212 0.874691
446 16 8162 8146 73.0243 48 1.47094 0.875018
447 16 8182 8166 73.0399 80 0.984956 0.875177
448 16 8200 8184 73.0375 72 0.511365 0.87497
449 16 8221 8205 73.0619 84 0.466792 0.874861
450 16 8241 8225 73.0773 80 0.498444 0.874711
451 16 8261 8245 73.0926 80 1.12324 0.874394
452 16 8277 8261 73.0725 64 0.358167 0.874631
453 16 8292 8276 73.0434 60 1.55083 0.874833
454 16 8317 8301 73.1027 100 1.08791 0.874758
455 16 8335 8319 73.0998 72 0.485704 0.87463
456 16 8364 8348 73.1937 116 0.106496 0.873635
457 16 8379 8363 73.1648 60 0.6704 0.873633
458 16 8391 8375 73.1098 48 0.142305 0.873526
459 16 8405 8389 73.0726 56 2.03715 0.874052
2020-06-08 20:18:38.190489 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.874502
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 8415 8399 73.0007 40 3.17999 0.874502
461 16 8430 8414 72.9724 60 1.38506 0.875866
462 16 8449 8433 72.979 76 0.138435 0.876087
463 16 8471 8455 73.0114 88 0.982731 0.875894
464 16 8489 8473 73.0092 72 1.12901 0.875441
465 16 8511 8495 73.0408 88 0.116214 0.875281
466 16 8526 8510 73.0128 60 1.47453 0.875189
467 16 8546 8530 73.0277 80 0.578646 0.874975
468 16 8562 8546 73.0084 64 1.86771 0.874965
469 16 8583 8567 73.0318 84 0.244918 0.874656
470 16 8599 8583 73.0126 64 0.275093 0.874632
471 16 8619 8603 73.0274 80 0.128726 0.874655
472 16 8634 8618 72.9998 60 0.572616 0.874788
473 16 8647 8631 72.9554 52 0.479875 0.875331
474 16 8659 8643 72.9027 48 0.688491 0.875632
475 16 8675 8659 72.884 64 2.74441 0.876297
476 16 8690 8674 72.8569 60 0.0918844 0.876375
477 16 8706 8690 72.8383 64 0.498907 0.876704
478 16 8721 8705 72.8114 60 0.751913 0.877124
479 16 8739 8723 72.8097 72 3.47103 0.877329
2020-06-08 20:18:58.197532 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.87719
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 8755 8739 72.7913 64 0.111138 0.87719
481 16 8766 8750 72.7315 44 0.506961 0.877573
482 16 8783 8767 72.7216 68 0.504694 0.877913
483 16 8794 8778 72.6621 44 0.489185 0.878579
484 16 8810 8794 72.6442 64 0.177624 0.87876
485 16 8825 8809 72.6181 60 1.07506 0.879152
486 16 8838 8822 72.5757 52 0.656139 0.87968
487 16 8854 8838 72.558 64 0.673322 0.880395
488 15 8869 8854 72.5405 64 0.509493 0.880211
489 16 8883 8867 72.4984 52 3.05994 0.880934
490 16 8893 8877 72.4321 40 1.04804 0.881329
491 16 8910 8894 72.4231 68 0.863421 0.881913
492 16 8926 8910 72.406 64 1.97329 0.882478
493 16 8941 8925 72.3808 60 0.448451 0.882659
494 16 8955 8939 72.3476 56 0.461017 0.882659
495 16 8970 8954 72.3227 60 0.642146 0.882912
496 16 8988 8972 72.322 72 1.11537 0.883378
497 16 9004 8988 72.3052 64 0.750038 0.883875
498 16 9026 9010 72.3367 88 0.223701 0.883509
499 16 9047 9031 72.36 84 0.901783 0.883695
2020-06-08 20:19:18.202528 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.883166
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 9073 9057 72.4231 104 0.445671 0.883166
501 16 9090 9074 72.4143 68 0.344235 0.882752
502 16 9114 9098 72.4612 96 0.574177 0.882186
503 16 9132 9116 72.4603 72 0.296478 0.881961
504 16 9146 9130 72.4276 56 2.85507 0.882088
505 16 9165 9149 72.4347 76 0.61285 0.882188
506 16 9185 9169 72.4496 80 0.568072 0.882126
507 16 9195 9179 72.3855 40 0.288035 0.8821
508 16 9215 9199 72.4005 80 0.444345 0.882319
509 16 9236 9220 72.4232 84 0.443104 0.882563
510 16 9258 9242 72.4537 88 0.512261 0.882188
511 16 9274 9258 72.4372 64 2.43589 0.882312
512 16 9284 9268 72.3738 40 0.431717 0.882272
513 16 9294 9278 72.3107 40 0.362419 0.882814
514 16 9308 9292 72.279 56 0.652186 0.883425
515 16 9330 9314 72.3095 88 0.361627 0.883462
516 16 9348 9332 72.3089 72 1.91057 0.883873
517 16 9364 9348 72.2928 64 0.236557 0.883988
518 16 9380 9364 72.2767 64 0.75314 0.884531
519 16 9396 9380 72.2608 64 0.466132 0.884371
2020-06-08 20:19:38.206381 min lat: 0.0718277 max lat: 4.4007 avg lat: 0.88435
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 9417 9401 72.2833 84 0.510479 0.88435
521 16 9437 9421 72.298 80 0.775545 0.884331
522 16 9454 9438 72.2898 68 1.57618 0.884393
523 16 9467 9451 72.251 52 0.653131 0.884449
524 16 9489 9473 72.281 88 0.48567 0.884436
525 16 9520 9504 72.3794 124 0.477646 0.883514
526 16 9539 9523 72.3863 76 0.778714 0.882993
527 16 9563 9547 72.431 96 1.65646 0.882646
528 16 9584 9568 72.4529 84 0.101928 0.882272
529 16 9602 9586 72.452 72 0.705437 0.881983
530 16 9627 9611 72.5038 100 0.643886 0.882033
531 16 9643 9627 72.4878 64 0.640231 0.882064
532 16 9663 9647 72.5019 80 0.450927 0.881662
533 16 9686 9670 72.5384 92 0.722139 0.881466
534 16 9702 9686 72.5224 64 0.493813 0.881443
535 16 9719 9703 72.5139 68 0.893322 0.881658
536 16 9738 9722 72.5204 76 0.416984 0.881512
537 16 9755 9739 72.5117 68 1.93052 0.881418
538 16 9768 9752 72.4736 52 2.15586 0.881727
539 16 9781 9765 72.4356 52 0.869417 0.881877
2020-06-08 20:19:58.213102 min lat: 0.0718277 max lat: 4.4007 avg lat: 0.882211
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 9800 9784 72.4422 76 0.359025 0.882211
541 16 9810 9794 72.3822 40 0.580788 0.882555
542 16 9836 9820 72.4405 104 0.693178 0.882941
543 16 9854 9838 72.4397 72 0.982478 0.8828
544 16 9872 9856 72.4388 72 0.546233 0.882561
545 16 9888 9872 72.4233 64 0.183194 0.882458
546 16 9911 9895 72.4591 92 0.518778 0.882679
547 16 9926 9910 72.4363 60 0.673578 0.882286
548 16 9943 9927 72.4282 68 1.47013 0.882573
549 16 9963 9947 72.4419 80 0.470011 0.8824
550 16 9980 9964 72.4338 68 1.13835 0.882455
551 16 9997 9981 72.4257 68 0.364902 0.882334
552 16 10015 9999 72.4249 72 0.375624 0.882508
553 16 10033 10017 72.4241 72 0.293664 0.882372
554 16 10048 10032 72.4017 60 2.33343 0.882648
555 16 10063 10047 72.3792 60 2.31065 0.88295
556 16 10081 10065 72.3785 72 0.66063 0.883219
557 16 10096 10080 72.3563 60 1.07249 0.883129
558 16 10114 10098 72.3556 72 0.750912 0.883415
559 16 10131 10115 72.3478 68 0.653086 0.883261
2020-06-08 20:20:18.219006 min lat: 0.0718277 max lat: 4.4007 avg lat: 0.883564
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 10150 10134 72.3543 76 0.512078 0.883564
561 16 10172 10156 72.3821 88 0.545524 0.883269
562 16 10188 10172 72.3672 64 0.630823 0.883461
563 16 10203 10187 72.3452 60 0.575679 0.883473
564 16 10221 10205 72.3445 72 0.0998445 0.883611
565 16 10241 10225 72.3559 80 0.723494 0.883808
566 16 10259 10243 72.3552 72 0.372775 0.883602
567 16 10270 10254 72.3052 44 0.978835 0.883669
568 16 10290 10274 72.3187 80 1.58439 0.883981
569 16 10314 10298 72.3603 96 1.12667 0.883854
570 16 10335 10319 72.3807 84 0.476324 0.883195
571 16 10351 10335 72.366 64 0.804836 0.882921
572 16 10363 10347 72.3234 48 0.642962 0.882924
573 16 10383 10367 72.3368 80 0.523085 0.883201
574 16 10399 10383 72.3222 64 0.776479 0.883045
575 16 10416 10400 72.3147 68 0.318009 0.88346
576 16 10436 10420 72.328 80 0.624059 0.883577
577 16 10455 10439 72.3343 76 0.789692 0.883085
578 16 10477 10461 72.3614 88 0.462701 0.883291
579 16 10491 10475 72.3332 56 0.299997 0.883183
2020-06-08 20:20:38.239770 min lat: 0.0718277 max lat: 4.4007 avg lat: 0.883197
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 10513 10497 72.3601 88 0.468421 0.883197
581 16 10526 10510 72.3251 52 0.516525 0.883276
582 16 10542 10526 72.3108 64 0.150456 0.883449
583 16 10561 10545 72.3171 76 0.152284 0.883633
584 16 10582 10566 72.337 84 0.903022 0.883638
585 16 10602 10586 72.3501 80 1.3925 0.883683
586 16 10624 10608 72.3767 88 0.584053 0.883096
587 16 10648 10632 72.4169 96 0.817164 0.882926
588 16 10668 10652 72.4298 80 0.501111 0.882698
589 16 10688 10672 72.4426 80 0.508018 0.882566
590 16 10721 10705 72.5434 132 0.569683 0.882076
591 16 10751 10735 72.6237 120 0.113236 0.880791
592 16 10782 10766 72.7104 124 0.189927 0.879653
593 16 10812 10796 72.7901 120 0.201906 0.878778
594 16 10852 10836 72.9368 160 0.602972 0.877035
595 16 10888 10872 73.0562 144 0.497497 0.875504
596 16 10923 10907 73.1684 140 0.33945 0.874248
597 16 10956 10940 73.2669 132 0.418792 0.873231
598 16 10985 10969 73.3383 116 0.124343 0.872056
599 16 11022 11006 73.4629 148 0.500349 0.870872
2020-06-08 20:20:58.244444 min lat: 0.0718277 max lat: 4.4007 avg lat: 0.869454
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 16 11055 11039 73.5604 132 0.552603 0.869454
Total time run: 600.488
Total writes made: 11055
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 73.6401
Stddev Bandwidth: 17.4025
Max bandwidth (MB/sec): 160
Min bandwidth (MB/sec): 32
Average IOPS: 18
Stddev IOPS: 4.35062
Max IOPS: 40
Min IOPS: 8
Average Latency(s): 0.869061
Stddev Latency(s): 0.692347
Max latency(s): 4.4007
Min latency(s): 0.0718277
Total time run: 600.488
Total writes made: 11055
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 73.6401
Stddev Bandwidth: 17.4025
Max bandwidth (MB/sec): 160
Min bandwidth (MB/sec): 32
Average IOPS: 18
Stddev IOPS: 4.35062
Max IOPS: 40
Min IOPS: 8
Average Latency(s): 0.869061
Stddev Latency(s): 0.692347
Max latency(s): 4.4007
Min latency(s): 0.0718277
Sequential Read Test, pool with 256 PGs
podman exec ceph-mon-f5-422-01 rados bench -p testbench2 600 seq | tee --append $(date +%F)-256PGs-rados-test-read-seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 52 36 143.952 144 0.10248 0.257708
2 16 80 64 127.97 112 0.395358 0.388466
3 16 112 96 127.975 128 0.332911 0.416507
4 16 141 125 124.848 116 0.0534434 0.447449
5 16 174 158 126.272 132 0.0459037 0.452606
6 16 203 187 124.559 116 0.0904399 0.466416
7 16 235 219 125.047 128 0.717821 0.475117
8 16 260 244 121.916 100 1.31779 0.49151
9 16 289 273 121.257 116 0.316016 0.494611
10 16 320 304 121.53 124 0.62921 0.497556
11 16 355 339 123.179 140 1.35594 0.49199
12 15 388 373 124.227 136 0.209883 0.493579
13 16 421 405 124.385 128 0.0983189 0.489222
14 16 452 436 124.357 124 0.248015 0.488435
15 16 483 467 124.332 124 0.0657991 0.493576
16 16 512 496 123.811 116 0.407897 0.498002
17 15 540 525 123.33 116 0.223065 0.498863
18 16 569 553 122.701 112 0.0935895 0.500051
19 16 598 582 122.348 116 0.0550563 0.502108
2020-06-08 20:23:55.835704 min lat: 0.0229199 max lat: 1.96882 avg lat: 0.501858
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 630 614 122.629 128 0.27236 0.501858
21 16 662 646 122.884 128 1.93863 0.501786
22 16 691 675 122.57 116 0.5895 0.506386
23 16 721 705 122.458 120 0.318985 0.505303
24 15 754 739 122.989 136 1.26615 0.505394
25 16 783 767 122.546 112 1.67834 0.504546
26 16 813 797 122.447 120 0.342159 0.507667
27 16 847 831 122.948 136 1.39683 0.505411
28 15 876 861 122.813 120 0.316757 0.504973
29 16 903 887 122.161 104 0.487501 0.507258
30 15 933 918 122.211 124 0.387951 0.510148
31 16 962 946 121.882 112 0.196314 0.510662
32 16 994 978 122.072 128 1.81746 0.511042
33 16 1023 1007 121.888 116 1.22816 0.511654
34 16 1054 1038 121.949 124 0.47151 0.510387
35 16 1082 1066 121.665 112 0.325294 0.510798
36 16 1121 1105 122.617 156 1.09879 0.510156
37 16 1155 1139 122.978 136 0.0704184 0.508363
38 16 1183 1167 122.689 112 0.274135 0.509345
39 16 1214 1198 122.722 124 0.333837 0.509944
2020-06-08 20:24:15.898027 min lat: 0.0229199 max lat: 2.16235 avg lat: 0.510367
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1242 1226 122.324 112 1.55592 0.510367
41 16 1268 1252 121.878 104 0.233863 0.511122
42 16 1299 1283 121.927 124 0.0489042 0.514665
43 16 1329 1313 121.882 120 1.22462 0.51466
44 16 1363 1347 122.202 136 0.868778 0.514214
45 16 1390 1374 121.886 108 0.552348 0.51362
46 16 1425 1409 122.279 140 0.0486978 0.511113
47 16 1453 1437 122.06 112 0.0799922 0.512212
48 16 1487 1471 122.349 136 0.380298 0.510939
49 16 1521 1505 122.627 136 0.250579 0.510254
50 15 1547 1532 122.334 108 0.726027 0.513268
51 16 1576 1560 122.13 112 0.221502 0.513779
52 16 1608 1592 122.242 128 0.43852 0.512775
53 16 1642 1626 122.5 136 0.404703 0.511807
54 16 1674 1658 122.6 128 0.274548 0.511684
55 16 1707 1691 122.77 132 0.320383 0.510371
56 16 1739 1723 122.863 128 0.214707 0.510048
57 16 1769 1753 122.812 120 1.60844 0.509381
58 16 1797 1781 122.625 112 0.198468 0.510414
59 16 1828 1812 122.645 124 1.29003 0.511771
2020-06-08 20:24:35.906436 min lat: 0.0229199 max lat: 2.16235 avg lat: 0.511011
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 1859 1843 122.665 124 0.0929808 0.511011
61 16 1890 1874 122.685 124 0.16983 0.510338
62 16 1924 1908 122.899 136 1.95647 0.511738
63 16 1954 1938 122.851 120 0.174255 0.510767
64 16 1987 1971 122.989 132 0.84841 0.511279
65 15 2012 1997 122.679 104 0.0474931 0.512642
66 16 2045 2029 122.76 128 0.310809 0.512644
67 15 2075 2060 122.738 124 0.573777 0.512099
68 16 2105 2089 122.639 116 0.0646949 0.511945
69 16 2138 2122 122.774 132 0.0624156 0.512316
70 16 2169 2153 122.791 124 1.36978 0.511709
71 15 2198 2183 122.703 120 0.0569101 0.512025
72 16 2232 2216 122.832 132 0.153078 0.511764
73 16 2262 2246 122.793 120 0.199818 0.511774
74 16 2293 2277 122.809 124 0.339496 0.51146
75 16 2324 2308 122.824 124 0.048679 0.51081
76 16 2353 2337 122.735 116 0.312595 0.511596
77 16 2384 2368 122.751 124 2.03753 0.511888
78 16 2413 2397 122.622 116 0.955439 0.511969
79 16 2443 2427 122.588 120 0.299719 0.51355
2020-06-08 20:24:56.000431 min lat: 0.0227928 max lat: 2.36116 avg lat: 0.513632
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 2473 2457 122.555 120 0.261607 0.513632
81 16 2512 2496 122.967 156 0.242555 0.511745
82 16 2545 2529 123.076 132 1.67602 0.510647
83 16 2579 2563 123.231 136 0.35015 0.510781
84 16 2610 2594 123.239 124 0.311045 0.511449
85 16 2641 2625 123.248 124 0.292918 0.51041
86 16 2668 2652 123.07 108 1.74807 0.511303
87 16 2698 2682 123.035 120 0.226664 0.5109
88 16 2727 2711 122.955 116 0.0801858 0.51106
89 16 2763 2747 123.191 144 1.41941 0.510999
90 16 2794 2778 123.2 124 0.275317 0.51053
91 16 2824 2808 123.164 120 1.44233 0.510651
92 16 2856 2840 123.217 128 0.2362 0.510411
93 16 2888 2872 123.265 128 1.8657 0.509986
94 16 2921 2905 123.356 132 0.0872072 0.509621
95 15 2958 2943 123.628 152 0.0598051 0.508228
96 16 2992 2976 123.715 132 0.107187 0.508602
97 16 3020 3004 123.594 112 0.583915 0.508965
98 16 3050 3034 123.557 120 1.37677 0.509671
99 16 3080 3064 123.52 120 1.66099 0.509464
2020-06-08 20:25:16.031657 min lat: 0.0227928 max lat: 2.36752 avg lat: 0.509699
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3110 3094 123.484 120 1.6026 0.509699
101 16 3144 3128 123.578 136 0.0654255 0.509263
102 16 3175 3159 123.582 124 0.329964 0.509369
103 16 3200 3184 123.353 100 0.21294 0.509977
104 16 3231 3215 123.357 124 1.71388 0.510178
105 16 3264 3248 123.439 132 1.72292 0.510106
106 16 3289 3273 123.219 100 0.0469407 0.509794
107 16 3324 3308 123.375 140 0.0559577 0.510508
108 16 3354 3338 123.344 120 1.71473 0.510917
109 16 3389 3373 123.496 140 0.256227 0.509952
110 16 3422 3406 123.573 132 0.0522499 0.510271
111 16 3454 3438 123.613 128 0.0741706 0.510096
112 16 3485 3469 123.615 124 0.346869 0.510085
113 16 3510 3494 123.406 100 0.408131 0.510194
114 16 3537 3521 123.271 108 0.072818 0.511152
115 16 3573 3557 123.41 144 0.0773894 0.509976
116 16 3603 3587 123.381 120 1.61408 0.510598
117 15 3637 3622 123.514 140 0.0639511 0.510057
118 16 3671 3655 123.585 132 1.19071 0.510168
119 16 3701 3685 123.555 120 0.258496 0.509793
2020-06-08 20:25:36.108114 min lat: 0.0227928 max lat: 2.36752 avg lat: 0.508928
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 15 3731 3716 123.558 124 0.0414287 0.508928
121 15 3763 3748 123.594 128 0.0556417 0.509653
122 16 3795 3779 123.592 124 0.0562003 0.510296
123 16 3828 3812 123.66 132 0.0570671 0.510063
124 16 3859 3843 123.663 124 0.262134 0.508943
125 15 3888 3873 123.632 120 2.94952 0.509582
126 16 3914 3898 123.445 100 0.42958 0.510385
127 16 3943 3927 123.386 116 1.12855 0.511349
128 16 3971 3955 123.297 112 1.13273 0.51139
129 16 3998 3982 123.179 108 1.12062 0.51201
130 16 4033 4017 123.305 140 0.0689844 0.511214
131 16 4066 4050 123.371 132 0.297624 0.510419
132 15 4095 4080 123.343 120 0.0498298 0.511205
133 16 4126 4110 123.318 120 0.705154 0.511123
134 16 4154 4138 123.233 112 0.0619335 0.511476
135 16 4182 4166 123.15 112 0.154478 0.511353
136 16 4212 4196 123.127 120 1.46591 0.512014
137 16 4238 4222 122.988 104 0.470556 0.512629
138 15 4269 4254 123.018 128 0.109019 0.512289
139 16 4303 4287 123.082 132 0.210095 0.511941
2020-06-08 20:25:56.129699 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.511053
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 4340 4324 123.26 148 0.0884521 0.511053
141 16 4367 4351 123.151 108 1.55959 0.511664
142 16 4394 4378 123.045 108 0.19392 0.511737
143 16 4426 4410 123.079 128 1.48088 0.511844
144 16 4455 4439 123.03 116 0.443785 0.512202
145 16 4486 4470 123.037 124 0.0610501 0.512105
146 16 4522 4506 123.18 144 0.64723 0.511634
147 16 4552 4536 123.158 120 0.164321 0.511933
148 16 4579 4563 123.056 108 0.217635 0.511822
149 16 4614 4598 123.169 140 2.19509 0.511897
150 15 4643 4628 123.146 120 0.209133 0.511937
151 16 4675 4659 123.152 124 0.575137 0.512007
152 16 4706 4690 123.157 124 0.0768242 0.512139
153 16 4736 4720 123.136 120 0.0726477 0.511735
154 15 4766 4751 123.14 124 0.283298 0.511607
155 16 4801 4785 123.222 136 0.964164 0.512191
156 16 4833 4817 123.253 128 0.325812 0.512042
157 16 4860 4844 123.156 108 0.297552 0.512269
158 16 4889 4873 123.11 116 0.46431 0.512605
159 16 4916 4900 123.015 108 0.511602 0.513101
2020-06-08 20:26:16.144374 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.513083
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 4943 4927 122.916 108 1.23488 0.513083
161 16 4974 4958 122.923 124 0.673927 0.512792
162 16 5006 4990 122.945 128 1.60016 0.513208
163 16 5036 5020 122.927 120 0.306099 0.513255
164 16 5066 5050 122.909 120 0.066428 0.512953
165 16 5095 5079 122.867 116 0.389608 0.512774
166 16 5126 5110 122.865 124 0.302484 0.512145
167 16 5163 5147 123.015 148 0.247665 0.512363
168 16 5196 5180 123.068 132 1.96385 0.512461
169 16 5227 5211 123.073 124 0.171269 0.512254
170 16 5260 5244 123.125 132 0.0228844 0.512298
171 16 5287 5271 123.036 108 1.03666 0.512897
172 16 5318 5302 123.041 124 0.104699 0.512502
173 16 5349 5333 123.047 124 0.263161 0.512221
174 16 5379 5363 123.012 120 0.210632 0.512855
175 16 5410 5394 123.017 124 0.289337 0.512695
176 15 5442 5427 123.066 132 0.353725 0.512443
177 16 5470 5454 122.981 108 0.215974 0.512398
178 15 5502 5487 123.031 132 1.60185 0.513026
179 16 5531 5515 122.969 112 1.55787 0.512736
2020-06-08 20:26:36.202581 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.512982
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 5562 5546 122.975 124 0.0278812 0.512982
181 15 5592 5577 122.979 124 0.153238 0.51253
182 16 5626 5610 123.028 132 1.34885 0.512791
183 16 5656 5640 123.012 120 0.446493 0.512759
184 16 5683 5667 122.914 108 0.276121 0.513173
185 16 5712 5696 122.876 116 0.11302 0.513048
186 16 5744 5728 122.903 128 0.163667 0.512958
187 16 5773 5757 122.865 116 0.246265 0.513342
188 16 5802 5786 122.827 116 1.2992 0.513934
189 16 5828 5812 122.727 104 0.411384 0.513875
190 16 5858 5842 122.712 120 0.226328 0.513802
191 16 5891 5875 122.76 132 1.93024 0.514104
192 15 5918 5903 122.686 112 0.334509 0.514099
193 16 5955 5939 122.796 144 0.232033 0.513861
194 16 5991 5975 122.905 144 1.82599 0.513048
195 16 6023 6007 122.931 128 0.185962 0.513075
196 16 6057 6041 122.998 136 0.388207 0.512848
197 16 6089 6073 123.004 128 1.9637 0.512527
198 15 6118 6103 122.987 120 2.11486 0.513053
199 16 6147 6131 122.932 112 0.319751 0.513075
2020-06-08 20:26:56.300622 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.513492
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 6179 6163 122.957 128 0.452565 0.513492
201 16 6213 6197 123.022 136 0.623237 0.512988
202 16 6242 6226 122.987 116 0.22832 0.512766
203 15 6270 6255 122.953 116 1.76105 0.513175
204 16 6301 6285 122.938 120 0.0488869 0.513091
205 16 6333 6317 122.962 128 0.176727 0.513104
206 16 6365 6349 122.986 128 0.474029 0.51275
207 16 6396 6380 122.99 124 0.358129 0.513158
208 16 6429 6413 123.033 132 0.165151 0.512857
209 16 6459 6443 123.019 120 0.227616 0.512817
210 16 6490 6474 123.023 124 0.193078 0.513148
211 16 6517 6501 122.951 108 0.295597 0.512844
212 16 6545 6529 122.9 112 0.0520579 0.513129
213 15 6577 6562 122.942 132 0.0428166 0.513279
214 16 6609 6593 122.947 124 0.0898623 0.513379
215 16 6638 6622 122.914 116 0.351055 0.513152
216 16 6664 6648 122.827 104 0.457336 0.513379
217 16 6694 6678 122.814 120 0.757288 0.514233
218 16 6723 6707 122.783 116 0.0415485 0.514156
219 16 6747 6731 122.66 96 0.079386 0.514509
2020-06-08 20:27:16.311787 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.515177
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 6775 6759 122.61 112 0.460664 0.515177
221 16 6802 6786 122.544 108 0.0552546 0.515491
222 16 6834 6818 122.568 128 0.0598628 0.515325
223 16 6858 6842 122.449 96 0.663182 0.515606
224 16 6890 6874 122.46 128 0.769975 0.515737
225 16 6920 6904 122.448 120 0.752443 0.515702
226 16 6948 6932 122.402 112 0.597319 0.515866
227 16 6980 6964 122.426 128 0.0529803 0.515565
228 16 7011 6995 122.432 124 0.0843282 0.515687
229 15 7044 7029 122.486 136 0.0415242 0.515231
230 16 7070 7054 122.388 100 0.300856 0.515405
231 16 7103 7087 122.43 132 0.275401 0.515418
232 16 7132 7116 122.402 116 0.426996 0.515655
233 16 7165 7149 122.443 132 0.0698307 0.515385
234 16 7195 7179 122.433 120 1.3029 0.515821
235 16 7225 7209 122.422 120 0.736337 0.515593
236 16 7256 7240 122.422 124 0.0908626 0.515584
237 16 7286 7270 122.396 120 0.250061 0.515574
238 16 7315 7299 122.369 116 0.0436245 0.515432
239 16 7347 7331 122.391 128 0.479437 0.515564
2020-06-08 20:27:36.400361 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.515349
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 7379 7363 122.414 128 0.073707 0.515349
241 16 7411 7395 122.437 128 0.176264 0.515559
242 16 7437 7421 122.361 104 1.60691 0.515857
243 16 7466 7450 122.335 116 0.392598 0.515885
244 16 7501 7485 122.407 140 0.126986 0.515987
245 16 7532 7516 122.413 124 0.260484 0.515732
246 16 7564 7548 122.436 128 1.38074 0.515799
247 16 7591 7575 122.377 108 0.390964 0.515951
248 16 7620 7604 122.352 116 0.232344 0.515948
249 16 7650 7634 122.342 120 1.2388 0.516239
250 15 7681 7666 122.365 128 0.130807 0.516233
251 16 7718 7702 122.451 144 0.039761 0.515476
252 16 7747 7731 122.425 116 0.286396 0.515473
253 16 7777 7761 122.415 120 1.76953 0.515806
254 16 7811 7795 122.468 136 0.156431 0.515406
255 16 7839 7823 122.427 112 1.6787 0.515651
256 16 7867 7851 122.387 112 0.26108 0.515656
257 15 7895 7880 122.361 116 0.229928 0.516206
258 15 7924 7909 122.336 116 0.0847597 0.516009
259 16 7954 7938 122.311 116 0.410862 0.516013
2020-06-08 20:27:56.409067 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.515997
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 15 7981 7966 122.271 112 0.406446 0.515997
261 16 8019 8003 122.368 148 0.190577 0.516212
262 16 8047 8031 122.329 112 0.260786 0.516008
263 16 8077 8061 122.32 120 1.77423 0.516061
264 16 8108 8092 122.326 124 0.320483 0.516016
265 16 8137 8121 122.302 116 0.480798 0.516348
266 16 8165 8149 122.264 112 0.0402743 0.516797
267 16 8191 8175 122.195 104 0.383156 0.51675
268 15 8219 8204 122.171 116 0.254408 0.517035
269 16 8250 8234 122.163 120 0.100691 0.516921
270 16 8280 8264 122.155 120 0.0998158 0.516964
271 16 8313 8297 122.191 132 0.551078 0.516838
272 16 8341 8325 122.153 112 1.91776 0.516964
273 16 8371 8355 122.145 120 0.702243 0.517167
274 16 8403 8387 122.167 128 0.0521393 0.51697
275 16 8431 8415 122.13 112 0.170423 0.516868
276 16 8461 8445 122.122 120 0.301784 0.51688
277 16 8489 8473 122.085 112 0.0697051 0.516845
278 16 8519 8503 122.078 120 0.22144 0.517237
279 16 8549 8533 122.07 120 1.50557 0.517312
2020-06-08 20:28:16.417187 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.51703
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 8584 8568 122.134 140 0.0833517 0.51703
281 16 8614 8598 122.127 120 0.268542 0.516842
282 16 8646 8630 122.147 128 1.61622 0.517074
283 16 8673 8657 122.097 108 0.306155 0.517013
284 15 8699 8684 122.047 108 0.187822 0.517219
285 16 8726 8710 121.983 104 1.71585 0.517776
286 16 8758 8742 121.996 128 1.38552 0.517483
287 16 8788 8772 121.989 120 0.584332 0.517755
288 16 8814 8798 121.926 104 0.28312 0.517649
289 16 8844 8828 121.918 120 1.8311 0.518047
290 16 8871 8855 121.87 108 0.271836 0.517879
291 16 8898 8882 121.822 108 0.237387 0.518344
292 16 8933 8917 121.884 140 0.19795 0.518179
293 16 8961 8945 121.849 112 0.0720802 0.518151
294 16 8992 8976 121.856 124 0.377498 0.518005
295 15 9023 9008 121.877 128 0.336213 0.518273
296 16 9055 9039 121.883 124 0.421593 0.518064
297 16 9088 9072 121.915 132 0.281163 0.517804
298 16 9121 9105 121.949 132 0.264226 0.517848
299 16 9155 9139 121.995 136 0.0452371 0.517625
2020-06-08 20:28:36.458626 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.517897
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 9183 9167 121.962 112 0.796513 0.517897
301 16 9212 9196 121.942 116 0.341818 0.517728
302 16 9247 9231 122.002 140 0.0415116 0.517609
303 16 9274 9258 121.956 108 0.28618 0.517839
304 16 9304 9288 121.949 120 0.197714 0.517763
305 16 9335 9319 121.956 124 0.0481885 0.517615
306 16 9367 9351 121.972 128 0.461866 0.517715
307 16 9398 9382 121.978 124 1.74955 0.517827
308 16 9427 9411 121.959 116 0.603792 0.517872
309 16 9458 9442 121.965 124 0.360782 0.517917
310 16 9494 9478 122.036 144 0.0545681 0.51755
311 15 9531 9516 122.131 152 1.58507 0.517139
312 16 9563 9547 122.137 124 0.243046 0.517091
313 16 9593 9577 122.13 120 0.328936 0.51697
314 16 9625 9609 122.148 128 2.09227 0.517021
315 16 9656 9640 122.154 124 0.232122 0.517195
316 16 9686 9670 122.147 120 1.36692 0.517148
317 16 9712 9696 122.09 104 1.64755 0.517278
318 16 9744 9728 122.109 128 0.224697 0.517303
319 16 9776 9760 122.127 128 0.20737 0.517126
2020-06-08 20:28:56.498477 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.517035
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 15 9807 9792 122.136 128 0.236348 0.517035
321 16 9835 9819 122.092 108 1.66787 0.51705
322 16 9861 9845 122.036 104 0.236375 0.517296
323 16 9895 9879 122.079 136 0.555548 0.517181
324 16 9922 9906 122.036 108 1.50661 0.517683
325 16 9946 9930 121.956 96 0.427905 0.517881
326 16 9975 9959 121.938 116 0.56027 0.518102
327 16 10005 9989 121.932 120 0.304654 0.51816
328 16 10035 10019 121.926 120 0.608291 0.518218
329 16 10065 10049 121.92 120 0.256241 0.518068
330 16 10090 10074 121.853 100 0.339154 0.518257
331 16 10121 10105 121.86 124 0.26115 0.518408
332 16 10150 10134 121.842 116 0.569751 0.518405
333 16 10180 10164 121.836 120 0.831929 0.518518
334 16 10213 10197 121.867 132 0.267892 0.518477
335 16 10243 10227 121.861 120 0.258912 0.51845
336 16 10275 10259 121.877 128 0.299752 0.518122
337 16 10305 10289 121.871 120 0.291985 0.518174
338 16 10337 10321 121.889 128 0.0579847 0.51824
339 16 10366 10350 121.859 116 0.202518 0.518286
2020-06-08 20:29:16.545616 min lat: 0.0227802 max lat: 2.94952 avg lat: 0.518194
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 10396 10380 121.853 120 0.229475 0.518194
341 16 10425 10409 121.836 116 0.0529861 0.518389
342 15 10459 10444 121.889 140 0.316579 0.518168
343 16 10492 10476 121.906 128 0.0581969 0.518135
344 16 10523 10507 121.912 124 0.244396 0.518129
345 16 10551 10535 121.884 112 1.77602 0.518198
346 16 10580 10564 121.867 116 0.601367 0.518402
347 16 10611 10595 121.873 124 0.414638 0.518427
348 16 10644 10628 121.902 132 0.0398358 0.518336
349 16 10674 10658 121.896 120 0.287942 0.51827
350 16 10704 10688 121.891 120 0.0742775 0.518127
351 16 10732 10716 121.862 112 0.0462632 0.51825
352 16 10759 10743 121.815 108 0.437449 0.518427
353 16 10793 10777 121.855 136 1.12846 0.518502
354 16 10828 10812 121.906 140 1.18425 0.518289
355 16 10853 10837 121.845 100 0.0665273 0.518143
356 16 10884 10868 121.842 124 0.323274 0.518262
357 16 10909 10893 121.781 100 0.719393 0.518434
358 15 10935 10920 121.741 108 0.59949 0.518959
359 16 10965 10949 121.725 116 0.789253 0.519
2020-06-08 20:29:36.601742 min lat: 0.0227802 max lat: 2.94952 avg lat: 0.518776
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 10999 10983 121.765 136 0.246673 0.518776
361 16 11032 11016 121.793 132 0.274904 0.518609
362 15 11055 11040 121.722 96 0.0521564 0.518555
Total time run: 363.124
Total reads made: 11055
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 121.777
Average IOPS: 30
Stddev IOPS: 2.75726
Max IOPS: 39
Min IOPS: 24
Average Latency(s): 0.519379
Max latency(s): 2.94952
Min latency(s): 0.0227471
Total time run: 363.124
Total reads made: 11055
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 121.777
Average IOPS: 30
Stddev IOPS: 2.75726
Max IOPS: 39
Min IOPS: 24
Average Latency(s): 0.519379
Max latency(s): 2.94952
Min latency(s): 0.0227471
Random Read Test, pool with 256 PGs
podman exec ceph-mon-f5-422-01 rados bench -p testbench2 600 rand | tee --append $(date +%F)-rados-test-read-rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 51 35 139.943 140 0.0586091 0.277962
2 16 81 65 129.966 120 0.066807 0.375681
3 16 113 97 129.305 128 0.268188 0.402898
4 16 153 137 136.964 160 1.44037 0.407198
5 16 182 166 132.741 116 0.0480781 0.419728
6 16 210 194 129.142 112 0.0466207 0.451808
7 16 244 228 130.118 136 0.829422 0.455489
8 16 271 255 127.354 108 0.0613913 0.467553
9 16 305 289 128.311 136 0.516398 0.463554
10 16 335 319 127.479 120 0.355922 0.466055
11 16 366 350 127.161 124 0.293341 0.473895
12 15 396 381 126.895 124 0.580237 0.48482
13 16 427 411 126.34 120 0.570833 0.483577
14 16 461 445 127.028 136 1.52073 0.482284
15 16 489 473 126.026 112 0.536009 0.483434
16 15 521 506 126.398 132 0.776196 0.486821
17 16 552 536 126.02 120 0.085827 0.486352
18 16 581 565 125.456 116 0.256079 0.488214
19 15 611 596 125.373 124 0.701293 0.495113
2020-06-08 20:37:28.247285 min lat: 0.00767023 max lat: 2.2389 avg lat: 0.495997
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 640 624 124.703 112 1.56505 0.495997
21 16 674 658 125.24 136 0.425942 0.495574
22 16 707 691 125.546 132 1.47021 0.495525
23 15 734 719 124.903 112 0.314597 0.497891
24 16 766 750 124.865 124 1.09931 0.49645
25 16 796 780 124.669 120 0.273213 0.495906
26 16 825 809 124.335 116 0.264268 0.496062
27 16 858 842 124.618 132 0.444402 0.495716
28 16 894 878 125.304 144 0.280718 0.495399
29 15 924 909 125.258 124 0.407136 0.495202
30 16 956 940 125.216 124 1.20638 0.496961
31 16 988 972 125.305 128 0.582289 0.49698
32 16 1020 1004 125.388 128 0.459654 0.49613
33 16 1050 1034 125.225 120 0.35612 0.496381
34 16 1081 1065 125.188 124 0.305698 0.495892
35 16 1111 1095 124.907 120 0.362349 0.495583
36 16 1141 1125 124.771 120 0.279651 0.497179
37 16 1170 1154 124.534 116 0.269168 0.500527
38 16 1204 1188 124.823 136 0.368712 0.498541
39 16 1239 1223 125.211 140 0.324084 0.498661
2020-06-08 20:37:48.302970 min lat: 0.0075863 max lat: 2.44289 avg lat: 0.499018
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1266 1250 124.778 108 0.425744 0.499018
41 15 1298 1283 124.953 132 0.0877296 0.502065
42 16 1326 1310 124.547 108 0.436453 0.501906
43 16 1355 1339 124.345 116 0.0865259 0.503146
44 15 1388 1373 124.589 136 0.282347 0.502295
45 16 1420 1404 124.575 124 2.15099 0.501313
46 16 1449 1433 124.389 116 0.184875 0.502487
47 16 1478 1462 124.21 116 0.275427 0.501894
48 16 1509 1493 124.206 124 0.0566634 0.503259
49 16 1540 1524 124.201 124 1.95947 0.50452
50 16 1571 1555 124.197 124 0.186263 0.50492
51 16 1602 1586 124.192 124 0.599994 0.504414
52 16 1633 1617 124.188 124 0.25071 0.503623
53 16 1672 1656 124.787 156 0.100314 0.502715
54 16 1698 1682 124.403 104 0.0575364 0.505972
55 15 1727 1712 124.323 120 0.776987 0.506102
56 16 1763 1747 124.602 140 1.28794 0.504847
57 16 1792 1776 124.451 116 0.0664667 0.504545
58 16 1819 1803 124.167 108 1.718 0.505636
59 16 1847 1831 123.961 112 0.390346 0.506569
2020-06-08 20:38:08.314952 min lat: 0.0075863 max lat: 2.66352 avg lat: 0.506042
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 15 1881 1866 124.228 140 0.0079828 0.506042
61 16 1915 1899 124.355 132 0.159932 0.505287
62 15 1945 1930 124.349 124 0.213707 0.504482
63 16 1977 1961 124.342 124 0.0752758 0.505421
64 16 2014 1998 124.711 148 0.0680605 0.504238
65 16 2041 2025 124.454 108 0.27941 0.503864
66 15 2070 2055 124.386 120 0.551604 0.504859
67 16 2103 2087 124.44 128 1.41295 0.506116
68 16 2131 2115 124.257 112 0.790664 0.506181
69 16 2161 2145 124.192 120 0.293269 0.505521
70 15 2190 2175 124.132 120 0.0441665 0.506671
71 16 2219 2203 123.961 112 0.454395 0.507879
72 16 2244 2228 123.628 100 0.56955 0.509076
73 16 2277 2261 123.742 132 0.0787766 0.50831
74 16 2304 2288 123.53 108 0.0784757 0.508522
75 16 2342 2326 123.909 152 0.0994228 0.508304
76 16 2373 2357 123.91 124 0.206 0.50693
77 16 2400 2384 123.703 108 0.499882 0.506759
78 15 2427 2412 123.553 112 0.180056 0.509061
79 16 2462 2446 123.71 136 0.0776175 0.50797
2020-06-08 20:38:28.320191 min lat: 0.0075863 max lat: 2.66352 avg lat: 0.50781
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 2495 2479 123.813 132 1.9827 0.50781
81 16 2528 2512 123.914 132 0.0470868 0.507869
82 16 2554 2538 123.671 104 1.09973 0.509034
83 16 2586 2570 123.721 128 0.471985 0.508107
84 16 2616 2600 123.676 120 2.2733 0.508633
85 16 2647 2631 123.68 124 0.0991686 0.508681
86 16 2679 2663 123.73 128 0.0486071 0.508748
87 16 2707 2691 123.595 112 0.0563499 0.508083
88 16 2740 2724 123.69 132 0.329987 0.509494
89 16 2770 2754 123.648 120 0.191511 0.508935
90 16 2800 2784 123.608 120 1.20885 0.509202
91 16 2832 2816 123.656 128 0.239141 0.508898
92 16 2864 2848 123.703 128 0.0446796 0.50945
93 16 2898 2882 123.835 136 0.0763327 0.50761
94 16 2930 2914 123.879 128 0.510066 0.508343
95 16 2964 2948 124.006 136 0.210589 0.507625
96 16 2991 2975 123.839 108 1.18171 0.508812
97 16 3020 3004 123.758 116 0.333442 0.508956
98 16 3054 3038 123.883 136 0.0905627 0.508407
99 16 3084 3068 123.843 120 0.379993 0.5087
2020-06-08 20:38:48.325116 min lat: 0.0075863 max lat: 2.66352 avg lat: 0.508697
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3118 3102 123.964 136 0.23994 0.508697
101 16 3149 3133 123.964 124 0.21101 0.508078
102 16 3180 3164 123.965 124 0.37649 0.509156
103 15 3207 3192 123.848 112 0.217731 0.50909
104 16 3239 3223 123.85 124 0.0523112 0.508872
105 16 3274 3258 124.003 140 0.0490936 0.507924
106 16 3302 3286 123.89 112 0.407885 0.508503
107 15 3331 3316 123.853 120 0.286567 0.508713
108 16 3364 3348 123.892 128 1.77374 0.509035
109 16 3392 3376 123.782 112 0.0684978 0.509142
110 16 3424 3408 123.82 128 0.0692926 0.509873
111 16 3455 3439 123.822 124 1.54867 0.508961
112 16 3488 3472 123.895 132 0.715173 0.508953
113 16 3519 3503 123.895 124 0.642609 0.508741
114 16 3553 3537 124.001 136 0.514705 0.508665
115 16 3586 3570 124.069 132 0.226899 0.508361
116 16 3613 3597 123.931 108 0.228104 0.50851
117 16 3646 3630 123.999 132 1.63747 0.508497
118 16 3675 3659 123.93 116 0.271013 0.508356
119 16 3705 3689 123.897 120 0.0547901 0.507972
2020-06-08 20:39:08.331750 min lat: 0.0075863 max lat: 2.66352 avg lat: 0.508951
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 15 3739 3724 124.03 140 0.232311 0.508951
121 16 3770 3754 123.997 120 1.67157 0.508966
122 16 3796 3780 123.833 104 0.20622 0.508606
123 16 3825 3809 123.769 116 2.27719 0.509577
124 16 3859 3843 123.867 136 0.0685277 0.509121
125 15 3888 3873 123.835 120 0.272679 0.509094
126 16 3919 3903 123.805 120 1.47513 0.509965
127 15 3949 3934 123.806 124 0.00774259 0.509008
128 16 3979 3963 123.745 116 0.0644373 0.509208
129 16 4013 3997 123.839 136 0.0466334 0.509202
130 16 4042 4026 123.779 116 0.311963 0.5092
131 15 4073 4058 123.811 128 1.58711 0.509654
132 16 4104 4088 123.782 120 0.244707 0.509497
133 16 4131 4115 123.663 108 0.0986581 0.510015
134 16 4161 4145 123.636 120 0.324041 0.510727
135 16 4194 4178 123.697 132 0.324672 0.510124
136 15 4223 4208 123.67 120 2.0338 0.51089
137 16 4256 4240 123.702 128 0.0555726 0.510462
138 16 4287 4271 123.704 124 0.383916 0.510635
139 16 4319 4303 123.734 128 0.284673 0.510007
2020-06-08 20:39:28.336477 min lat: 0.0075863 max lat: 2.66352 avg lat: 0.510037
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 4350 4334 123.736 124 0.375594 0.510037
141 16 4380 4364 123.709 120 0.175716 0.509952
142 16 4410 4394 123.683 120 0.0568104 0.510282
143 16 4444 4428 123.769 136 0.0601803 0.509923
144 16 4475 4459 123.771 124 0.305176 0.509981
145 16 4506 4490 123.772 124 0.28875 0.509653
146 16 4539 4523 123.828 132 1.11541 0.510107
147 15 4573 4558 123.938 140 0.318495 0.509141
148 15 4607 4592 124.019 136 1.52266 0.509268
149 16 4639 4623 124.019 124 1.42369 0.509024
150 16 4667 4651 123.939 112 0.157892 0.508891
151 16 4698 4682 123.939 124 0.317812 0.5092
152 15 4727 4712 123.913 120 1.73716 0.509819
153 16 4757 4741 123.861 116 0.399952 0.509763
154 16 4794 4778 124.018 148 0.405619 0.509288
155 16 4825 4809 124.018 124 0.699032 0.509096
156 16 4859 4843 124.094 136 0.0835734 0.508968
157 16 4887 4871 124.017 112 0.0672621 0.508836
158 16 4918 4902 124.017 124 0.245538 0.509377
159 16 4958 4942 124.243 160 0.104857 0.508502
2020-06-08 20:39:48.339445 min lat: 0.0075863 max lat: 2.66352 avg lat: 0.508473
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 4986 4970 124.166 112 0.207302 0.508473
161 16 5015 4999 124.116 116 0.0557762 0.508803
162 16 5048 5032 124.164 132 0.0650655 0.508544
163 15 5074 5059 124.065 108 0.211816 0.508679
164 16 5107 5091 124.089 128 0.0534329 0.509043
165 16 5145 5129 124.258 152 0.206019 0.508435
166 16 5174 5158 124.208 116 0.0635734 0.508493
167 16 5202 5186 124.135 112 0.321672 0.508552
168 16 5232 5216 124.11 120 1.85804 0.508933
169 16 5263 5247 124.109 124 0.0518495 0.508773
170 16 5294 5278 124.109 124 0.400471 0.508858
171 16 5324 5308 124.084 120 0.590401 0.509171
172 16 5352 5336 124.014 112 0.389228 0.509427
173 16 5381 5365 123.968 116 0.226304 0.509386
174 16 5417 5401 124.083 144 0.228978 0.509089
175 16 5445 5429 124.014 112 0.58449 0.509205
176 16 5476 5460 124.013 124 1.33933 0.509297
177 16 5505 5489 123.968 116 0.0111495 0.509446
178 16 5535 5519 123.946 120 0.30085 0.509571
179 15 5564 5549 123.923 120 0.185957 0.509627
2020-06-08 20:40:08.342729 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.509528
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 5597 5581 123.946 128 0.275075 0.509528
181 15 5629 5614 123.99 132 0.398652 0.509065
182 16 5662 5646 124.012 128 0.279213 0.509617
183 16 5695 5679 124.056 132 0.315781 0.509399
184 16 5728 5712 124.099 132 0.0618818 0.5089
185 16 5757 5741 124.055 116 0.170632 0.509219
186 16 5789 5773 124.075 128 0.052908 0.50922
187 16 5822 5806 124.117 132 0.0700102 0.509207
188 16 5852 5836 124.095 120 1.34414 0.508999
189 16 5882 5866 124.073 120 0.340822 0.508673
190 15 5916 5901 124.156 140 0.587977 0.508722
191 16 5946 5930 124.113 116 0.173819 0.50917
192 16 5983 5967 124.238 148 0.408126 0.508085
193 16 6017 6001 124.298 136 2.29032 0.507992
194 15 6046 6031 124.276 120 1.46933 0.50861
195 16 6079 6063 124.295 128 0.335119 0.50832
196 16 6106 6090 124.211 108 1.4968 0.508679
197 16 6136 6120 124.19 120 0.246017 0.508745
198 16 6168 6152 124.209 128 0.0817655 0.508565
199 16 6195 6179 124.128 108 1.5363 0.509042
2020-06-08 20:40:28.350671 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.508874
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 6225 6209 124.106 120 0.222804 0.508874
201 16 6257 6241 124.123 128 0.53307 0.509389
202 16 6284 6268 124.043 108 0.374498 0.509456
203 16 6314 6298 124.023 120 0.0111701 0.509575
204 16 6341 6325 123.945 108 0.265543 0.509497
205 16 6373 6357 123.965 128 0.0546058 0.509751
206 15 6404 6389 123.984 128 0.394781 0.509861
207 15 6432 6417 123.926 112 1.78949 0.510346
208 16 6468 6452 124.003 140 0.193595 0.50978
209 16 6500 6484 124.022 128 0.289926 0.509269
210 16 6532 6516 124.04 128 2.12999 0.509346
211 16 6560 6544 123.983 112 0.160417 0.509588
212 16 6587 6571 123.907 108 0.189348 0.509916
213 16 6620 6604 123.945 132 1.54803 0.509682
214 16 6651 6635 123.945 124 0.267665 0.509534
215 16 6682 6666 123.946 124 0.0804845 0.509949
216 16 6712 6696 123.927 120 1.4027 0.50989
217 16 6744 6728 123.946 128 0.207887 0.509583
218 16 6776 6760 123.963 128 1.45251 0.509927
219 16 6808 6792 123.981 128 0.0771457 0.509744
2020-06-08 20:40:48.361940 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.509848
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 6841 6825 124.018 132 1.40176 0.509848
221 16 6871 6855 123.999 120 0.153146 0.509766
222 15 6905 6890 124.071 140 0.236409 0.509499
223 16 6937 6921 124.07 124 2.17101 0.509445
224 16 6972 6956 124.141 140 1.75282 0.509013
225 16 7001 6985 124.105 116 0.051531 0.509253
226 15 7030 7015 124.087 120 0.420543 0.509651
227 16 7065 7049 124.139 136 0.423939 0.509544
228 16 7102 7086 124.244 148 0.353998 0.508848
229 16 7131 7115 124.206 116 0.301212 0.508678
230 16 7160 7144 124.17 116 0.0625969 0.508972
231 16 7191 7175 124.153 124 0.332617 0.509179
232 15 7223 7208 124.187 132 1.13588 0.509092
233 16 7252 7236 124.134 112 0.174538 0.508877
234 16 7281 7265 124.099 116 0.213757 0.50902
235 16 7310 7294 124.065 116 1.64307 0.509362
236 16 7334 7318 123.946 96 0.339064 0.50974
237 16 7366 7350 123.963 128 0.167864 0.510063
238 16 7396 7380 123.946 120 0.238383 0.509722
239 16 7424 7408 123.896 112 0.370573 0.510142
2020-06-08 20:41:08.400492 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510074
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 15 7456 7441 123.93 132 0.085039 0.510074
241 16 7491 7475 123.98 136 0.0534243 0.509853
242 16 7526 7510 124.046 140 1.07023 0.509888
243 16 7554 7538 123.996 112 0.307925 0.50968
244 16 7583 7567 123.963 116 0.373594 0.509595
245 16 7611 7595 123.914 112 0.363044 0.510341
246 16 7639 7623 123.866 112 1.42921 0.510401
247 16 7671 7655 123.882 128 0.485589 0.510211
248 16 7701 7685 123.866 120 0.168639 0.510337
249 16 7727 7711 123.787 104 1.37117 0.510641
250 16 7759 7743 123.803 128 0.248629 0.510484
251 16 7791 7775 123.82 128 1.42661 0.510676
252 16 7827 7811 123.9 144 0.502747 0.510141
253 15 7854 7839 123.853 112 0.0930292 0.510181
254 16 7885 7869 123.838 120 0.339748 0.510154
255 16 7918 7902 123.87 132 0.20261 0.510449
256 16 7948 7932 123.854 120 0.0505014 0.510476
257 16 7978 7962 123.839 120 0.29542 0.510559
258 16 8005 7989 123.778 108 1.5502 0.510944
259 16 8036 8020 123.779 124 0.0426524 0.51089
2020-06-08 20:41:28.407660 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510816
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 15 8068 8053 123.809 132 1.58499 0.510816
261 16 8098 8082 123.779 116 0.302931 0.510664
262 16 8128 8112 123.764 120 0.460322 0.510907
263 16 8158 8142 123.75 120 0.297061 0.5111
264 16 8187 8171 123.72 116 0.0733328 0.511164
265 16 8225 8209 123.827 152 0.0473569 0.510416
266 16 8252 8236 123.767 108 2.20866 0.510748
267 16 8286 8270 123.813 136 0.0572092 0.510573
268 16 8315 8299 123.784 116 1.75067 0.510689
269 16 8345 8329 123.769 120 1.65631 0.510747
270 16 8382 8366 123.858 148 0.0979424 0.510329
271 16 8411 8395 123.829 116 0.0590657 0.510407
272 16 8437 8421 123.757 104 0.275211 0.510897
273 15 8467 8452 123.757 124 0.297085 0.510831
274 16 8499 8483 123.757 124 0.0824411 0.510844
275 16 8531 8515 123.772 128 0.0639115 0.510776
276 16 8563 8547 123.787 128 0.010436 0.510743
277 16 8591 8575 123.745 112 0.232358 0.51088
278 16 8623 8607 123.76 128 0.427809 0.510844
279 16 8659 8643 123.832 144 0.103848 0.510587
2020-06-08 20:41:48.418451 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.51031
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 8695 8679 123.903 144 0.139037 0.51031
281 16 8730 8714 123.96 140 0.0542311 0.510136
282 16 8761 8745 123.96 124 0.511144 0.510336
283 16 8791 8775 123.946 120 0.0513455 0.510252
284 16 8823 8807 123.96 128 0.221567 0.510231
285 16 8852 8836 123.932 116 0.0483402 0.510023
286 16 8877 8861 123.849 100 1.614 0.510573
287 15 8910 8895 123.891 136 0.510751 0.51047
288 16 8940 8924 123.864 116 1.17024 0.510512
289 16 8966 8950 123.795 104 1.43761 0.510837
290 16 8996 8980 123.782 120 0.20469 0.510744
291 15 9025 9010 123.769 120 1.318 0.510886
292 16 9057 9041 123.769 124 0.276344 0.510791
293 16 9085 9069 123.729 112 0.343372 0.510957
294 15 9115 9100 123.729 124 0.235895 0.511052
295 16 9145 9129 123.703 116 0.0550538 0.51104
296 16 9176 9160 123.704 124 0.292128 0.511164
297 15 9204 9189 123.678 116 1.50093 0.511552
298 16 9233 9217 123.639 112 1.03965 0.511498
299 16 9265 9249 123.653 128 0.126052 0.511238
2020-06-08 20:42:08.424242 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.511379
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 9294 9278 123.627 116 0.220644 0.511379
301 16 9325 9309 123.628 124 0.291652 0.511307
302 16 9355 9339 123.616 120 0.0227457 0.5116
303 16 9387 9371 123.631 128 0.628228 0.51152
304 16 9414 9398 123.579 108 0.052724 0.511771
305 16 9448 9432 123.62 136 0.404111 0.511363
306 15 9481 9466 123.66 136 1.4108 0.511454
307 16 9512 9496 123.648 120 0.232413 0.511393
308 15 9547 9532 123.713 144 0.274889 0.511283
309 16 9580 9564 123.726 128 0.0500387 0.510885
310 15 9612 9597 123.753 132 0.273428 0.51064
311 16 9648 9632 123.805 140 0.0607635 0.510788
312 16 9678 9662 123.793 120 0.0657142 0.51075
313 15 9711 9696 123.831 136 0.29422 0.510858
314 16 9740 9724 123.793 112 0.167771 0.510908
315 16 9772 9756 123.807 128 1.47696 0.510824
316 16 9801 9785 123.782 116 0.262162 0.510938
317 16 9832 9816 123.783 124 0.262578 0.510694
318 16 9862 9846 123.771 120 1.92793 0.511048
319 16 9893 9877 123.771 124 0.0111321 0.510698
2020-06-08 20:42:28.433528 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510866
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 9928 9912 123.822 140 0.0747197 0.510866
321 16 9959 9943 123.822 124 0.0622337 0.510632
322 16 9990 9974 123.823 124 1.15184 0.510831
323 16 10018 10002 123.786 112 0.283216 0.510686
324 15 10052 10037 123.836 140 0.0570723 0.510882
325 16 10084 10068 123.837 124 0.180581 0.510689
326 16 10114 10098 123.825 120 1.57316 0.510687
327 16 10142 10126 123.787 112 0.0700195 0.510686
328 16 10175 10159 123.812 132 0.078846 0.510876
329 16 10211 10195 123.874 144 1.66505 0.510516
330 15 10238 10223 123.838 112 0.237626 0.510647
331 16 10267 10251 123.802 112 0.242313 0.510869
332 16 10299 10283 123.814 128 1.59151 0.511055
333 16 10333 10317 123.851 136 0.307842 0.51065
334 16 10365 10349 123.863 128 1.69979 0.510759
335 16 10396 10380 123.864 124 1.42013 0.51076
336 16 10426 10410 123.852 120 0.291663 0.510681
337 16 10456 10440 123.841 120 0.305409 0.510878
338 16 10485 10469 123.817 116 0.799237 0.510997
339 16 10519 10503 123.852 136 0.054971 0.511053
2020-06-08 20:42:48.443454 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510834
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 10552 10536 123.876 132 0.312648 0.510834
341 16 10581 10565 123.853 116 0.342335 0.510661
342 16 10614 10598 123.876 132 0.0851834 0.510513
343 15 10644 10629 123.877 124 0.425976 0.510556
344 16 10677 10661 123.889 128 0.333656 0.510431
345 16 10710 10694 123.912 132 0.237909 0.51058
346 16 10738 10722 123.878 112 0.267664 0.510516
347 16 10766 10750 123.843 112 0.0632882 0.51055
348 15 10793 10778 123.809 112 0.721677 0.510889
349 16 10827 10811 123.833 132 0.0511126 0.510809
350 16 10860 10844 123.856 132 0.0960083 0.510583
351 16 10891 10875 123.856 124 0.272481 0.510424
352 15 10923 10908 123.879 132 0.523735 0.510421
353 16 10955 10939 123.88 124 0.203108 0.510501
354 16 10987 10971 123.891 128 0.345238 0.510592
355 16 11020 11004 123.914 132 0.307542 0.510327
356 16 11051 11035 123.914 124 0.0802598 0.510384
357 16 11080 11064 123.892 116 1.3456 0.510543
358 16 11109 11093 123.87 116 0.518924 0.510555
359 16 11144 11128 123.915 140 0.726929 0.510407
2020-06-08 20:43:08.446750 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510504
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 11175 11159 123.915 124 0.544431 0.510504
361 16 11204 11188 123.893 116 0.823161 0.510755
362 16 11232 11216 123.86 112 0.0609196 0.510956
363 16 11263 11247 123.86 124 1.03556 0.510885
364 16 11292 11276 123.839 116 0.323328 0.510859
365 16 11326 11310 123.872 136 0.0742619 0.510547
366 16 11362 11346 123.927 144 0.0560132 0.510614
367 16 11393 11377 123.927 124 0.294645 0.510476
368 16 11423 11407 123.916 120 0.431146 0.510378
369 16 11457 11441 123.948 136 0.544814 0.510194
370 16 11482 11466 123.884 100 2.14508 0.510682
371 16 11515 11499 123.905 132 0.647952 0.510804
372 16 11541 11525 123.851 104 0.977412 0.511017
373 16 11572 11556 123.851 124 0.0453287 0.511044
374 16 11602 11586 123.84 120 0.245995 0.510803
375 16 11633 11617 123.841 124 0.243285 0.510837
376 16 11660 11644 123.799 108 0.0433665 0.511015
377 15 11685 11670 123.746 104 1.48479 0.511142
378 16 11722 11706 123.799 144 0.0460282 0.511031
379 16 11757 11741 123.842 140 0.0804824 0.510634
2020-06-08 20:43:28.456555 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510991
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 11787 11771 123.832 120 0.272876 0.510991
381 15 11816 11801 123.822 120 0.395054 0.511066
382 16 11849 11833 123.833 128 0.0641325 0.510832
383 16 11881 11865 123.844 128 1.67749 0.510731
384 15 11912 11897 123.854 128 0.0629009 0.51084
385 16 11940 11924 123.813 108 0.192493 0.510849
386 16 11974 11958 123.845 136 0.0761082 0.510637
387 16 12004 11988 123.835 120 0.228055 0.510834
388 16 12032 12016 123.804 112 0.754459 0.511176
389 16 12063 12047 123.805 124 1.38822 0.511204
390 16 12093 12077 123.795 120 0.0464053 0.511294
391 15 12120 12105 123.765 112 0.827743 0.511319
392 16 12151 12135 123.755 120 0.632207 0.511411
393 16 12188 12172 123.817 148 0.204327 0.510969
394 16 12218 12202 123.807 120 1.64355 0.511059
395 16 12251 12235 123.826 132 0.328473 0.510752
396 16 12286 12270 123.866 140 0.525551 0.510663
397 16 12315 12299 123.847 116 1.74368 0.510904
398 16 12350 12334 123.887 140 0.215078 0.510593
399 16 12381 12365 123.887 124 0.34144 0.510754
2020-06-08 20:43:48.465409 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510466
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 12415 12399 123.918 136 0.257447 0.510466
401 16 12446 12430 123.918 124 1.9718 0.510434
402 15 12474 12459 123.898 116 0.253661 0.510514
403 16 12504 12488 123.878 116 0.194725 0.510584
404 16 12530 12514 123.829 104 2.13664 0.510738
405 15 12557 12542 123.8 112 0.38411 0.510892
406 15 12591 12576 123.83 136 0.264805 0.510878
407 16 12617 12601 123.771 100 0.254247 0.511037
408 16 12646 12630 123.752 116 0.254397 0.511287
409 16 12674 12658 123.723 112 0.0555876 0.511232
410 16 12703 12687 123.704 116 0.191088 0.511346
411 16 12732 12716 123.686 116 0.594941 0.51133
412 16 12758 12742 123.638 104 0.297959 0.511442
413 16 12792 12776 123.668 136 0.292224 0.511596
414 16 12824 12808 123.678 128 0.213199 0.511473
415 16 12850 12834 123.631 104 0.214737 0.511679
416 16 12881 12865 123.632 124 0.204426 0.51165
417 16 12917 12901 123.68 144 0.128506 0.511486
418 15 12944 12929 123.652 112 0.0426699 0.511671
419 16 12974 12958 123.634 116 0.0489743 0.511653
2020-06-08 20:44:08.469140 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.511459
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 13007 12991 123.654 132 0.329233 0.511459
421 16 13035 13019 123.626 112 0.0821875 0.511665
422 16 13068 13052 123.646 132 0.45616 0.511607
423 16 13097 13081 123.628 116 0.0570792 0.511798
424 16 13129 13113 123.638 128 0.755786 0.511739
425 16 13158 13142 123.62 116 0.416511 0.511909
426 16 13198 13182 123.697 160 0.0623024 0.511689
427 16 13228 13212 123.689 120 0.42818 0.511579
428 15 13261 13246 123.716 136 1.13032 0.511563
429 16 13291 13275 123.698 116 0.303452 0.51161
430 16 13321 13305 123.69 120 0.042945 0.511613
431 16 13352 13336 123.69 124 1.67741 0.511533
432 15 13380 13365 123.673 116 0.344221 0.511665
433 16 13412 13396 123.673 124 0.347792 0.511613
434 16 13441 13425 123.655 116 0.045106 0.511641
435 16 13475 13459 123.684 136 0.0460147 0.51156
436 16 13503 13487 123.657 112 0.250473 0.511563
437 16 13536 13520 123.676 132 0.0093678 0.51146
438 16 13568 13552 123.686 128 0.196691 0.511626
439 16 13602 13586 123.714 136 0.145612 0.511335
2020-06-08 20:44:28.506419 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.511612
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 15 13629 13614 123.686 112 1.34441 0.511612
441 16 13661 13645 123.687 124 0.378196 0.511486
442 15 13692 13677 123.697 128 0.0570218 0.511661
443 16 13721 13705 123.67 112 1.17798 0.51175
444 16 13751 13735 123.662 120 0.792439 0.511746
445 16 13783 13767 123.671 128 0.0360382 0.51176
446 16 13806 13790 123.6 92 1.49323 0.511914
447 16 13838 13822 123.609 128 0.407204 0.511809
448 16 13863 13847 123.556 100 0.251138 0.511931
449 16 13892 13876 123.539 116 0.0844853 0.511986
450 15 13921 13906 123.531 120 0.352163 0.512236
451 16 13950 13934 123.505 112 0.0181103 0.512367
452 16 13979 13963 123.489 116 0.176074 0.512193
453 16 14009 13993 123.481 120 2.31226 0.512233
454 16 14040 14024 123.482 124 0.258509 0.512215
455 16 14070 14054 123.474 120 1.97507 0.512309
456 16 14099 14083 123.458 116 1.51515 0.512425
457 16 14126 14110 123.424 108 1.89717 0.512474
458 16 14161 14145 123.46 140 0.0106847 0.512576
459 16 14193 14177 123.47 128 0.0684864 0.512439
2020-06-08 20:44:48.518700 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.512432
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 14226 14210 123.488 132 1.73628 0.512432
461 16 14257 14241 123.489 124 1.28408 0.512418
462 15 14287 14272 123.49 124 0.233685 0.512345
463 16 14321 14305 123.508 132 1.66653 0.512237
464 16 14354 14338 123.527 132 1.36833 0.512191
465 16 14391 14375 123.579 148 0.0596241 0.512001
466 16 14421 14405 123.571 120 0.189901 0.511978
467 15 14448 14433 123.547 112 0.173545 0.511998
468 16 14477 14461 123.522 112 0.0812387 0.51206
469 16 14514 14498 123.574 148 0.141564 0.511996
470 16 14543 14527 123.558 116 0.0714603 0.512027
471 16 14571 14555 123.533 112 0.112975 0.512121
472 16 14597 14581 123.492 104 1.32184 0.512311
473 16 14627 14611 123.484 120 0.0777479 0.512374
474 16 14661 14645 123.511 136 1.45962 0.512317
475 16 14689 14673 123.487 112 0.167458 0.512192
476 16 14718 14702 123.471 116 2.28247 0.512498
477 15 14748 14733 123.472 124 0.283824 0.512558
478 15 14780 14765 123.481 128 0.329042 0.512501
479 16 14810 14794 123.466 116 1.24327 0.512461
2020-06-08 20:45:08.524332 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.512639
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 15 14837 14822 123.441 112 0.490039 0.512639
481 15 14866 14851 123.425 116 0.0658814 0.512787
482 16 14895 14879 123.402 112 0.397216 0.512757
483 16 14927 14911 123.411 128 0.299788 0.512714
484 15 14963 14948 123.462 148 0.20779 0.512583
485 16 15000 14984 123.504 144 0.244432 0.512344
486 16 15030 15014 123.497 120 1.49644 0.512378
487 16 15062 15046 123.506 128 0.0931012 0.512297
488 16 15092 15076 123.499 120 0.126937 0.512349
489 16 15125 15109 123.516 132 1.58117 0.512278
490 15 15152 15137 123.492 112 0.305903 0.512468
491 15 15185 15170 123.509 132 0.306541 0.512361
492 16 15220 15204 123.534 136 1.46515 0.51238
493 16 15250 15234 123.527 120 1.44872 0.512218
494 16 15277 15261 123.495 108 0.214639 0.5125
495 16 15312 15296 123.528 140 0.290033 0.512335
496 15 15339 15324 123.504 112 1.51602 0.512473
497 16 15368 15352 123.481 112 0.244055 0.512459
498 16 15397 15381 123.466 116 1.69339 0.512489
499 15 15428 15413 123.475 128 0.435628 0.512387
2020-06-08 20:45:28.538352 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.512573
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 15455 15439 123.436 104 2.12661 0.512573
501 16 15486 15470 123.437 124 0.074086 0.512603
502 16 15519 15503 123.454 132 0.379324 0.512493
503 16 15550 15534 123.455 124 1.79179 0.512662
504 16 15584 15568 123.479 136 0.550992 0.512538
505 16 15611 15595 123.448 108 0.423585 0.512483
506 16 15643 15627 123.457 128 0.266231 0.512626
507 15 15671 15656 123.442 116 0.236596 0.512636
508 16 15707 15691 123.475 140 0.277762 0.512599
509 15 15740 15725 123.498 136 0.0762671 0.51243
510 16 15773 15757 123.506 128 1.55159 0.512327
511 16 15802 15786 123.492 116 0.228607 0.512305
512 16 15835 15819 123.508 132 1.72701 0.512333
513 16 15864 15848 123.494 116 0.0790003 0.512417
514 16 15896 15880 123.502 128 0.483712 0.512423
515 16 15928 15912 123.511 128 1.58394 0.512415
516 16 15957 15941 123.496 116 0.198625 0.512344
517 16 15988 15972 123.497 124 1.78059 0.512348
518 16 16019 16003 123.498 124 0.0429903 0.512494
519 16 16052 16036 123.514 132 0.26881 0.51225
2020-06-08 20:45:48.556328 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.512329
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 16087 16071 123.546 140 1.25423 0.512329
521 15 16116 16101 123.539 120 0.155699 0.51224
522 15 16143 16128 123.509 108 0.91247 0.512328
523 16 16177 16161 123.525 132 1.61275 0.512304
524 16 16207 16191 123.519 120 1.59436 0.512343
525 16 16234 16218 123.489 108 0.285469 0.51244
526 16 16265 16249 123.49 124 0.0631698 0.512357
527 16 16296 16280 123.491 124 0.676844 0.512588
528 16 16329 16313 123.506 132 0.0671672 0.512477
529 16 16364 16348 123.537 140 0.486114 0.512255
530 16 16392 16376 123.515 112 0.0638656 0.512293
531 16 16420 16404 123.493 112 0.0655929 0.51254
532 16 16450 16434 123.486 120 0.803907 0.512571
533 16 16477 16461 123.457 108 0.0450396 0.512646
534 15 16503 16488 123.428 108 0.322278 0.512734
535 16 16534 16518 123.421 120 0.266941 0.512612
536 16 16566 16550 123.429 128 0.238308 0.512608
537 16 16594 16578 123.408 112 0.237731 0.512712
538 16 16621 16605 123.379 108 0.231232 0.512884
539 16 16649 16633 123.357 112 0.216255 0.512844
2020-06-08 20:46:08.575554 min lat: 0.00726443 max lat: 3.03968 avg lat: 0.512956
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 16678 16662 123.344 116 0.346213 0.512956
541 16 16707 16691 123.33 116 0.274218 0.513066
542 16 16741 16725 123.353 136 0.0876994 0.513019
543 16 16774 16758 123.369 132 1.21974 0.512968
544 16 16804 16788 123.363 120 0.405559 0.512912
545 16 16839 16823 123.394 140 0.0606176 0.512845
546 16 16870 16854 123.395 124 0.0749025 0.512826
547 16 16903 16887 123.406 132 1.70951 0.512732
548 16 16933 16917 123.399 120 0.00824786 0.512624
549 16 16963 16947 123.393 120 0.165414 0.512828
550 15 16988 16973 123.358 104 0.189045 0.51294
551 16 17013 16997 123.308 96 0.566199 0.513122
552 16 17037 17021 123.259 96 0.205699 0.51339
553 15 17066 17051 123.253 120 1.50583 0.513442
554 16 17095 17079 123.232 112 1.95111 0.513388
555 16 17127 17111 123.241 128 0.0519989 0.51348
556 16 17159 17143 123.249 128 1.0562 0.513629
557 16 17188 17172 123.236 116 0.321593 0.513379
558 16 17220 17204 123.245 128 0.34435 0.513274
559 15 17254 17239 123.275 140 0.0733466 0.513406
2020-06-08 20:46:28.600176 min lat: 0.00726443 max lat: 3.03968 avg lat: 0.513256
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 17288 17272 123.29 132 0.127781 0.513256
561 16 17316 17300 123.27 112 0.339301 0.513318
562 16 17343 17327 123.243 108 0.236572 0.513485
563 16 17372 17356 123.23 116 1.6081 0.513577
564 16 17403 17387 123.231 124 0.0666848 0.513504
565 16 17433 17417 123.226 120 1.62159 0.513606
566 16 17458 17442 123.185 100 0.610262 0.513778
567 16 17490 17474 123.193 128 0.0960859 0.513776
568 16 17522 17506 123.201 128 0.691232 0.513809
569 16 17553 17537 123.203 124 0.516746 0.51385
570 15 17581 17566 123.19 116 1.38389 0.513848
571 16 17614 17598 123.198 128 0.0420619 0.513756
572 16 17642 17626 123.179 112 0.438171 0.513841
573 16 17671 17655 123.166 116 0.952635 0.51396
574 16 17697 17681 123.133 104 0.0575533 0.513836
575 16 17727 17711 123.127 120 0.297035 0.513906
576 16 17755 17739 123.108 112 0.270716 0.514065
577 15 17785 17770 123.11 124 1.35757 0.514109
578 15 17814 17799 123.097 116 0.256755 0.513921
579 15 17848 17833 123.119 136 0.0650893 0.514007
2020-06-08 20:46:48.604263 min lat: 0.00726443 max lat: 3.03968 avg lat: 0.514024
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 17880 17864 123.121 124 0.0475618 0.514024
581 16 17909 17893 123.109 116 0.367496 0.514028
582 16 17936 17920 123.083 108 0.0784211 0.513969
583 16 17967 17951 123.084 124 0.203588 0.514001
584 15 17999 17984 123.099 132 0.0760753 0.514197
585 15 18027 18012 123.08 112 1.54829 0.514226
586 16 18058 18042 123.075 120 0.0488779 0.514229
587 16 18088 18072 123.07 120 0.788804 0.514299
588 16 18120 18104 123.078 128 0.0492432 0.514288
589 16 18148 18132 123.059 112 0.439958 0.514217
590 16 18183 18167 123.088 140 0.263807 0.514094
591 16 18211 18195 123.069 112 0.795766 0.51436
592 16 18243 18227 123.078 128 0.0553352 0.514339
593 16 18271 18255 123.059 112 1.40097 0.514367
594 15 18300 18285 123.054 120 0.314329 0.514379
595 16 18334 18318 123.069 132 0.0830795 0.514141
596 16 18363 18347 123.057 116 0.343694 0.514137
597 16 18391 18375 123.038 112 1.8729 0.514305
598 16 18419 18403 123.02 112 0.166918 0.514431
599 16 18448 18432 123.008 116 0.323109 0.514472
2020-06-08 20:47:08.607204 min lat: 0.00726443 max lat: 3.03968 avg lat: 0.514576
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 14 18466 18452 122.936 80 1.91121 0.514576
Total time run: 600.536
Total reads made: 18466
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 122.997
Average IOPS: 30
Stddev IOPS: 2.7656
Max IOPS: 40
Min IOPS: 20
Average Latency(s): 0.514993
Max latency(s): 3.03968
Min latency(s): 0.00726443
Total time run: 600.536
Total reads made: 18466
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 122.997
Average IOPS: 30
Stddev IOPS: 2.7656
Max IOPS: 40
Min IOPS: 20
Average Latency(s): 0.514993
Max latency(s): 3.03968
Min latency(s): 0.00726443
Cleanup Pool with 256 PGs and remove it
Note that I could also have simply deleted the pool testbench2 from within the Ceph Dashboard.
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 rados -p testbench2 cleanup
Removed 11055 objects
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph osd pool delete testbench2 testbench2 --yes-i-really-really-mean-it
Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool
Error: non zero exit code: 1: OCI runtime error
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph tell mon.* injectargs --mon-allow-pool-delete=true
mon.f5-422-01: injectargs:mon_allow_pool_delete = 'true'
mon.f5-422-02: injectargs:mon_allow_pool_delete = 'true'
mon.f5-422-03: injectargs:mon_allow_pool_delete = 'true'
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph osd pool delete testbench2 testbench2 --yes-i-really-really-mean-it
pool 'testbench2' removed
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph tell mon.* injectargs --mon-allow-pool-delete=false
mon.f5-422-01: injectargs:mon_allow_pool_delete = 'false'
mon.f5-422-02: injectargs:mon_allow_pool_delete = 'false'
mon.f5-422-03: injectargs:mon_allow_pool_delete = 'false'
[root@f5-422-01 ~]#
Quick libvirt and Ceph RBD Smoke Test
Since I already have
an RBD pool libvirt-test that is configured for my Fedora32 hypervisor
and Nautilus can scale PGs up and down as well as auto-scale,
I ran my first tests withe the current 16 PGs.
This is not ideal though, since my last tests were with the EPYC based hypervisor which has a 10 Gig network connection, but I broke something in my RBD setup on that box and have not fixed it yet :-/
FIXME: re-test with 256 PGs and VM on EPYC for comparison with earlier results.
root@hypervisor ~ # virsh vol-list --pool Ceph-HouseNet-libvirt-test
Name Path
---------------------------------------------------------------------
20200608-fio-testing-1TiB libvirt-test/20200608-fio-testing-1TiB
Prepare Test VM
NOTE: The hypervisor used is not my main one, this one only has a single 1 Gbit network connection. So even with 0 overhead and no other traffic, it simply can not reach the 140MiB/s I saw in the earlier test. And since in previous tests there was no huge difference between a 1st and a 2nd run, as a quick smoke test, I only ran each test once.
As last time, I attached the 1 TiB RBD (20200608-fio-testing-1TiB) to a VM (CentOS 8) as a virtio block device, formatted it with XFS, mounted it and ran fio.
But since the VM in question only had 4 GiB RAM and I wanted this smoke test done quickly, I opted to do my fio tests with 16 GiB as size, not 160 like last time.
VM side of taking the RBD image in use (click to expand)
[root@guest13 ~]# dmesg | tail -n 4
[15125.785114] pcieport 0000:00:02.6: bridge window [mem 0xfc000000-0xfc1fffff]
[15125.787750] pcieport 0000:00:02.6: bridge window [mem 0xfde00000-0xfdffffff 64bit pref]
[15125.791450] virtio-pci 0000:07:00.0: enabling device (0000 -> 0002)
[15125.797291] virtio_blk virtio5: [vdb] 2147483648 512-byte logical blocks (1.10 TB/1.00 TiB)
[root@guest13 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 19G 0 part
├─cl_guest13-root 253:0 0 17G 0 lvm /
└─cl_guest13-swap 253:1 0 2G 0 lvm [SWAP]
vdb 252:16 0 1T 0 disk
[root@guest13 ~]# mkfs.xfs /dev/vdb
meta-data=/dev/vdb isize=512 agcount=4, agsize=67108864 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=268435456, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=131072, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@guest13 ~]# mount /dev/vdb /mnt/
[root@guest13 ~]# df -h /mnt/
Filesystem Size Used Avail Use% Mounted on
/dev/vdb 1.0T 7.2G 1017G 1% /mnt
[root@guest13 ~]# mkdir /mnt/fiotest/
[root@guest13 ~]# uname -a # I think the used kickstart is without package upgrades in %post
Linux guest13.internal.pcfe.net 4.18.0-147.el8.x86_64 #1 SMP Wed Dec 4 21:51:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@guest13 ~]# dnf upgrade # yes it was, let's upgrade
[reboot and mount again]
[root@guest13 ~]# uname -a
Linux guest13.internal.pcfe.net 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Sequential Read, 4k Blocksize
READ: bw=85.8MiB/siops: avg=21940.29
fio --name=readtest --rw=read --size=16g --directory=/mnt/fiotest/ --bs=4k
readtest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
readtest: Laying out IO file (1 file / 16384MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=101MiB/s,w=0KiB/s][r=25.9k,w=0 IOPS][eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=4207: Mon Jun 8 22:14:32 2020
read: IOPS=21.0k, BW=85.8MiB/s (89.9MB/s)(16.0GiB/191065msec)
clat (nsec): min=1843, max=328591k, avg=41103.33, stdev=1721413.61
lat (usec): min=2, max=328591, avg=42.15, stdev=1721.41
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3],
| 30.00th=[ 3], 40.00th=[ 3], 50.00th=[ 4], 60.00th=[ 4],
| 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 5], 95.00th=[ 5],
| 99.00th=[ 10], 99.50th=[ 18], 99.90th=[ 1004], 99.95th=[25822],
| 99.99th=[91751]
bw ( KiB/s): min=16384, max=123819, per=99.95%, avg=87761.29, stdev=17705.34, samples=382
iops : min= 4096, max=30954, avg=21940.29, stdev=4426.32, samples=382
lat (usec) : 2=0.07%, 4=81.84%, 10=17.18%, 20=0.59%, 50=0.19%
lat (usec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=0.03%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%
cpu : usr=9.13%, sys=9.14%, ctx=4482, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4194304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=85.8MiB/s (89.9MB/s), 85.8MiB/s-85.8MiB/s (89.9MB/s-89.9MB/s), io=16.0GiB (17.2GB), run=191065-191065msec
Disk stats (read/write):
vdb: ios=16473/8, merge=0/1, ticks=2094592/785, in_queue=2087094, util=6.09%
Sequential Read, 4m Blocksize
READ: bw=89.6MiB/siops: avg=22.34
fio --name=readtest --rw=read --size=16g --directory=/mnt/fiotest/ --bs=4m
readtest: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=112MiB/s,w=0KiB/s][r=28,w=0 IOPS][eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=4210: Mon Jun 8 22:17:36 2020
read: IOPS=22, BW=89.6MiB/s (93.9MB/s)(16.0GiB/182944msec)
clat (usec): min=882, max=613961, avg=44528.03, stdev=51858.65
lat (usec): min=883, max=613963, avg=44530.35, stdev=51858.37
clat percentiles (usec):
| 1.00th=[ 1106], 5.00th=[ 1303], 10.00th=[ 1369], 20.00th=[ 1614],
| 30.00th=[ 3458], 40.00th=[ 9634], 50.00th=[ 25035], 60.00th=[ 42206],
| 70.00th=[ 61080], 80.00th=[ 86508], 90.00th=[121111], 95.00th=[145753],
| 99.00th=[200279], 99.50th=[225444], 99.90th=[295699], 99.95th=[354419],
| 99.99th=[616563]
bw ( KiB/s): min=32768, max=131072, per=100.00%, avg=91833.11, stdev=17359.10, samples=364
iops : min= 8, max= 32, avg=22.34, stdev= 4.24, samples=364
lat (usec) : 1000=0.22%
lat (msec) : 2=22.12%, 4=11.43%, 10=6.52%, 20=6.45%, 50=17.26%
lat (msec) : 100=20.65%, 250=15.06%, 500=0.27%, 750=0.02%
cpu : usr=0.04%, sys=6.24%, ctx=4269, majf=0, minf=524
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=89.6MiB/s (93.9MB/s), 89.6MiB/s-89.6MiB/s (93.9MB/s-93.9MB/s), io=16.0GiB (17.2GB), run=182944-182944msec
Disk stats (read/write):
vdb: ios=16536/0, merge=2/0, ticks=2140652/0, in_queue=2132410, util=7.00%
Sequential Write, 4k Blocksize
WRITE: bw=79.9MiB/siops: avg=20440.18
fio --name=writetest --rw=write --size=16g --directory=/mnt/fiotest/ --bs=4k
writetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
writetest: Laying out IO file (1 file / 16384MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=87.2MiB/s][r=0,w=22.3k IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=4213: Mon Jun 8 22:21:01 2020
write: IOPS=20.5k, BW=79.9MiB/s (83.8MB/s)(16.0GiB/204930msec)
clat (usec): min=2, max=44243, avg=43.39, stdev=559.10
lat (usec): min=3, max=44244, avg=44.73, stdev=559.10
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6],
| 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 7],
| 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 8], 95.00th=[ 10],
| 99.00th=[ 25], 99.50th=[ 50], 99.90th=[ 9372], 99.95th=[11076],
| 99.99th=[12125]
bw ( KiB/s): min=35840, max=705232, per=99.87%, avg=81760.88, stdev=44608.86, samples=409
iops : min= 8960, max=176308, avg=20440.18, stdev=11152.19, samples=409
lat (usec) : 4=5.33%, 10=90.85%, 20=2.13%, 50=1.19%, 100=0.04%
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.38%, 20=0.08%, 50=0.01%
cpu : usr=10.63%, sys=14.15%, ctx=19217, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4194304,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=79.9MiB/s (83.8MB/s), 79.9MiB/s-79.9MiB/s (83.8MB/s-83.8MB/s), io=16.0GiB (17.2GB), run=204930-204930msec
Disk stats (read/write):
vdb: ios=0/1613, merge=0/4, ticks=0/38770165, in_queue=38762165, util=1.30%
Sequential Write, 4m Blocksize
WRITE: bw=77.8MiB/siops: avg=19.95
fio --name=writetest --rw=write --size=16g --directory=/mnt/fiotest/ --bs=4m
writetest: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=76.1MiB/s][r=0,w=19 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=4219: Mon Jun 8 22:24:32 2020
write: IOPS=19, BW=77.8MiB/s (81.6MB/s)(16.0GiB/210545msec)
clat (usec): min=1754, max=109764, avg=49735.40, stdev=16393.01
lat (usec): min=1791, max=109914, avg=49904.31, stdev=16394.86
clat percentiles (usec):
| 1.00th=[ 1778], 5.00th=[ 21103], 10.00th=[ 30016], 20.00th=[ 38536],
| 30.00th=[ 44827], 40.00th=[ 47973], 50.00th=[ 52167], 60.00th=[ 53740],
| 70.00th=[ 55837], 80.00th=[ 62653], 90.00th=[ 69731], 95.00th=[ 71828],
| 99.00th=[ 83362], 99.50th=[ 87557], 99.90th=[ 99091], 99.95th=[108528],
| 99.99th=[109577]
bw ( KiB/s): min=40960, max=827392, per=100.00%, avg=81999.56, stdev=47883.25, samples=409
iops : min= 10, max= 202, avg=19.95, stdev=11.69, samples=409
lat (msec) : 2=2.61%, 4=1.51%, 10=0.07%, 20=0.66%, 50=42.80%
lat (msec) : 100=52.25%, 250=0.10%
cpu : usr=0.41%, sys=6.64%, ctx=22460, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4096,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=77.8MiB/s (81.6MB/s), 77.8MiB/s-77.8MiB/s (81.6MB/s-81.6MB/s), io=16.0GiB (17.2GB), run=210545-210545msec
Disk stats (read/write):
vdb: ios=0/16660, merge=0/24, ticks=0/39547058, in_queue=39538722, util=1.36%
Sequential, 80% Read, 20% Write, 4k Blocksize
READ: bw=81.9MiB/siops: avg=20959.49WRITE: bw=20.5MiB/siops: avg=5239.82
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=16g --directory=/mnt/fiotest/ --bs=4k
read80write20test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
read80write20test: Laying out IO file (1 file / 16384MiB)
Jobs: 1 (f=1): [M(1)][100.0%][r=96.1MiB/s,w=24.6MiB/s][r=24.6k,w=6306 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=4394: Mon Jun 8 22:30:11 2020
read: IOPS=20.0k, BW=81.9MiB/s (85.9MB/s)(12.8GiB/160073msec)
clat (nsec): min=1866, max=439393k, avg=41066.81, stdev=1806675.83
lat (usec): min=2, max=439393, avg=42.06, stdev=1806.81
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3],
| 30.00th=[ 3], 40.00th=[ 3], 50.00th=[ 3], 60.00th=[ 4],
| 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 5], 95.00th=[ 5],
| 99.00th=[ 9], 99.50th=[ 19], 99.90th=[ 971], 99.95th=[24511],
| 99.99th=[91751]
bw ( KiB/s): min=16384, max=122900, per=99.99%, avg=83838.07, stdev=20691.10, samples=320
iops : min= 4096, max=30725, avg=20959.49, stdev=5172.77, samples=320
write: IOPS=5240, BW=20.5MiB/s (21.5MB/s)(3277MiB/160073msec)
clat (usec): min=2, max=1532, avg= 5.59, stdev= 3.83
lat (usec): min=3, max=1533, avg= 6.62, stdev= 4.11
clat percentiles (nsec):
| 1.00th=[ 3408], 5.00th=[ 3536], 10.00th=[ 3728], 20.00th=[ 3920],
| 30.00th=[ 4080], 40.00th=[ 4512], 50.00th=[ 5024], 60.00th=[ 5536],
| 70.00th=[ 5984], 80.00th=[ 6560], 90.00th=[ 7328], 95.00th=[ 8896],
| 99.00th=[18560], 99.50th=[24192], 99.90th=[40192], 99.95th=[51456],
| 99.99th=[80384]
bw ( KiB/s): min= 4384, max=30780, per=99.99%, avg=20959.44, stdev=5163.88, samples=320
iops : min= 1096, max= 7695, avg=5239.82, stdev=1290.97, samples=320
lat (usec) : 2=0.10%, 4=74.33%, 10=24.20%, 20=0.89%, 50=0.38%
lat (usec) : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=10.41%, sys=11.11%, ctx=3742, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3355445,838859,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=81.9MiB/s (85.9MB/s), 81.9MiB/s-81.9MiB/s (85.9MB/s-85.9MB/s), io=12.8GiB (13.7GB), run=160073-160073msec
WRITE: bw=20.5MiB/s (21.5MB/s), 20.5MiB/s-20.5MiB/s (21.5MB/s-21.5MB/s), io=3277MiB (3436MB), run=160073-160073msec
Disk stats (read/write):
vdb: ios=13249/3272, merge=0/3, ticks=1751978/464437, in_queue=2208132, util=6.93%
Sequential, 80% Read, 20% Write, 4m Blocksize
READ: bw=85.4MiB/siops: avg=21.34WRITE: bw=22.3MiB/siops: avg=6.21
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=16g --directory=/mnt/fiotest/ --bs=4m
read80write20test: (g=0): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=60.0MiB/s,w=40.0MiB/s][r=15,w=10 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=4397: Mon Jun 8 22:32:44 2020
read: IOPS=21, BW=85.4MiB/s (89.6MB/s)(12.7GiB/152077msec)
clat (usec): min=824, max=541502, avg=45766.26, stdev=57657.21
lat (usec): min=825, max=541504, avg=45768.34, stdev=57657.00
clat percentiles (usec):
| 1.00th=[ 1020], 5.00th=[ 1237], 10.00th=[ 1319], 20.00th=[ 1598],
| 30.00th=[ 3458], 40.00th=[ 9503], 50.00th=[ 24511], 60.00th=[ 41681],
| 70.00th=[ 60031], 80.00th=[ 85459], 90.00th=[121111], 95.00th=[149947],
| 99.00th=[263193], 99.50th=[308282], 99.90th=[408945], 99.95th=[480248],
| 99.99th=[541066]
bw ( KiB/s): min=24526, max=131072, per=100.00%, avg=87759.99, stdev=24141.96, samples=303
iops : min= 5, max= 32, avg=21.34, stdev= 5.89, samples=303
write: IOPS=5, BW=22.3MiB/s (23.4MB/s)(3392MiB/152077msec)
clat (usec): min=1789, max=6071, avg=2933.39, stdev=590.52
lat (usec): min=1885, max=6340, avg=3080.36, stdev=600.38
clat percentiles (usec):
| 1.00th=[ 1893], 5.00th=[ 2147], 10.00th=[ 2278], 20.00th=[ 2343],
| 30.00th=[ 2442], 40.00th=[ 2704], 50.00th=[ 3064], 60.00th=[ 3195],
| 70.00th=[ 3261], 80.00th=[ 3326], 90.00th=[ 3523], 95.00th=[ 3818],
| 99.00th=[ 4817], 99.50th=[ 5342], 99.90th=[ 6063], 99.95th=[ 6063],
| 99.99th=[ 6063]
bw ( KiB/s): min= 8126, max=81920, per=100.00%, avg=25816.79, stdev=16308.32, samples=269
iops : min= 1, max= 20, avg= 6.21, stdev= 4.00, samples=269
lat (usec) : 1000=0.44%
lat (msec) : 2=17.48%, 4=28.66%, 10=5.96%, 20=5.35%, 50=14.33%
lat (msec) : 100=15.84%, 250=10.99%, 500=0.93%, 750=0.02%
cpu : usr=0.13%, sys=7.44%, ctx=3522, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3248,848,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=85.4MiB/s (89.6MB/s), 85.4MiB/s-85.4MiB/s (89.6MB/s-89.6MB/s), io=12.7GiB (13.6GB), run=152077-152077msec
WRITE: bw=22.3MiB/s (23.4MB/s), 22.3MiB/s-22.3MiB/s (23.4MB/s-23.4MB/s), io=3392MiB (3557MB), run=152077-152077msec
Disk stats (read/write):
vdb: ios=13120/3226, merge=1/0, ticks=1781440/780190, in_queue=2553537, util=7.56%
sequential write, 2300 bytes, fsync
Normally this test is not part of my baseline but a colleague inquired, so pasting here.
WRITE: bw=206KiB/siops: avg=93.01
fio --rw=write --ioengine=sync --fdatasync=1 --size=22m --bs=2300 --name=etcdtest --directory=/mnt/fiotest/
etcdtest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.7
Starting 1 process
etcdtest: Laying out IO file (1 file / 22MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=217KiB/s][r=0,w=97 IOPS][eta 00m:00s]
etcdtest: (groupid=0, jobs=1): err= 0: pid=4402: Mon Jun 8 22:34:34 2020
write: IOPS=91, BW=206KiB/s (211kB/s)(21.0MiB/109322msec)
clat (usec): min=12, max=526, avg=26.99, stdev=11.56
lat (usec): min=13, max=528, avg=29.14, stdev=11.65
clat percentiles (usec):
| 1.00th=[ 18], 5.00th=[ 19], 10.00th=[ 20], 20.00th=[ 23],
| 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 25], 60.00th=[ 27],
| 70.00th=[ 30], 80.00th=[ 31], 90.00th=[ 34], 95.00th=[ 39],
| 99.00th=[ 58], 99.50th=[ 67], 99.90th=[ 82], 99.95th=[ 88],
| 99.99th=[ 515]
bw ( KiB/s): min= 4, max= 247, per=100.00%, avg=208.53, stdev=31.13, samples=215
iops : min= 2, max= 110, avg=93.01, stdev=13.84, samples=215
lat (usec) : 20=11.57%, 50=86.65%, 100=1.74%, 500=0.02%, 750=0.02%
fsync/fdatasync/sync_file_range:
sync (msec): min=4, max=1145, avg=10.86, stdev=19.30
sync percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 6],
| 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 13], 80.00th=[ 13], 90.00th=[ 14], 95.00th=[ 18],
| 99.00th=[ 34], 99.50th=[ 38], 99.90th=[ 71], 99.95th=[ 105],
| 99.99th=[ 986]
cpu : usr=0.18%, sys=0.69%, ctx=35703, majf=0, minf=12
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,10029,0,0 short=10029,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=206KiB/s (211kB/s), 206KiB/s-206KiB/s (211kB/s-211kB/s), io=21.0MiB (23.1MB), run=109322-109322msec
Disk stats (read/write):
vdb: ios=0/25943, merge=0/15, ticks=0/259443, in_queue=248278, util=14.06%
sequential write, 2300 bytes, NO fsync
Since it seems silly to me to call fsync after 2300 bytes, same test without that option to have a comparison.
WRITE: bw=100MiB/siops: avg=45595.99
fio --rw=write --ioengine=sync --size=16g --bs=2300 --name=etcdtest --directory=/mnt/fiotest/
etcdtest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.7
Starting 1 process
etcdtest: Laying out IO file (1 file / 16384MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=102MiB/s][r=0,w=46.6k IOPS][eta 00m:00s]
etcdtest: (groupid=0, jobs=1): err= 0: pid=4411: Mon Jun 8 22:37:18 2020
write: IOPS=45.6k, BW=100MiB/s (105MB/s)(15.0GiB/163747msec)
clat (nsec): min=1570, max=44945k, avg=17722.52, stdev=351436.48
lat (usec): min=2, max=44946, avg=18.75, stdev=351.47
clat percentiles (nsec):
| 1.00th=[ 1624], 5.00th=[ 2160], 10.00th=[ 2512],
| 20.00th=[ 2832], 30.00th=[ 3152], 40.00th=[ 3408],
| 50.00th=[ 4576], 60.00th=[ 5216], 70.00th=[ 5664],
| 80.00th=[ 6048], 90.00th=[ 6624], 95.00th=[ 7648],
| 99.00th=[ 14784], 99.50th=[ 19584], 99.90th=[ 7766016],
| 99.95th=[10027008], 99.99th=[11599872]
bw ( KiB/s): min=71583, max=447138, per=99.96%, avg=102412.57, stdev=35742.40, samples=327
iops : min=31870, max=199074, avg=45595.99, stdev=15913.14, samples=327
lat (usec) : 2=3.02%, 4=41.94%, 10=53.04%, 20=1.53%, 50=0.32%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.10%, 20=0.05%, 50=0.01%
cpu : usr=18.24%, sys=21.96%, ctx=11258, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,7469508,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=15.0GiB (17.2GB), run=163747-163747msec
Disk stats (read/write):
vdb: ios=0/15883, merge=0/4, ticks=0/30558816, in_queue=30550869, util=1.59%
New Shelves for Cluster
In late August 2020 I finally moved the 4 cubes off the desk they were hogging and onto a wall.
I went from this
to this