TerraMaster F5-422 with Fedora 29
Table of Contents
I installed Fedora 29 on my TerraMaster F5-422 for a quick Ceph test.
Obviously if you plan to use Fedora on this hardware for anything more than a quick test, use the latest version! Or try RHEL7 or CentOS Stream on it.
This is my braindump.
Install Fedora
- create USB boot medium as per docs
- connect screen, keyboard, mouse and USB boot medium
- enter firmware setup
- change OS type from Windows to Intel Linux
- set boot to UEFI only
- save and exit
- boot from USB
- I did a minimal install with the Ansible node add-on selected
I started with F29, even though today F31 is the current version, as my existing Ceph cluster runs that and before I order at least 3 more of these F5-422 I wanted to see the F5-422 do Ceph for a week or three. So I quickly added it to my known working Ceph setup.
The plan is to eventually remove the 32 bit hardware (there’s nothing wrong with ARM, aarch64 is great). But I want to rid myself of the ODROID-HC2 boards, the 32 bit arch is old and the hardware turned out to have too many issues for my liking.
Install ssh Public Key and Test
pcfe@workstation ~ $ ssh-copy-id -f -i ~/.ssh/id_USBkey.pub root@f5-422-01
pcfe@workstation ansible (master) $ ansible f5-422-01 -m ping -e ansible_user=root
f5-422-01 | SUCCESS => {
"changed": false,
"ping": "pong"
}
Configure
since this is only for a quick test, I did not bother to properly rename my existing arm playbooks, I just added the host.
pcfe@workstation pcfe.net (master) $ ansible-playbook -l f5-422-01 arm-fedora-initial-setup.yml
pcfe@workstation pcfe.net (master) $ ansible-playbook -l f5-422-01 ceph-prepare-arm.yml
pcfe@workstation pcfe.net (master) $ ansible-playbook -l f5-422-01 terramaster-general-setup.yml
Setup for Ceph
The F5-422 was added to my known working Ceph setup
simply by adding the host to my [osds] section and in my hostvars file for it
devices:
- /dev/sdc
(the latter because until now all my ODROID OSDs used /dev/sda).
And then I ran ceph-ansible again.
[ansible@ceph-ansible ceph-ansible]$ ansible-playbook -vv site.yml
Check
OSD shows up as expected
[root@ceph-mon-01 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 11.82544 root default
-17 1.81929 host f5-422-01
7 hdd 1.81929 osd.7 up 1.00000 1.00000
-13 0.90959 host odroid-hc2-00
6 hdd 0.90959 osd.6 up 1.00000 1.00000
-11 0.90970 host odroid-hc2-01
0 hdd 0.90970 osd.0 up 1.00000 1.00000
-9 0.90970 host odroid-hc2-02
4 hdd 0.90970 osd.4 up 1.00000 1.00000
-15 1.81929 host odroid-hc2-03
5 hdd 1.81929 osd.5 up 1.00000 1.00000
-3 1.81929 host odroid-hc2-04
1 hdd 1.81929 osd.1 up 1.00000 1.00000
-5 1.81929 host odroid-hc2-05
2 hdd 1.81929 osd.2 up 1.00000 1.00000
-7 1.81929 host odroid-hc2-06
3 hdd 1.81929 osd.3 up 1.00000 1.00000
Testing Expectations
What follows is just a quick test to see if things work as expected.
I started with only a single HDD with BlueStore in the F5-422.
The plan is to eventually use 1 SATA SSD and up to 4 SATA HDD in this 5 slot chassis,
as well as the 10G NIC for cluster_network, once I get rid of the ODROID-HC2 boards
and purchased a 10Gig switch.
Note that since the ODROID-HC2 nodes (⅞ of the storage cluster) have only one network interface, the below quick test was run with only the PCIe 1Gig connection of the F5-422 (⅛ of cluster).
Since ⅞ of my OSDs are small ARM 32 bit boards, this is not about how much performance I get, this is just to give the F5-422 a smoke test. My expectation is to see roughly the numbers I get from ODROID-HC nodes with possibly a small amount of performance increase. If only because since the last test I moved everything besides the OSD daemons away from the ODROID-HC2 nodes onto separate VMs on my hypervisor.
root@epyc ~ # virsh list | grep -i ceph
22 ceph-rgw running
23 ceph-dashboard running
24 ceph-ansible.sattest.pcfe.net running
25 ceph-mds-01 running
26 ceph-mds-02 running
27 ceph-mon-mgr-01 running
28 ceph-mon-mgr-02 running
29 ceph-mon-mgr-03 running
Performance Measurements with rados bench
All rados bench tests were run on ceph-mon-01.
Sequential Write Test
Total time run: 600.902758
Total writes made: 9075
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 60.4091
Stddev Bandwidth: 16.1433
Max bandwidth (MB/sec): 104
Min bandwidth (MB/sec): 0
Average IOPS: 15
Stddev IOPS: 4
Max IOPS: 26
Min IOPS: 0
Average Latency(s): 1.05936
Stddev Latency(s): 0.528235
Max latency(s): 5.63799
Min latency(s): 0.205941
rados bench -p testbench 600 write --no-cleanup | tee --append $(date +%F)-rados-test-write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_ceph-mon-01.internal.pcfe.net_9766
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 16 16 0 0 0 - 0
1 16 25 9 35.9437 36 0.999667 0.719038
2 16 36 20 39.9654 44 0.965181 0.820069
3 16 53 37 49.303 68 0.514562 0.984278
4 16 70 54 53.9737 68 1.62787 0.970278
5 16 87 71 56.7762 68 1.85171 0.988755
6 16 101 85 56.6459 56 1.78279 1.02105
7 16 119 103 58.8366 72 1.83816 1.02544
8 16 132 116 57.9816 52 1.01994 1.02574
9 16 151 135 59.9821 76 0.84474 1.00335
10 16 168 152 60.783 68 1.2096 0.985232
11 16 181 165 59.9841 52 1.47608 0.993869
12 16 194 178 59.3184 52 0.355273 1.01453
13 16 209 193 59.3703 60 1.44779 1.01178
14 16 224 208 59.4148 60 0.421049 1.02496
15 16 237 221 58.92 52 1.33695 1.03794
16 16 258 242 60.4866 84 0.867177 1.02964
17 16 270 254 59.7519 48 1.15287 1.02833
18 16 292 276 61.3191 88 0.512198 1.01418
19 16 303 287 60.4074 44 1.33367 1.01234
2020-02-09 13:03:27.986707 min lat: 0.257212 max lat: 2.39554 avg lat: 1.0285
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 321 305 60.9865 72 0.299995 1.0285
21 16 337 321 61.1281 64 0.462599 1.01964
22 16 351 335 60.8947 56 1.3906 1.02236
23 16 371 355 61.7249 80 0.279012 1.01151
24 16 384 368 61.3194 52 0.494773 1.0052
25 16 405 389 62.2262 84 0.979229 1.00596
26 16 423 407 62.6018 72 0.920855 1.00169
27 16 441 425 62.9495 72 0.446715 0.997841
28 16 454 438 62.5583 52 1.40589 0.995986
29 16 470 454 62.6078 64 0.707754 0.997711
30 16 483 467 62.254 52 1.31482 1.00094
31 16 495 479 61.793 48 0.502646 1.00576
32 16 513 497 62.1117 72 0.72096 1.0149
33 16 526 510 61.8051 52 0.641428 1.01502
34 16 534 518 60.9285 32 1.52362 1.02316
35 16 548 532 60.7875 56 0.396146 1.02767
36 16 572 556 61.7652 96 1.01682 1.02673
37 16 586 570 61.6092 56 1.0081 1.02211
38 16 595 579 60.9353 36 0.545813 1.02074
39 16 615 599 61.4238 80 0.551727 1.03196
2020-02-09 13:03:47.990056 min lat: 0.220596 max lat: 2.39554 avg lat: 1.02556
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 630 614 61.3881 60 1.0477 1.02556
41 16 639 623 60.7688 36 0.367287 1.02865
42 16 662 646 61.5121 92 1.22463 1.03095
43 16 675 659 61.2908 52 1.12965 1.02829
44 16 688 672 61.0795 52 0.346489 1.03076
45 16 707 691 61.4108 76 0.923547 1.02874
46 16 724 708 61.5538 68 0.459035 1.02586
47 16 745 729 62.0312 84 1.41263 1.02193
48 16 756 740 61.6554 44 0.315988 1.02113
49 16 765 749 61.1318 36 1.92003 1.02667
50 16 785 769 61.5089 80 0.678639 1.02788
51 16 805 789 61.871 80 0.438226 1.02257
52 16 825 809 62.2194 80 0.703293 1.01647
53 16 844 828 62.4793 76 0.822998 1.01489
54 16 868 852 63.0998 96 0.518895 1.00806
55 16 879 863 62.7525 44 0.297265 1.00619
56 16 892 876 62.5604 52 2.09594 1.00896
57 16 910 894 62.7258 72 1.16125 1.01106
58 16 919 903 62.265 36 1.81771 1.01342
59 16 935 919 62.2943 64 0.564432 1.01436
2020-02-09 13:04:07.992664 min lat: 0.220596 max lat: 2.39554 avg lat: 1.01463
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 955 939 62.5892 80 0.37489 1.01463
61 16 973 957 62.7433 72 0.499918 1.01136
62 16 986 970 62.5699 52 1.69739 1.00877
63 16 1005 989 62.7829 76 0.40583 1.01176
64 16 1015 999 62.4269 40 0.334687 1.0126
65 16 1025 1009 62.0818 40 0.371407 1.0149
66 16 1039 1023 61.9896 56 0.793355 1.02191
67 16 1061 1045 62.3776 88 0.494712 1.02043
68 16 1075 1059 62.2837 56 0.392389 1.01806
69 16 1092 1076 62.3664 68 0.558789 1.01538
70 16 1101 1085 61.9897 36 1.67686 1.01626
71 16 1113 1097 61.7926 48 1.3903 1.02393
72 16 1125 1109 61.601 48 1.29205 1.027
73 16 1145 1129 61.8529 80 0.759281 1.02669
74 16 1162 1146 61.9358 68 1.09196 1.0274
75 16 1181 1165 62.1232 76 0.558122 1.02336
76 16 1195 1179 62.0426 56 0.36944 1.02193
77 16 1212 1196 62.1199 68 1.50207 1.02285
78 16 1220 1204 61.7337 32 0.881767 1.02454
79 16 1234 1218 61.661 56 1.5472 1.02887
2020-02-09 13:04:27.995120 min lat: 0.220596 max lat: 2.39554 avg lat: 1.02964
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 1245 1229 61.4401 44 0.46947 1.02964
81 16 1266 1250 61.7185 84 0.869824 1.03075
82 16 1286 1270 61.9413 80 0.940127 1.02798
83 16 1297 1281 61.7251 44 0.543775 1.02864
84 16 1320 1304 62.0854 92 1.1444 1.02551
85 16 1335 1319 62.0607 60 0.91686 1.02384
86 16 1349 1333 61.9902 56 1.34621 1.0261
87 16 1361 1345 61.8293 48 0.436642 1.02657
88 16 1380 1364 61.9903 76 0.982497 1.02576
89 16 1399 1383 62.1476 76 0.886141 1.02289
90 16 1413 1397 62.0792 56 0.236951 1.02206
91 16 1431 1415 62.1881 72 0.794521 1.02333
92 16 1446 1430 62.1643 60 0.750718 1.02147
93 16 1461 1445 62.1409 60 1.67608 1.02152
94 16 1471 1455 61.9053 40 1.20769 1.02554
95 16 1483 1467 61.7589 48 0.359532 1.0249
96 16 1496 1480 61.6572 52 2.05729 1.02768
97 16 1518 1502 61.9286 88 1.32309 1.02994
98 16 1529 1513 61.7456 44 1.5995 1.03004
99 16 1539 1523 61.5259 40 0.360335 1.03056
2020-02-09 13:04:47.997533 min lat: 0.220596 max lat: 2.39554 avg lat: 1.03137
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 1552 1536 61.4306 52 0.695813 1.03137
101 16 1566 1550 61.3768 56 0.506788 1.03692
102 16 1582 1566 61.4024 64 0.707194 1.0361
103 16 1595 1579 61.3111 52 1.37598 1.03694
104 16 1612 1596 61.3753 68 0.910059 1.03735
105 16 1626 1610 61.3241 56 1.39157 1.04042
106 16 1636 1620 61.1229 40 0.320451 1.03944
107 16 1641 1625 60.7385 20 0.517629 1.0398
108 16 1654 1638 60.6576 52 1.74088 1.04754
109 16 1674 1658 60.8349 80 0.82953 1.04772
110 16 1694 1678 61.009 80 0.924831 1.04541
111 16 1708 1692 60.9639 56 1.07404 1.04466
112 16 1726 1710 61.0623 72 0.767294 1.04258
113 16 1743 1727 61.1236 68 1.18545 1.04204
114 16 1761 1745 61.219 72 0.70038 1.0401
115 16 1779 1763 61.3126 72 1.2081 1.03977
116 16 1795 1779 61.3357 64 0.493872 1.03732
117 16 1814 1798 61.4609 76 1.53608 1.03645
118 16 1827 1811 61.3807 52 0.86784 1.03661
119 16 1845 1829 61.4699 72 1.41413 1.03794
2020-02-09 13:05:08.000090 min lat: 0.220596 max lat: 2.49436 avg lat: 1.03769
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 1854 1838 61.2576 36 1.569 1.03769
121 16 1869 1853 61.2471 60 0.31099 1.03764
122 16 1886 1870 61.3024 68 1.62688 1.04082
123 16 1903 1887 61.3568 68 0.990556 1.04129
124 16 1921 1905 61.4425 72 1.00014 1.0381
125 16 1938 1922 61.4949 68 1.18729 1.03735
126 16 1954 1938 61.5148 64 0.362773 1.0356
127 16 1968 1952 61.4713 56 1.49735 1.03482
128 16 1984 1968 61.491 64 0.670749 1.03615
129 16 2000 1984 61.5104 64 0.837719 1.03516
130 16 2016 2000 61.5295 64 0.465029 1.03617
131 16 2037 2021 61.7009 84 0.513157 1.0352
132 16 2051 2035 61.6577 56 0.875675 1.03358
133 16 2065 2049 61.6151 56 0.358026 1.03268
134 16 2082 2066 61.6627 68 0.730844 1.03323
135 16 2100 2084 61.7392 72 0.93341 1.0318
136 16 2122 2106 61.932 88 0.542591 1.03002
137 16 2140 2124 62.0054 72 1.29368 1.02922
138 16 2151 2135 61.8749 44 0.255559 1.02919
139 16 2168 2152 61.9189 68 0.884169 1.02933
2020-02-09 13:05:28.003044 min lat: 0.220596 max lat: 2.49436 avg lat: 1.02885
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 2178 2162 61.7623 40 0.750233 1.02885
141 16 2192 2176 61.7214 56 1.64731 1.0324
142 16 2212 2196 61.85 80 0.374879 1.03084
143 16 2228 2212 61.865 64 0.869236 1.02976
144 16 2253 2237 62.1297 100 0.776265 1.02849
145 16 2266 2250 62.0598 52 0.822938 1.02738
146 16 2273 2257 61.8265 28 1.36192 1.02872
147 16 2284 2268 61.7052 44 0.403214 1.03043
148 16 2298 2282 61.6666 56 1.69363 1.03147
149 16 2323 2307 61.9238 100 0.855189 1.03117
150 16 2341 2325 61.9909 72 0.608582 1.02907
151 16 2362 2346 62.1366 84 1.35466 1.02789
152 16 2373 2357 62.0173 44 0.300306 1.02737
153 16 2388 2372 62.004 60 0.651139 1.02708
154 16 2403 2387 61.991 60 0.563983 1.02757
155 16 2425 2409 62.1587 88 1.13456 1.02645
156 16 2437 2421 62.0679 48 0.343948 1.02596
157 16 2458 2442 62.2075 84 0.773788 1.0263
158 16 2471 2455 62.1429 52 0.512806 1.02592
159 16 2485 2469 62.1042 56 0.517473 1.02558
2020-02-09 13:05:48.005461 min lat: 0.220596 max lat: 2.49436 avg lat: 1.02636
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 2499 2483 62.066 56 0.375696 1.02636
161 16 2517 2501 62.1276 72 1.54075 1.02669
162 16 2528 2512 62.0157 44 1.56129 1.02713
163 16 2542 2526 61.9788 56 1.02257 1.02927
164 16 2555 2539 61.9179 52 0.438684 1.02941
165 16 2570 2554 61.9062 60 0.601587 1.02947
166 16 2584 2568 61.8706 56 0.540074 1.03022
167 16 2601 2585 61.9073 68 1.05229 1.03053
168 16 2619 2603 61.9673 72 0.74145 1.02944
169 16 2634 2618 61.9556 60 0.883269 1.02854
170 16 2654 2638 62.0617 80 0.254259 1.02775
171 16 2672 2656 62.1198 72 0.728238 1.0275
172 16 2686 2670 62.0841 56 0.515241 1.02622
173 16 2697 2681 61.9796 44 0.365901 1.02593
174 16 2713 2697 61.9911 64 0.770919 1.027
175 16 2736 2720 62.1626 92 0.887043 1.02693
176 16 2748 2732 62.0821 48 0.505998 1.02611
177 16 2767 2751 62.1606 76 0.520819 1.0279
178 16 2779 2763 62.0811 48 0.813506 1.02719
179 16 2797 2781 62.1364 72 0.838757 1.02715
2020-02-09 13:06:08.007867 min lat: 0.220596 max lat: 2.49436 avg lat: 1.02639
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 2815 2799 62.1912 72 0.458732 1.02639
181 16 2828 2812 62.1348 52 1.21387 1.02615
182 16 2851 2835 62.2989 92 0.968025 1.02555
183 16 2867 2851 62.3081 64 0.601778 1.02494
184 16 2882 2866 62.2955 60 1.03278 1.02478
185 16 2895 2879 62.2399 52 1.03038 1.025
186 16 2905 2889 62.1203 40 1.01385 1.02566
187 16 2921 2905 62.1303 64 0.684071 1.02599
188 16 2936 2920 62.1189 60 0.378872 1.02502
189 16 2951 2935 62.1077 60 1.00953 1.02695
190 16 2967 2951 62.1176 64 0.337264 1.02701
191 16 2980 2964 62.0646 52 1.38757 1.02714
192 16 3002 2986 62.1996 88 0.459886 1.02686
193 16 3016 3000 62.1674 56 0.524664 1.0253
194 16 3027 3011 62.0738 44 1.8324 1.02701
195 16 3039 3023 62.0016 48 0.405856 1.02924
196 16 3057 3041 62.0525 72 0.963887 1.0288
197 16 3077 3061 62.1436 80 0.71254 1.02875
198 16 3090 3074 62.0923 52 0.975912 1.02771
199 16 3103 3087 62.0416 52 1.04967 1.02893
2020-02-09 13:06:28.010229 min lat: 0.220596 max lat: 2.50054 avg lat: 1.02913
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 3117 3101 62.0113 56 0.585423 1.02913
201 16 3137 3121 62.1008 80 1.21548 1.0293
202 16 3149 3133 62.0309 48 0.287415 1.02835
203 16 3159 3143 61.9224 40 0.837013 1.02883
204 16 3179 3163 62.0109 80 0.303604 1.02961
205 16 3197 3181 62.0596 72 1.26161 1.0295
206 16 3211 3195 62.0302 56 0.529859 1.02928
207 16 3227 3211 62.0397 64 1.00323 1.02997
208 16 3244 3228 62.0683 68 0.54914 1.02875
209 16 3261 3245 62.0966 68 0.879331 1.02803
210 16 3282 3266 62.2009 84 0.443783 1.02641
211 16 3302 3286 62.2852 80 0.759304 1.02482
212 16 3321 3305 62.3493 76 1.22509 1.02394
213 16 3338 3322 62.3758 68 0.344837 1.02359
214 16 3357 3341 62.4394 76 0.813468 1.02318
215 16 3370 3354 62.3908 52 1.29412 1.02281
216 16 3389 3373 62.4538 76 0.676758 1.0224
217 16 3406 3390 62.4793 68 0.682132 1.02198
218 16 3419 3403 62.4312 52 0.630897 1.02085
219 16 3441 3425 62.5479 88 0.605661 1.0207
2020-02-09 13:06:48.014569 min lat: 0.205941 max lat: 2.50054 avg lat: 1.01987
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 3460 3444 62.609 76 0.629193 1.01987
221 16 3472 3456 62.5429 48 0.311749 1.019
222 16 3489 3473 62.5674 68 0.346805 1.01954
223 16 3501 3485 62.5021 48 1.54776 1.0207
224 16 3517 3501 62.5087 64 0.48248 1.02122
225 16 3526 3510 62.3909 36 0.383721 1.02057
226 16 3544 3528 62.4333 72 0.45851 1.0231
227 16 3558 3542 62.405 56 1.00434 1.02321
228 16 3577 3561 62.4646 76 1.13025 1.02347
229 16 3593 3577 62.4712 64 1.20973 1.02262
230 16 3610 3594 62.4952 68 0.841369 1.02209
231 16 3622 3606 62.4325 48 1.14124 1.02199
232 16 3630 3614 62.3013 32 2.05326 1.02212
233 16 3644 3628 62.2742 56 2.20645 1.02501
234 16 3660 3644 62.2816 64 0.442021 1.02496
235 16 3676 3660 62.2888 64 1.01779 1.0256
236 16 3689 3673 62.2452 52 0.429697 1.02504
237 16 3711 3695 62.3538 88 0.884283 1.025
238 16 3719 3703 62.2263 32 0.350776 1.02422
239 16 3737 3721 62.2671 72 1.92895 1.02503
2020-02-09 13:07:08.016976 min lat: 0.205941 max lat: 2.76841 avg lat: 1.02506
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 3749 3733 62.2077 48 1.68159 1.02506
241 16 3763 3747 62.1819 56 2.23375 1.02586
242 16 3777 3761 62.1563 56 0.651304 1.02582
243 16 3794 3778 62.1803 68 1.20135 1.02726
244 16 3812 3796 62.2205 72 0.304737 1.02673
245 16 3829 3813 62.2441 68 0.827886 1.02579
246 16 3848 3832 62.3 76 1.30815 1.02562
247 16 3866 3850 62.339 72 1.24866 1.02474
248 16 3885 3869 62.3941 76 0.344778 1.02343
249 16 3903 3887 62.4326 72 1.20339 1.02308
250 16 3915 3899 62.3749 48 0.388509 1.02282
251 16 3936 3920 62.461 84 0.612729 1.02368
252 16 3949 3933 62.4194 52 0.800179 1.02307
253 16 3964 3948 62.4098 60 0.569355 1.02277
254 16 3984 3968 62.4791 80 0.538567 1.02193
255 16 4006 3990 62.5791 88 0.943727 1.02117
256 16 4018 4002 62.5221 48 0.893481 1.02086
257 16 4031 4015 62.4812 52 0.307218 1.02115
258 16 4045 4029 62.4559 56 1.63501 1.02232
259 16 4064 4048 62.5082 76 1.0753 1.02197
2020-02-09 13:07:28.020600 min lat: 0.205941 max lat: 3.24391 avg lat: 1.02176
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 4075 4059 62.437 44 1.25923 1.02176
261 16 4088 4072 62.3969 52 0.575064 1.02285
262 16 4107 4091 62.4488 76 1.15361 1.0226
263 16 4125 4109 62.4851 72 0.734936 1.02225
264 16 4147 4131 62.5817 88 0.881404 1.02143
265 16 4158 4142 62.5116 44 1.11599 1.02103
266 16 4173 4157 62.5021 60 1.4441 1.02109
267 16 4187 4171 62.4777 56 0.777546 1.02133
268 16 4194 4178 62.3491 28 0.345568 1.02147
269 16 4204 4188 62.266 40 0.392465 1.02385
270 16 4221 4205 62.2872 68 0.6742 1.02485
271 16 4235 4219 62.264 56 0.457756 1.02509
272 16 4252 4236 62.285 68 1.14141 1.02512
273 16 4271 4255 62.3352 76 0.403897 1.02461
274 16 4280 4264 62.2391 36 1.11847 1.02491
275 16 4300 4284 62.3036 80 0.416841 1.02543
276 16 4312 4296 62.2518 48 0.480459 1.025
277 16 4324 4308 62.2003 48 0.880668 1.0258
278 16 4344 4328 62.2643 80 0.50112 1.02584
279 16 4360 4344 62.2705 64 1.51894 1.02538
2020-02-09 13:07:48.022958 min lat: 0.205941 max lat: 3.24391 avg lat: 1.02544
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 4376 4360 62.2767 64 0.298465 1.02544
281 16 4387 4371 62.2116 44 1.2007 1.02566
282 16 4406 4390 62.2605 76 0.852084 1.02632
283 16 4421 4405 62.2525 60 1.22746 1.02621
284 16 4437 4421 62.2586 64 0.537122 1.02592
285 16 4452 4436 62.2506 60 1.12339 1.02612
286 16 4473 4457 62.3265 84 1.25211 1.0259
287 16 4485 4469 62.2766 48 1.56951 1.02577
288 16 4499 4483 62.2548 56 1.07617 1.02578
289 16 4520 4504 62.33 84 0.464628 1.02538
290 16 4537 4521 62.3495 68 1.02101 1.02499
291 16 4553 4537 62.3552 64 0.971103 1.0249
292 16 4571 4555 62.3882 72 0.407792 1.02403
293 16 4586 4570 62.38 60 0.960583 1.02378
294 16 4599 4583 62.3447 52 1.39614 1.02458
295 16 4621 4605 62.4316 88 0.921028 1.02364
296 16 4629 4613 62.3288 32 0.499578 1.02363
297 16 4647 4631 62.3613 72 0.767119 1.02416
298 16 4666 4650 62.4069 76 0.520095 1.02394
299 16 4681 4665 62.3988 60 1.3195 1.02341
2020-02-09 13:08:08.026566 min lat: 0.205941 max lat: 3.24391 avg lat: 1.02314
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 4700 4684 62.4441 76 0.912207 1.02314
301 16 4713 4697 62.4094 52 1.26264 1.02314
302 16 4730 4714 62.4279 68 1.30469 1.0233
303 16 4747 4731 62.4462 68 1.1475 1.02362
304 16 4763 4747 62.4513 64 0.515132 1.02338
305 16 4777 4761 62.4302 56 0.776413 1.02327
306 16 4782 4766 62.2915 20 1.72976 1.0234
307 16 4798 4782 62.297 64 2.05171 1.02572
308 16 4811 4795 62.2636 52 0.66068 1.02538
309 16 4822 4806 62.2045 44 1.474 1.02616
310 16 4826 4810 62.0554 16 1.61875 1.02632
311 16 4848 4832 62.1388 88 1.60971 1.02848
312 16 4862 4846 62.1191 56 0.648688 1.02866
313 16 4874 4858 62.074 48 1.0823 1.02853
314 16 4892 4876 62.1055 72 0.68574 1.02839
315 16 4909 4893 62.1242 68 1.50787 1.02867
316 16 4922 4906 62.0922 52 1.4479 1.02934
317 16 4935 4919 62.0603 52 0.950579 1.02919
318 16 4952 4936 62.0787 68 1.40094 1.02886
319 16 4964 4948 62.0346 48 1.52399 1.02925
2020-02-09 13:08:28.030367 min lat: 0.205941 max lat: 3.24391 avg lat: 1.02983
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 4979 4963 62.0282 60 0.773104 1.02983
321 16 4996 4980 62.0468 68 0.380772 1.02956
322 16 5014 4998 62.0776 72 0.968735 1.02929
323 16 5029 5013 62.0712 60 0.921594 1.02895
324 16 5044 5028 62.0648 60 1.42509 1.02931
325 16 5057 5041 62.0338 52 0.494559 1.02956
326 16 5073 5057 62.0398 64 1.42386 1.02956
327 16 5084 5068 61.9846 44 1.67015 1.02991
328 16 5103 5087 62.0273 76 1.70636 1.03066
329 16 5113 5097 61.9603 40 1.56148 1.03027
330 16 5126 5110 61.9301 52 1.44692 1.03113
331 16 5136 5120 61.8639 40 0.520712 1.03115
332 16 5152 5136 61.8703 64 1.78718 1.03266
333 16 5168 5152 61.8766 64 1.34369 1.03248
334 16 5188 5172 61.9309 80 0.82375 1.03196
335 16 5202 5186 61.9131 56 0.468702 1.03185
336 16 5213 5197 61.8598 44 1.36567 1.03194
337 16 5226 5210 61.8305 52 1.7922 1.03301
338 16 5243 5227 61.8488 68 1.41708 1.03352
339 16 5256 5240 61.8197 52 1.06945 1.03398
2020-02-09 13:08:48.032895 min lat: 0.205941 max lat: 3.24391 avg lat: 1.03419
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 5267 5251 61.7673 44 1.05264 1.03419
341 16 5286 5270 61.809 76 0.781764 1.03431
342 16 5298 5282 61.7686 48 0.400236 1.03419
343 16 5315 5299 61.7867 68 0.30216 1.03396
344 16 5333 5317 61.8164 72 1.18166 1.03362
345 16 5350 5334 61.8343 68 0.517266 1.03348
346 16 5366 5350 61.8405 64 0.446471 1.03369
347 16 5376 5360 61.7776 40 1.23066 1.0335
348 16 5392 5376 61.7839 64 0.267112 1.03374
349 16 5407 5391 61.7788 60 0.263442 1.0342
350 16 5420 5404 61.7508 52 0.432377 1.03446
351 16 5434 5418 61.7344 56 1.53818 1.03475
352 16 5447 5431 61.7068 52 1.09152 1.03539
353 16 5460 5444 61.6793 52 1.44189 1.03574
354 16 5480 5464 61.7309 80 0.733023 1.03564
355 16 5493 5477 61.7034 52 0.698052 1.03498
356 16 5506 5490 61.6762 52 1.34615 1.03611
357 16 5522 5506 61.6827 64 0.524145 1.03558
358 16 5538 5522 61.6891 64 1.37196 1.03607
359 16 5554 5538 61.6955 64 1.24049 1.03582
2020-02-09 13:09:08.035982 min lat: 0.205941 max lat: 3.24391 avg lat: 1.03504
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 5575 5559 61.7575 84 0.286644 1.03504
361 16 5588 5572 61.7304 52 0.34341 1.03465
362 16 5604 5588 61.7367 64 0.550665 1.0351
363 16 5618 5602 61.7208 56 0.455966 1.03539
364 16 5636 5620 61.7491 72 0.998482 1.03507
365 16 5653 5637 61.7662 68 1.0843 1.03458
366 16 5671 5655 61.7941 72 0.950376 1.03467
367 16 5688 5672 61.811 68 0.593894 1.03392
368 16 5704 5688 61.8169 64 1.31227 1.03378
369 16 5718 5702 61.8011 56 0.293306 1.03356
370 16 5734 5718 61.8071 64 0.63192 1.0345
371 16 5750 5734 61.8129 64 1.27207 1.03411
372 16 5761 5745 61.7651 44 1.99774 1.03409
373 16 5776 5760 61.7603 60 2.19982 1.03442
374 16 5794 5778 61.7877 72 0.535765 1.03488
375 16 5808 5792 61.7722 56 1.44843 1.03473
376 16 5828 5812 61.8207 80 0.336351 1.034
377 16 5843 5827 61.8158 60 0.390151 1.03366
378 16 5862 5846 61.8533 76 1.31242 1.03316
379 16 5875 5859 61.8273 52 0.82727 1.03291
2020-02-09 13:09:28.038370 min lat: 0.205941 max lat: 3.24391 avg lat: 1.03359
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 5890 5874 61.8225 60 0.632804 1.03359
381 16 5902 5886 61.7862 48 0.483065 1.03344
382 16 5912 5896 61.7291 40 2.1345 1.03466
383 16 5930 5914 61.7559 72 0.800936 1.03515
384 16 5947 5931 61.7721 68 1.24939 1.03471
385 16 5966 5950 61.8091 76 0.334738 1.03429
386 16 5978 5962 61.7733 48 1.24451 1.03434
387 16 5991 5975 61.748 52 1.20561 1.03431
388 16 6008 5992 61.7641 68 0.575021 1.03441
389 16 6023 6007 61.7596 60 1.43968 1.03453
390 16 6041 6025 61.7858 72 1.52231 1.03519
391 16 6050 6034 61.7196 36 0.943453 1.03478
392 16 6067 6051 61.7356 68 1.2327 1.03546
393 16 6081 6065 61.721 56 0.251913 1.03533
394 16 6098 6082 61.737 68 0.693522 1.03495
395 16 6114 6098 61.7427 64 0.297509 1.03494
396 16 6127 6111 61.718 52 0.35496 1.035
397 16 6143 6127 61.7238 64 0.465007 1.03474
398 16 6157 6141 61.7094 56 0.602841 1.03455
399 16 6176 6160 61.7452 76 0.67921 1.03493
2020-02-09 13:09:48.041949 min lat: 0.205941 max lat: 3.24391 avg lat: 1.03464
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 6194 6178 61.7708 72 0.55027 1.03464
401 16 6211 6195 61.7863 68 0.628714 1.03428
402 16 6230 6214 61.8216 76 0.96338 1.0342
403 16 6249 6233 61.8568 76 1.15239 1.03366
404 16 6266 6250 61.872 68 1.11788 1.03303
405 16 6283 6267 61.8871 68 1.0447 1.03279
406 16 6297 6281 61.8726 56 1.56996 1.03256
407 16 6316 6300 61.9073 76 1.51083 1.03228
408 16 6326 6310 61.8536 40 0.585202 1.03191
409 16 6343 6327 61.8686 68 0.289246 1.03288
410 16 6360 6344 61.8835 68 0.79128 1.03246
411 16 6379 6363 61.9178 76 0.370006 1.03258
412 16 6394 6378 61.9132 60 1.04894 1.03244
413 16 6408 6392 61.8988 56 1.21645 1.03262
414 16 6420 6404 61.8652 48 1.51116 1.03268
415 16 6432 6416 61.8318 48 1.8108 1.03353
416 16 6444 6428 61.7985 48 1.35205 1.03379
417 16 6462 6446 61.823 72 0.477198 1.03392
418 16 6481 6465 61.8569 76 0.281612 1.03334
419 16 6496 6480 61.8524 60 0.6118 1.03339
2020-02-09 13:10:08.044431 min lat: 0.205941 max lat: 3.24391 avg lat: 1.03394
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 6513 6497 61.867 68 1.10869 1.03394
421 16 6521 6505 61.7961 32 0.59 1.03379
422 16 6537 6521 61.8013 64 0.531809 1.03386
423 16 6550 6534 61.7781 52 1.64083 1.03413
424 16 6567 6551 61.7928 68 0.960975 1.03447
425 16 6579 6563 61.7603 48 0.871078 1.0346
426 16 6597 6581 61.7843 72 0.488322 1.03464
427 16 6615 6599 61.8082 72 0.54849 1.03441
428 16 6626 6610 61.7666 44 0.949446 1.0342
429 16 6652 6636 61.865 104 0.375506 1.03365
430 16 6667 6651 61.8607 60 0.944868 1.03333
431 16 6689 6673 61.9213 88 0.790143 1.03269
432 16 6705 6689 61.9261 64 1.20592 1.03216
433 16 6718 6702 61.9031 52 1.29116 1.03226
434 16 6726 6710 61.8342 32 0.625513 1.03219
435 16 6743 6727 61.8484 68 0.967439 1.03313
436 16 6764 6748 61.8992 84 0.868599 1.03273
437 16 6777 6761 61.8765 52 0.967096 1.03258
438 16 6794 6778 61.8905 68 0.365691 1.03241
439 16 6811 6795 61.9044 68 1.16796 1.03228
2020-02-09 13:10:28.046795 min lat: 0.205941 max lat: 3.24391 avg lat: 1.03218
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 6832 6816 61.9546 84 0.596438 1.03218
441 16 6846 6830 61.941 56 0.497536 1.03174
442 16 6861 6845 61.9366 60 1.31282 1.03149
443 16 6872 6856 61.8961 44 1.73188 1.03211
444 16 6885 6869 61.8738 52 1.89426 1.03242
445 16 6897 6881 61.8426 48 1.24249 1.03348
446 16 6913 6897 61.8475 64 1.13488 1.03326
447 16 6921 6905 61.7807 32 1.74518 1.03368
448 16 6934 6918 61.7588 52 1.59823 1.03522
449 16 6951 6935 61.7727 68 0.545009 1.03464
450 16 6971 6955 61.8132 80 1.20922 1.03434
451 16 6989 6973 61.8358 72 0.573685 1.03393
452 16 7005 6989 61.8406 64 0.873935 1.03362
453 16 7012 6996 61.7658 28 0.417151 1.03337
454 16 7012 6996 61.6298 0 - 1.03337
455 16 7012 6996 61.4944 0 - 1.03337
456 16 7012 6996 61.3595 0 - 1.03337
457 16 7028 7012 61.3653 16 3.907 1.04027
458 16 7043 7027 61.3623 60 0.793008 1.04172
459 16 7055 7039 61.3331 48 1.4537 1.04231
2020-02-09 13:10:48.049082 min lat: 0.205941 max lat: 5.63799 avg lat: 1.0424
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 7064 7048 61.2781 36 2.31373 1.0424
461 16 7080 7064 61.2839 64 0.371133 1.0431
462 16 7085 7069 61.1946 20 2.07235 1.04333
463 16 7104 7088 61.2265 76 0.718602 1.04417
464 16 7114 7098 61.1808 40 0.907796 1.04395
465 16 7130 7114 61.1868 64 1.7112 1.04449
466 16 7142 7126 61.1585 48 2.52743 1.04489
467 16 7154 7138 61.1303 48 0.390537 1.04527
468 16 7164 7148 61.0852 40 1.80031 1.04572
469 16 7182 7166 61.1084 72 1.44167 1.04643
470 16 7195 7179 61.089 52 0.828284 1.04628
471 16 7205 7189 61.0442 40 0.867002 1.04642
472 16 7209 7193 60.9488 16 0.867754 1.04633
473 16 7211 7195 60.8369 8 3.20431 1.04694
474 16 7224 7208 60.8182 52 2.64505 1.04986
475 16 7239 7223 60.8165 60 1.38929 1.05108
476 16 7256 7240 60.8315 68 1.11787 1.05092
477 16 7271 7255 60.8298 60 1.21253 1.05097
478 16 7287 7271 60.8364 64 0.576509 1.05062
479 16 7303 7287 60.843 64 0.922556 1.05058
2020-02-09 13:11:08.051491 min lat: 0.205941 max lat: 5.63799 avg lat: 1.05043
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 7321 7305 60.8662 72 0.953901 1.05043
481 16 7331 7315 60.8228 40 0.238526 1.04998
482 16 7348 7332 60.8377 68 0.715733 1.05086
483 16 7363 7347 60.836 60 0.621127 1.05049
484 16 7380 7364 60.8507 68 1.41647 1.05051
485 16 7385 7369 60.7665 20 1.03776 1.05071
486 16 7391 7375 60.6909 24 0.323356 1.05092
487 16 7392 7376 60.5744 4 3.43949 1.05125
488 16 7406 7390 60.5651 56 3.41944 1.0548
489 16 7416 7400 60.523 40 1.59742 1.0557
490 16 7427 7411 60.4893 44 1.65248 1.05627
491 16 7442 7426 60.4882 60 0.544389 1.0571
492 16 7455 7439 60.471 52 0.390467 1.057
493 16 7468 7452 60.4538 52 0.339658 1.05667
494 16 7487 7471 60.4852 76 0.309148 1.05714
495 16 7505 7489 60.5085 72 1.154 1.05704
496 16 7523 7507 60.5316 72 1.08829 1.0564
497 16 7534 7518 60.4984 44 0.507418 1.05593
498 16 7540 7524 60.4251 24 0.285469 1.05536
499 16 7550 7534 60.3841 40 0.427295 1.05712
2020-02-09 13:11:28.054004 min lat: 0.205941 max lat: 5.63799 avg lat: 1.05812
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 7570 7554 60.4233 80 1.57423 1.05812
501 16 7579 7563 60.3746 36 1.37468 1.05814
502 16 7599 7583 60.4136 80 0.388659 1.05825
503 16 7620 7604 60.4605 84 1.21426 1.05765
504 16 7633 7617 60.4437 52 0.737158 1.05731
505 16 7650 7634 60.4587 68 0.467067 1.05726
506 16 7667 7651 60.4735 68 1.2342 1.05714
507 16 7677 7661 60.4331 40 1.277 1.05744
508 16 7689 7673 60.4087 48 1.52622 1.05781
509 16 7708 7692 60.4393 76 0.328656 1.05804
510 16 7725 7709 60.4541 68 0.787158 1.0575
511 16 7734 7718 60.4062 36 0.637417 1.05738
512 16 7747 7731 60.3898 52 0.589714 1.05757
513 16 7763 7747 60.3968 64 1.94518 1.05817
514 16 7786 7770 60.4583 92 0.453422 1.0579
515 16 7798 7782 60.4341 48 0.310308 1.05757
516 16 7817 7801 60.4642 76 0.739695 1.05725
517 16 7837 7821 60.502 80 0.880167 1.05674
518 16 7855 7839 60.5242 72 0.367912 1.0567
519 16 7872 7856 60.5386 68 1.26291 1.0563
2020-02-09 13:11:48.056526 min lat: 0.205941 max lat: 5.63799 avg lat: 1.05603
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 7883 7867 60.5067 44 0.707214 1.05603
521 16 7901 7885 60.5288 72 1.37964 1.05634
522 16 7917 7901 60.5354 64 0.326238 1.05644
523 16 7930 7914 60.5191 52 0.867512 1.05616
524 16 7951 7935 60.5639 84 0.30196 1.05572
525 16 7967 7951 60.5704 64 1.27139 1.05564
526 16 7987 7971 60.6073 80 1.01489 1.05524
527 16 8002 7986 60.6062 60 0.511636 1.05469
528 16 8013 7997 60.5747 44 0.873698 1.05473
529 16 8029 8013 60.5812 64 1.31828 1.05529
530 16 8031 8015 60.482 8 0.695452 1.0553
531 16 8056 8040 60.5564 100 0.938113 1.05621
532 16 8066 8050 60.5177 40 0.842141 1.05611
533 16 8081 8065 60.5167 60 1.50015 1.05665
534 16 8093 8077 60.4933 48 0.371377 1.05673
535 16 8111 8095 60.5148 72 1.20225 1.05663
536 16 8123 8107 60.4914 48 0.377575 1.05652
537 16 8141 8125 60.5128 72 0.27147 1.05627
538 16 8156 8140 60.5119 60 1.15635 1.05645
539 16 8174 8158 60.5332 72 0.572036 1.05609
2020-02-09 13:12:08.058892 min lat: 0.205941 max lat: 5.63799 avg lat: 1.05573
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 8192 8176 60.5544 72 1.07973 1.05573
541 16 8206 8190 60.5459 56 1.43701 1.05559
542 16 8215 8199 60.5006 36 1.3601 1.05605
543 16 8235 8219 60.5365 80 0.602381 1.05592
544 16 8255 8239 60.5723 80 0.934667 1.05592
545 16 8269 8253 60.5639 56 0.493145 1.05555
546 16 8285 8269 60.5702 64 0.593794 1.05541
547 16 8299 8283 60.5618 56 0.736539 1.05568
548 16 8314 8298 60.5608 60 0.49003 1.0555
549 16 8332 8316 60.5816 72 1.57046 1.05556
550 16 8337 8321 60.5078 20 0.915039 1.0555
551 16 8353 8337 60.5141 64 1.79984 1.05619
552 16 8363 8347 60.477 40 1.85301 1.05661
553 16 8373 8357 60.4399 40 0.383462 1.05652
554 16 8389 8373 60.4463 64 0.475581 1.05718
555 16 8402 8386 60.4311 52 1.40516 1.05802
556 16 8417 8401 60.4303 60 0.892883 1.05786
557 16 8437 8421 60.4654 80 0.789933 1.05754
558 16 8448 8432 60.4359 44 1.91718 1.05759
559 16 8462 8446 60.428 56 0.837368 1.05799
2020-02-09 13:12:28.061174 min lat: 0.205941 max lat: 5.63799 avg lat: 1.0572
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 8484 8468 60.4772 88 0.273061 1.0572
561 16 8497 8481 60.4621 52 1.58961 1.05717
562 16 8509 8493 60.4399 48 1.29513 1.05744
563 16 8526 8510 60.4533 68 1.16795 1.05767
564 16 8542 8526 60.4595 64 0.499569 1.05746
565 16 8559 8543 60.4729 68 1.53187 1.05725
566 16 8575 8559 60.4791 64 1.36968 1.05724
567 16 8592 8576 60.4923 68 0.385573 1.05697
568 16 8614 8598 60.5407 88 0.658797 1.05656
569 16 8626 8610 60.5187 48 0.461638 1.05637
570 16 8641 8625 60.5177 60 1.32336 1.05649
571 16 8659 8643 60.5378 72 0.60318 1.05626
572 16 8678 8662 60.5648 76 1.04481 1.05568
573 16 8697 8681 60.5917 76 1.22821 1.05536
574 16 8713 8697 60.5976 64 1.27147 1.05506
575 16 8725 8709 60.5757 48 1.68361 1.05526
576 16 8743 8727 60.5955 72 0.549064 1.05523
577 16 8763 8747 60.6289 80 0.837017 1.05506
578 16 8775 8759 60.607 48 0.276577 1.05475
579 16 8782 8766 60.5507 28 1.85951 1.05469
2020-02-09 13:12:48.067492 min lat: 0.205941 max lat: 5.63799 avg lat: 1.05548
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 8796 8780 60.5428 56 2.12867 1.05548
581 16 8808 8792 60.5212 48 1.83074 1.05589
582 16 8824 8808 60.5272 64 0.703301 1.05628
583 16 8841 8825 60.54 68 1.34371 1.05618
584 16 8856 8840 60.5391 60 0.358973 1.05592
585 16 8871 8855 60.5381 60 0.496655 1.05595
586 16 8884 8868 60.5235 52 1.37576 1.05582
587 16 8893 8877 60.4818 36 1.74324 1.05644
588 16 8901 8885 60.4333 32 2.31317 1.05685
589 16 8917 8901 60.4394 64 2.58398 1.0575
590 16 8931 8915 60.4318 56 0.314227 1.05761
591 16 8944 8928 60.4176 52 0.336832 1.05789
592 16 8962 8946 60.4371 72 1.03321 1.058
593 16 8982 8966 60.4701 80 0.674892 1.05755
594 16 8997 8981 60.4693 60 0.546432 1.05692
595 16 9006 8990 60.4281 36 2.02345 1.05733
596 16 9024 9008 60.4475 72 1.52289 1.05789
597 16 9034 9018 60.4133 40 0.559424 1.0579
598 16 9044 9028 60.3791 40 0.295894 1.05781
599 16 9059 9043 60.3785 60 1.34412 1.05868
2020-02-09 13:13:08.069836 min lat: 0.205941 max lat: 5.63799 avg lat: 1.05868
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 16 9074 9058 60.3779 60 1.33099 1.05868
Total time run: 600.902758
Total writes made: 9075
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 60.4091
Stddev Bandwidth: 16.1433
Max bandwidth (MB/sec): 104
Min bandwidth (MB/sec): 0
Average IOPS: 15
Stddev IOPS: 4
Max IOPS: 26
Min IOPS: 0
Average Latency(s): 1.05936
Stddev Latency(s): 0.528235
Max latency(s): 5.63799
Min latency(s): 0.205941
Sequential Read Test
Total time run: 243.001923
Total reads made: 9075
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 149.382
Average IOPS: 37
Stddev IOPS: 7
Max IOPS: 57
Min IOPS: 14
Average Latency(s): 0.427557
Max latency(s): 3.33833
Min latency(s): 0.0524101
rados bench -p testbench 600 seq | tee --append $(date +%F)-rados-test-read-seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 16 16 0 0 0 - 0
1 16 60 44 175.847 176 0.0775366 0.266531
2 16 95 79 157.914 140 0.577033 0.290436
3 16 131 115 153.266 144 0.053747 0.344753
4 16 161 145 144.947 120 0.0933719 0.373127
5 16 200 184 147.152 156 0.0804509 0.38342
6 16 235 219 145.957 140 1.06459 0.401643
7 16 287 271 154.808 208 0.259545 0.402002
8 16 335 319 159.453 192 0.7613 0.387464
9 16 381 365 162.177 184 0.0532609 0.380072
10 16 424 408 163.157 172 1.05304 0.38128
11 16 478 462 167.958 216 0.0669362 0.369206
12 16 520 504 167.96 168 0.0750025 0.365122
13 16 547 531 163.347 108 0.180901 0.366318
14 16 585 569 162.535 152 0.273444 0.370124
15 16 625 609 162.365 160 1.14715 0.381174
16 16 669 653 163.216 176 1.51887 0.379999
17 16 707 691 162.555 152 0.162649 0.380454
18 16 747 731 162.411 160 0.133613 0.383509
19 16 778 762 160.389 124 1.23666 0.38894
2020-02-09 13:19:55.326900 min lat: 0.0532609 max lat: 2.40014 avg lat: 0.390717
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 816 800 159.969 152 0.381071 0.390717
21 16 864 848 161.491 192 0.145493 0.387432
22 16 894 878 159.605 120 0.25385 0.390173
23 16 926 910 158.23 128 0.295835 0.390703
24 16 960 944 157.303 136 0.0815618 0.394341
25 16 994 978 156.451 136 2.38637 0.397148
26 16 1021 1005 154.586 108 0.0533196 0.400685
27 16 1055 1039 153.897 136 2.06706 0.404393
28 16 1101 1085 154.972 184 0.12265 0.406813
29 16 1129 1113 153.489 112 1.63875 0.408063
30 16 1159 1143 152.373 120 0.134224 0.410572
31 16 1209 1193 153.908 200 0.200264 0.410413
32 16 1243 1227 153.348 136 0.305073 0.409512
33 16 1267 1251 151.61 96 1.06397 0.410404
34 16 1306 1290 151.738 156 0.635821 0.413746
35 16 1349 1333 152.316 172 0.265106 0.414041
36 16 1395 1379 153.195 184 0.169535 0.413877
37 16 1440 1424 153.919 180 0.198539 0.411154
38 16 1476 1460 153.657 144 0.468223 0.408878
39 16 1504 1488 152.589 112 0.564755 0.410133
2020-02-09 13:20:15.329868 min lat: 0.0532609 max lat: 2.63311 avg lat: 0.413358
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1540 1524 152.374 144 0.217399 0.413358
41 16 1573 1557 151.877 132 0.263172 0.413827
42 16 1605 1589 151.305 128 0.179301 0.414932
43 16 1628 1612 149.926 92 0.171398 0.417736
44 16 1661 1645 149.518 132 0.336689 0.419903
45 16 1708 1692 150.371 188 0.145116 0.420488
46 16 1742 1726 150.058 136 0.598588 0.419873
47 16 1779 1763 150.014 148 0.285508 0.421714
48 16 1820 1804 150.305 164 0.054415 0.421285
49 16 1855 1839 150.094 140 1.96416 0.420845
50 16 1912 1896 151.652 228 0.148299 0.419428
51 16 1951 1935 151.734 156 0.173392 0.417619
52 16 1981 1965 151.124 120 0.194782 0.418048
53 16 2017 2001 150.989 144 2.05148 0.418404
54 16 2049 2033 150.563 128 0.620795 0.42025
55 16 2096 2080 151.243 188 0.236774 0.419035
56 16 2137 2121 151.47 164 0.549282 0.419392
57 16 2175 2159 151.48 152 0.128907 0.419475
58 16 2195 2179 150.247 80 0.506913 0.420962
59 16 2227 2211 149.87 128 0.59782 0.420952
2020-02-09 13:20:35.334493 min lat: 0.0531553 max lat: 2.63311 avg lat: 0.422538
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 2270 2254 150.238 172 0.105766 0.422538
61 16 2323 2307 151.25 212 0.246775 0.420618
62 16 2362 2346 151.326 156 0.127161 0.419163
63 16 2393 2377 150.892 124 0.0817976 0.419073
64 16 2433 2417 151.034 160 0.120645 0.419084
65 16 2473 2457 151.171 160 0.209661 0.419329
66 16 2509 2493 151.062 144 0.267626 0.420065
67 16 2548 2532 151.135 156 0.141572 0.419748
68 16 2581 2565 150.852 132 1.36026 0.420943
69 16 2612 2596 150.462 124 0.188965 0.421119
70 16 2649 2633 150.427 148 1.85529 0.421967
71 16 2689 2673 150.561 160 1.51653 0.421757
72 16 2729 2713 150.692 160 0.360682 0.421859
73 16 2759 2743 150.272 120 1.85738 0.421937
74 16 2785 2769 149.646 104 0.316697 0.422557
75 16 2824 2808 149.731 156 0.259173 0.423537
76 16 2860 2844 149.655 144 0.373676 0.424584
77 16 2892 2876 149.374 128 0.16918 0.424612
78 16 2919 2903 148.843 108 0.348421 0.425454
79 16 2963 2947 149.187 176 0.283136 0.425862
2020-02-09 13:20:55.338275 min lat: 0.0531553 max lat: 2.63311 avg lat: 0.42543
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 3007 2991 149.521 176 0.204953 0.42543
81 16 3049 3033 149.749 168 0.231071 0.424879
82 16 3088 3072 149.825 156 0.500131 0.423616
83 16 3123 3107 149.707 140 0.288267 0.42383
84 16 3160 3144 149.686 148 1.97354 0.42396
85 16 3199 3183 149.76 156 0.372268 0.423324
86 16 3244 3228 150.111 180 0.405586 0.422984
87 16 3293 3277 150.639 196 0.157563 0.422653
88 16 3331 3315 150.654 152 0.123038 0.422012
89 16 3363 3347 150.399 128 1.09442 0.422538
90 16 3388 3372 149.839 100 0.496452 0.423152
91 16 3425 3409 149.819 148 0.760818 0.425024
92 16 3468 3452 150.06 172 0.138161 0.424631
93 16 3509 3493 150.209 164 0.160674 0.424151
94 16 3558 3542 150.696 196 0.0738909 0.423001
95 16 3592 3576 150.535 136 0.953022 0.422387
96 16 3627 3611 150.423 140 0.205762 0.422503
97 16 3663 3647 150.357 144 0.273836 0.423229
98 16 3698 3682 150.251 140 0.858472 0.423501
99 16 3750 3734 150.834 208 0.127689 0.421821
2020-02-09 13:21:15.345916 min lat: 0.0531553 max lat: 2.63311 avg lat: 0.421694
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3788 3772 150.845 152 0.0752186 0.421694
101 16 3829 3813 150.975 164 0.145111 0.421627
102 16 3872 3856 151.176 172 0.320933 0.42083
103 16 3929 3913 151.922 228 0.792309 0.419872
104 16 3969 3953 151.999 160 0.154354 0.419512
105 16 4006 3990 151.961 148 0.7042 0.419115
106 16 4034 4018 151.584 112 1.379 0.419524
107 16 4062 4046 151.214 112 2.03765 0.419363
108 16 4084 4068 150.629 88 0.679098 0.420921
109 16 4125 4109 150.751 164 0.114924 0.421896
110 16 4161 4145 150.69 144 0.888137 0.422524
111 16 4194 4178 150.519 132 0.511809 0.422612
112 16 4226 4210 150.318 128 1.89515 0.423278
113 16 4272 4256 150.616 184 0.323965 0.423004
114 16 4302 4286 150.347 120 1.68971 0.422566
115 16 4332 4316 150.083 120 0.10206 0.422606
116 16 4358 4342 149.686 104 0.106256 0.422926
117 16 4396 4380 149.705 152 0.123916 0.423597
118 16 4420 4404 149.25 96 0.149496 0.42524
119 16 4458 4442 149.273 152 0.261799 0.425465
2020-02-09 13:21:35.353246 min lat: 0.0531303 max lat: 3.30881 avg lat: 0.426789
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 4492 4476 149.162 136 1.75688 0.426789
121 16 4529 4513 149.153 148 0.0788137 0.426728
122 16 4567 4551 149.176 152 0.129649 0.426331
123 16 4609 4593 149.329 168 0.251372 0.426463
124 16 4654 4638 149.576 180 0.116999 0.426195
125 16 4704 4688 149.979 200 1.19288 0.425084
126 16 4760 4744 150.566 224 0.359976 0.42397
127 16 4800 4784 150.64 160 0.151588 0.423336
128 16 4845 4829 150.869 180 0.148456 0.422336
129 16 4877 4861 150.692 128 0.666148 0.422662
130 16 4913 4897 150.64 144 0.135793 0.422667
131 16 4947 4931 150.528 136 0.264926 0.423104
132 16 4972 4956 150.146 100 1.55976 0.423592
133 16 5001 4985 149.889 116 0.198971 0.423892
134 16 5033 5017 149.725 128 0.0874727 0.424883
135 16 5063 5047 149.505 120 0.162228 0.425276
136 16 5093 5077 149.288 120 0.1525 0.426292
137 16 5128 5112 149.22 140 0.179021 0.425921
138 16 5168 5152 149.298 160 0.0896286 0.426577
139 16 5203 5187 149.231 140 0.11625 0.426265
2020-02-09 13:21:55.355939 min lat: 0.0531303 max lat: 3.30881 avg lat: 0.426662
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 5230 5214 148.936 108 0.137863 0.426662
141 16 5261 5245 148.759 124 0.0851853 0.427245
142 16 5296 5280 148.698 140 0.215745 0.428069
143 16 5329 5313 148.581 132 0.0771022 0.427788
144 16 5364 5348 148.521 140 0.0797474 0.42885
145 16 5395 5379 148.352 124 2.16831 0.428771
146 16 5433 5417 148.377 152 0.541375 0.428757
147 16 5464 5448 148.211 124 0.184271 0.428186
148 16 5478 5462 147.588 56 0.17725 0.430023
149 16 5508 5492 147.403 120 0.160663 0.431688
150 16 5561 5545 147.833 212 0.198937 0.431373
151 16 5598 5582 147.834 148 0.75565 0.431367
152 16 5638 5622 147.914 160 0.0535967 0.431154
153 16 5687 5671 148.228 196 0.391958 0.430557
154 16 5728 5712 148.33 164 0.179783 0.429416
155 16 5763 5747 148.276 140 0.0885647 0.429004
156 16 5795 5779 148.146 128 0.167609 0.429838
157 16 5829 5813 148.069 136 0.315391 0.430178
158 16 5876 5860 148.321 188 0.0664483 0.430024
159 16 5915 5899 148.37 156 0.176451 0.429568
2020-02-09 13:22:15.358399 min lat: 0.0531303 max lat: 3.33833 avg lat: 0.429197
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 5951 5935 148.342 144 1.7685 0.429197
161 16 5988 5972 148.34 148 1.61828 0.429532
162 16 6025 6009 148.338 148 0.0536185 0.429621
163 16 6070 6054 148.532 180 0.253061 0.429276
164 16 6124 6108 148.943 216 0.0534622 0.428329
165 16 6168 6152 149.107 176 0.707872 0.42752
166 16 6204 6188 149.076 144 0.313464 0.42766
167 16 6257 6241 149.452 212 0.0583883 0.427309
168 16 6295 6279 149.468 152 0.077126 0.427123
169 16 6349 6333 149.861 216 0.230346 0.425888
170 16 6393 6377 150.015 176 0.437854 0.425142
171 16 6442 6426 150.283 196 0.203009 0.424346
172 16 6477 6461 150.223 140 1.80927 0.424414
173 16 6515 6499 150.234 152 0.128883 0.424008
174 16 6549 6533 150.152 136 0.665493 0.424011
175 16 6589 6573 150.208 160 0.369871 0.424103
176 16 6616 6600 149.968 108 1.82216 0.425042
177 16 6650 6634 149.889 136 1.13681 0.425122
178 16 6686 6670 149.856 144 0.366945 0.425318
179 16 6727 6711 149.935 164 0.264844 0.425365
2020-02-09 13:22:35.360916 min lat: 0.0531303 max lat: 3.33833 avg lat: 0.425429
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 6764 6748 149.924 148 1.2884 0.425429
181 16 6807 6791 150.046 172 0.227826 0.42546
182 16 6851 6835 150.188 176 0.989765 0.424759
183 16 6889 6873 150.198 152 0.153524 0.424457
184 16 6927 6911 150.208 152 1.72376 0.424285
185 16 6970 6954 150.325 172 1.43522 0.424324
186 16 7009 6993 150.356 156 0.117665 0.423928
187 16 7045 7029 150.321 144 1.39232 0.424155
188 16 7085 7069 150.372 160 0.18541 0.423924
189 16 7129 7113 150.508 176 0.400542 0.423455
190 16 7158 7142 150.326 116 1.81757 0.423803
191 16 7200 7184 150.419 168 2.155 0.423877
192 16 7239 7223 150.447 156 0.241659 0.423825
193 16 7280 7264 150.517 164 0.227747 0.423551
194 16 7330 7314 150.772 200 0.12798 0.423205
195 16 7370 7354 150.819 160 0.289021 0.42322
196 16 7402 7386 150.703 128 0.205269 0.422886
197 16 7429 7413 150.486 108 0.367656 0.423419
198 16 7466 7450 150.473 148 0.373544 0.423394
199 16 7506 7490 150.521 160 0.104452 0.423617
2020-02-09 13:22:55.365401 min lat: 0.0531303 max lat: 3.33833 avg lat: 0.423257
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 7553 7537 150.708 188 0.147034 0.423257
201 16 7608 7592 151.053 220 1.50131 0.422462
202 16 7664 7648 151.414 224 0.158317 0.421623
203 16 7710 7694 151.574 184 0.0540003 0.420806
204 16 7745 7729 151.517 140 0.185686 0.420671
205 16 7784 7768 151.539 156 0.264479 0.42058
206 16 7840 7824 151.891 224 0.328503 0.420301
207 16 7872 7856 151.775 128 0.200534 0.420101
208 16 7902 7886 151.622 120 0.210371 0.420415
209 16 7935 7919 151.528 132 1.79975 0.420584
210 16 7959 7943 151.264 96 0.162232 0.420895
211 16 7998 7982 151.286 156 0.455472 0.421297
212 16 8037 8021 151.308 156 2.19187 0.421391
213 16 8075 8059 151.312 152 0.141791 0.42122
214 16 8117 8101 151.39 168 0.123355 0.420982
215 16 8156 8140 151.411 156 1.94551 0.421002
216 16 8202 8186 151.562 184 0.399453 0.420992
217 16 8251 8235 151.766 196 0.251485 0.420436
218 16 8303 8287 152.024 208 0.159079 0.419871
219 16 8347 8331 152.134 176 0.0818713 0.419449
2020-02-09 13:23:15.367578 min lat: 0.0531303 max lat: 3.33833 avg lat: 0.419242
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 8377 8361 151.987 120 0.224291 0.419242
221 16 8404 8388 151.788 108 0.554626 0.419395
222 16 8431 8415 151.591 108 0.0530849 0.42018
223 16 8464 8448 151.503 132 0.147943 0.4205
224 16 8491 8475 151.309 108 0.109361 0.4212
225 16 8512 8496 151.01 84 1.79258 0.421941
226 16 8538 8522 150.802 104 0.144007 0.422318
227 16 8565 8549 150.613 108 0.103389 0.422579
228 16 8593 8577 150.444 112 0.12866 0.423409
229 16 8625 8609 150.346 128 0.087667 0.42361
230 16 8648 8632 150.092 92 0.381609 0.42446
231 16 8683 8667 150.048 140 0.11909 0.424701
232 16 8721 8705 150.056 152 0.105297 0.424458
233 16 8752 8736 149.944 124 0.140606 0.42491
234 16 8792 8776 149.987 160 0.216735 0.425044
235 16 8836 8820 150.098 176 0.0852299 0.424983
236 16 8869 8853 150.021 132 0.180573 0.424986
237 16 8907 8891 150.03 152 2.16582 0.424848
238 16 8941 8925 149.971 136 0.114504 0.42462
239 16 8969 8953 149.812 112 0.380402 0.424974
2020-02-09 13:23:35.369986 min lat: 0.0524101 max lat: 3.33833 avg lat: 0.425338
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 9006 8990 149.804 148 2.44401 0.425338
241 16 9037 9021 149.697 124 0.188784 0.425254
242 16 9075 9059 149.706 152 0.241629 0.42553
Total time run: 243.001923
Total reads made: 9075
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 149.382
Average IOPS: 37
Stddev IOPS: 7
Max IOPS: 57
Min IOPS: 14
Average Latency(s): 0.427557
Max latency(s): 3.33833
Min latency(s): 0.0524101
Random Read Test
Total time run: 601.086899
Total reads made: 22341
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 148.671
Average IOPS: 37
Stddev IOPS: 7
Max IOPS: 58
Min IOPS: 16
Average Latency(s): 0.429814
Max latency(s): 4.37235
Min latency(s): 0.0371554
rados bench -p testbench 600 rand | tee --append $(date +%F)-rados-test-read-rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 16 16 0 0 0 - 0
1 16 60 44 175.858 176 0.158665 0.267732
2 16 98 82 163.923 152 0.288999 0.316885
3 16 124 108 143.949 104 0.943632 0.352532
4 16 162 146 145.957 152 0.350461 0.37387
5 16 202 186 148.761 160 0.508248 0.382613
6 16 234 218 145.157 128 1.36778 0.398673
7 16 266 250 142.696 128 0.203283 0.401364
8 16 315 299 149.35 196 0.161564 0.401738
9 16 343 327 145.202 112 0.312733 0.406985
10 16 371 355 141.883 112 0.158043 0.416767
11 16 410 394 143.164 156 0.172161 0.420876
12 16 455 439 146.23 180 0.121291 0.420175
13 16 478 462 142.06 92 0.170826 0.428056
14 16 503 487 139.056 100 0.164159 0.436489
15 16 546 530 141.25 172 0.306101 0.437643
16 16 590 574 143.42 176 0.0792493 0.434023
17 16 627 611 143.688 148 0.0807894 0.435228
18 16 672 656 145.703 180 0.277246 0.428428
19 16 717 701 147.507 180 0.452415 0.421986
2020-02-09 13:29:03.131281 min lat: 0.0376865 max lat: 2.61661 avg lat: 0.421167
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 757 741 148.13 160 0.183407 0.421167
21 16 787 771 146.732 120 0.126201 0.420784
22 16 826 810 147.152 156 0.0730665 0.424724
23 16 867 851 147.883 164 0.809074 0.427656
24 16 908 892 148.553 164 0.0804222 0.424356
25 16 953 937 149.81 180 0.811713 0.420847
26 16 987 971 149.278 136 0.135855 0.422013
27 16 1027 1011 149.674 160 0.0787468 0.421993
28 16 1075 1059 151.184 192 0.770815 0.418425
29 16 1124 1108 152.728 196 1.15775 0.413927
30 16 1165 1149 153.103 164 1.52749 0.410547
31 16 1204 1188 153.196 156 0.0740682 0.410169
32 16 1243 1227 153.283 156 0.101785 0.410051
33 16 1281 1265 153.242 152 0.141857 0.410422
34 16 1326 1310 154.028 180 0.177054 0.409684
35 16 1365 1349 154.084 156 0.0819008 0.411697
36 16 1407 1391 154.467 168 0.124575 0.408415
37 16 1448 1432 154.724 164 0.277221 0.406714
38 16 1472 1456 153.179 96 0.115277 0.409442
39 16 1500 1484 152.123 112 0.730056 0.410855
2020-02-09 13:29:23.142961 min lat: 0.0376865 max lat: 2.61661 avg lat: 0.41539
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1532 1516 151.52 128 0.139827 0.41539
41 16 1576 1560 152.116 176 0.294529 0.415493
42 16 1624 1608 153.065 192 0.160342 0.41267
43 16 1653 1637 152.203 116 0.0773479 0.413331
44 16 1686 1670 151.744 132 2.2709 0.414801
45 16 1729 1713 152.193 172 0.419738 0.415394
46 16 1770 1754 152.45 164 0.107991 0.414208
47 16 1807 1791 152.354 148 0.260025 0.414528
48 16 1844 1828 152.263 148 0.119184 0.41535
49 16 1868 1852 151.113 96 2.15841 0.414248
50 16 1911 1895 151.53 172 0.334442 0.418284
51 16 1957 1941 152.166 184 0.322556 0.417423
52 16 1997 1981 152.317 160 0.117873 0.415896
53 16 2035 2019 152.31 152 0.129315 0.415912
54 16 2075 2059 152.452 160 0.143052 0.415983
55 16 2113 2097 152.443 152 0.857828 0.415232
56 16 2154 2138 152.649 164 0.0790984 0.415533
57 16 2180 2164 151.796 104 0.91278 0.416059
58 16 2201 2185 150.627 84 2.36744 0.418061
59 16 2224 2208 149.633 92 0.594693 0.420612
2020-02-09 13:29:43.146362 min lat: 0.0376865 max lat: 2.76688 avg lat: 0.421511
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 2254 2238 149.139 120 0.138713 0.421511
61 16 2292 2276 149.186 152 0.215504 0.422926
62 16 2318 2302 148.457 104 2.38942 0.42553
63 16 2353 2337 148.322 140 0.0976515 0.42575
64 16 2384 2368 147.942 124 0.0859998 0.42803
65 16 2420 2404 147.881 144 1.58816 0.428678
66 16 2450 2434 147.459 120 1.79534 0.429239
67 16 2494 2478 147.884 176 0.126388 0.428008
68 16 2534 2518 148.062 160 0.241649 0.427744
69 16 2581 2565 148.64 188 0.0893798 0.426862
70 16 2618 2602 148.631 148 0.154619 0.427007
71 16 2645 2629 148.059 108 0.194826 0.42797
72 16 2677 2661 147.78 128 0.666227 0.428323
73 16 2709 2693 147.509 128 1.73092 0.429047
74 16 2746 2730 147.515 148 0.133421 0.429745
75 16 2777 2761 147.202 124 2.1762 0.43091
76 16 2812 2796 147.107 140 1.5366 0.431482
77 16 2836 2820 146.443 96 0.158553 0.431817
78 16 2872 2856 146.411 144 0.223758 0.433251
79 16 2906 2890 146.279 136 0.316876 0.434334
2020-02-09 13:30:03.148796 min lat: 0.0376865 max lat: 3.3428 avg lat: 0.433943
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 2937 2921 146.001 124 0.967747 0.433943
81 16 2971 2955 145.877 136 0.106123 0.434621
82 16 3003 2987 145.659 128 1.65283 0.436448
83 16 3049 3033 146.121 184 0.127987 0.435355
84 16 3081 3065 145.905 128 0.313379 0.43569
85 16 3122 3106 146.117 164 0.188534 0.434673
86 16 3162 3146 146.279 160 0.816611 0.43517
87 16 3198 3182 146.252 144 0.109034 0.434223
88 16 3231 3215 146.09 132 0.116493 0.43513
89 16 3259 3243 145.707 112 2.10761 0.435495
90 16 3298 3282 145.821 156 1.53151 0.436008
91 16 3340 3324 146.065 168 0.169865 0.434881
92 16 3380 3364 146.216 160 0.138548 0.434978
93 16 3430 3414 146.793 200 0.123296 0.433745
94 16 3461 3445 146.551 124 1.67255 0.433966
95 16 3489 3473 146.187 112 0.0379765 0.434331
96 16 3523 3507 146.081 136 2.33516 0.433726
97 16 3565 3549 146.307 168 0.192651 0.434026
98 16 3602 3586 146.324 148 0.13436 0.434322
99 16 3637 3621 146.26 140 0.371054 0.434028
2020-02-09 13:30:23.151396 min lat: 0.0374512 max lat: 3.3428 avg lat: 0.434689
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3670 3654 146.117 132 2.05905 0.434689
101 16 3706 3690 146.096 144 0.182232 0.435588
102 16 3740 3724 145.997 136 0.167826 0.435741
103 16 3766 3750 145.585 104 1.0061 0.436095
104 16 3789 3773 145.07 92 0.176444 0.436826
105 16 3822 3806 144.945 132 0.0790594 0.436852
106 16 3863 3847 145.125 164 0.0379119 0.436696
107 16 3903 3887 145.264 160 0.328483 0.437669
108 16 3938 3922 145.215 140 0.0385383 0.437589
109 16 3969 3953 145.02 124 2.10143 0.438391
110 16 3996 3980 144.683 108 0.465435 0.438746
111 16 4022 4006 144.317 104 1.81195 0.440043
112 16 4044 4028 143.814 88 0.122725 0.440843
113 16 4082 4066 143.886 152 0.330636 0.440686
114 16 4118 4102 143.887 144 0.0806929 0.441327
115 16 4148 4132 143.679 120 0.151149 0.441818
116 16 4184 4168 143.682 144 0.080323 0.442375
117 16 4216 4200 143.548 128 0.0744687 0.442939
118 16 4259 4243 143.789 172 0.605825 0.44214
119 16 4307 4291 144.194 192 0.0746907 0.442066
2020-02-09 13:30:43.156331 min lat: 0.0374512 max lat: 3.3428 avg lat: 0.44203
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 4340 4324 144.092 132 1.44918 0.44203
121 16 4379 4363 144.19 156 1.27486 0.441935
122 16 4417 4401 144.254 152 0.142835 0.441548
123 16 4447 4431 144.057 120 0.0786897 0.441423
124 16 4473 4457 143.733 104 0.102837 0.441509
125 16 4506 4490 143.639 132 0.18636 0.442169
126 16 4554 4538 144.023 192 2.08191 0.443009
127 16 4592 4576 144.084 152 0.121165 0.441919
128 16 4636 4620 144.333 176 0.154682 0.441381
129 16 4672 4656 144.33 144 0.0383836 0.441397
130 16 4714 4698 144.512 168 0.124568 0.440771
131 16 4753 4737 144.6 156 0.482613 0.440308
132 16 4773 4757 144.11 80 0.354087 0.440609
133 16 4809 4793 144.109 144 0.257533 0.440989
134 16 4850 4834 144.257 164 0.126673 0.44116
135 16 4870 4854 143.781 80 1.63788 0.441457
136 16 4895 4879 143.459 100 0.0840143 0.442665
137 16 4932 4916 143.492 148 0.385376 0.443755
138 16 4988 4972 144.075 224 0.239204 0.442806
139 16 5038 5022 144.477 200 0.198079 0.441498
2020-02-09 13:31:03.160995 min lat: 0.0374512 max lat: 3.3428 avg lat: 0.440425
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 5083 5067 144.731 180 0.11063 0.440425
141 16 5122 5106 144.811 156 0.101885 0.439882
142 16 5164 5148 144.974 168 0.164648 0.439604
143 16 5197 5181 144.883 132 0.117255 0.439584
144 16 5236 5220 144.96 156 0.0808238 0.439155
145 16 5283 5267 145.257 188 0.122624 0.438969
146 16 5323 5307 145.358 160 0.224853 0.43844
147 16 5361 5345 145.403 152 0.0380491 0.438389
148 16 5396 5380 145.366 140 0.0818891 0.437947
149 16 5422 5406 145.088 104 0.0885778 0.437951
150 16 5455 5439 145.001 132 2.38527 0.438839
151 16 5482 5466 144.756 108 0.307687 0.439159
152 16 5519 5503 144.777 148 0.223955 0.439952
153 16 5557 5541 144.824 152 0.0383704 0.439732
154 16 5594 5578 144.845 148 0.213578 0.439746
155 16 5626 5610 144.736 128 1.9044 0.440315
156 16 5673 5657 145.013 188 0.445401 0.440056
157 16 5723 5707 145.362 200 0.0738036 0.43908
158 16 5767 5751 145.556 176 0.205701 0.438277
159 16 5804 5788 145.572 148 0.546832 0.438022
2020-02-09 13:31:23.164115 min lat: 0.0374512 max lat: 3.3428 avg lat: 0.438017
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 5842 5826 145.612 152 0.334605 0.438017
161 16 5886 5870 145.8 176 0.163778 0.437324
162 16 5941 5925 146.258 220 0.498852 0.436071
163 16 5993 5977 146.636 208 0.35404 0.435192
164 16 6041 6025 146.912 192 1.18478 0.434504
165 16 6090 6074 147.209 196 0.130566 0.43365
166 16 6135 6119 147.406 180 0.123261 0.432333
167 16 6176 6160 147.505 164 0.14226 0.431713
168 16 6211 6195 147.461 140 0.0720339 0.432034
169 16 6253 6237 147.582 168 0.140557 0.431869
170 16 6278 6262 147.302 100 0.655208 0.431866
171 16 6317 6301 147.352 156 0.185645 0.431953
172 16 6355 6339 147.379 152 2.38242 0.43247
173 16 6387 6371 147.267 128 0.323564 0.432333
174 16 6420 6404 147.179 132 0.1495 0.432383
175 16 6451 6435 147.047 124 0.472103 0.433516
176 16 6492 6476 147.143 164 0.243844 0.43325
177 16 6527 6511 147.102 140 0.0758257 0.432862
178 16 6558 6542 146.972 124 0.159484 0.433098
179 16 6597 6581 147.023 156 0.0995779 0.433892
2020-02-09 13:31:43.169046 min lat: 0.0373611 max lat: 3.3428 avg lat: 0.433515
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 6638 6622 147.117 164 1.35009 0.433515
181 16 6677 6661 147.166 156 0.151905 0.433101
182 16 6707 6691 147.017 120 0.866934 0.433229
183 16 6750 6734 147.153 172 0.198544 0.433153
184 16 6789 6773 147.201 156 0.0832541 0.433275
185 16 6833 6817 147.356 176 0.155824 0.432757
186 16 6873 6857 147.424 160 0.203863 0.432418
187 16 6919 6903 147.62 184 0.169655 0.432303
188 16 6960 6944 147.706 164 0.399398 0.431848
189 16 7001 6985 147.792 164 0.120394 0.43144
190 16 7045 7029 147.941 176 0.407866 0.430779
191 16 7079 7063 147.878 136 0.0792319 0.431384
192 16 7116 7100 147.878 148 0.387026 0.43102
193 16 7150 7134 147.817 136 0.296334 0.431241
194 16 7190 7174 147.88 160 0.119027 0.431118
195 16 7220 7204 147.736 120 0.336479 0.430837
196 16 7264 7248 147.881 176 0.121619 0.431127
197 16 7296 7280 147.78 128 0.0916305 0.431389
198 16 7330 7314 147.72 136 0.39415 0.431362
199 16 7376 7360 147.902 184 0.323069 0.431474
2020-02-09 13:32:03.172450 min lat: 0.0372422 max lat: 3.3428 avg lat: 0.431176
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 7420 7404 148.043 176 0.0489673 0.431176
201 16 7461 7445 148.122 164 0.273936 0.430824
202 16 7497 7481 148.101 144 0.206799 0.430592
203 16 7533 7517 148.081 144 0.154888 0.430434
204 16 7574 7558 148.159 164 0.166686 0.430103
205 16 7610 7594 148.139 144 0.135203 0.430414
206 16 7650 7634 148.196 160 1.9791 0.430173
207 16 7687 7671 148.195 148 0.187932 0.429802
208 16 7737 7721 148.444 200 0.368127 0.429801
209 16 7782 7766 148.595 180 0.0771454 0.429503
210 16 7810 7794 148.42 112 0.197459 0.42953
211 16 7843 7827 148.343 132 0.154616 0.429829
212 16 7883 7867 148.397 160 1.26504 0.429809
213 16 7918 7902 148.358 140 0.112432 0.429789
214 16 7939 7923 148.057 84 0.115489 0.430173
215 16 7969 7953 147.927 120 1.87709 0.430906
216 16 8015 7999 148.093 184 0.0791509 0.431065
217 16 8058 8042 148.203 172 0.392373 0.430655
218 16 8100 8084 148.294 168 0.267835 0.430392
219 16 8134 8118 148.238 136 0.0934882 0.430519
2020-02-09 13:32:23.175328 min lat: 0.0372422 max lat: 3.3428 avg lat: 0.430025
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 8179 8163 148.382 180 0.146404 0.430025
221 16 8201 8185 148.109 88 0.102131 0.430244
222 16 8247 8231 148.27 184 0.151443 0.430281
223 16 8277 8261 148.144 120 0.113215 0.430249
224 16 8309 8293 148.054 128 1.90335 0.430535
225 16 8348 8332 148.089 156 0.137005 0.430501
226 16 8393 8377 148.23 180 1.47221 0.430566
227 16 8440 8424 148.405 188 0.0736608 0.430109
228 16 8482 8466 148.491 168 0.310622 0.42972
229 16 8522 8506 148.541 160 0.113297 0.429471
230 16 8563 8547 148.608 164 0.0804247 0.429462
231 16 8615 8599 148.865 208 0.152066 0.429022
232 16 8666 8650 149.102 204 0.235952 0.428382
233 16 8715 8699 149.303 196 0.278372 0.427459
234 16 8760 8744 149.435 180 0.252322 0.427162
235 16 8803 8787 149.53 172 0.10573 0.426811
236 16 8841 8825 149.541 152 1.57202 0.426538
237 16 8869 8853 149.382 112 0.308915 0.426468
238 16 8898 8882 149.242 116 2.07807 0.427112
239 16 8928 8912 149.12 120 0.353452 0.427552
2020-02-09 13:32:43.178247 min lat: 0.0372422 max lat: 3.3428 avg lat: 0.427817
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 8953 8937 148.915 100 0.359972 0.427817
241 16 8983 8967 148.795 120 0.195311 0.428462
242 16 9018 9002 148.759 140 0.0790235 0.428465
243 16 9061 9045 148.854 172 0.131788 0.428893
244 16 9114 9098 149.113 212 0.18917 0.428431
245 16 9161 9145 149.271 188 0.568679 0.427854
246 16 9207 9191 149.412 184 0.878253 0.427395
247 16 9253 9237 149.552 184 0.290427 0.427006
248 16 9290 9274 149.546 148 0.0996611 0.426601
249 16 9329 9313 149.572 156 0.633384 0.426593
250 16 9366 9350 149.565 148 0.201538 0.426602
251 16 9399 9383 149.495 132 1.96777 0.426704
252 16 9433 9417 149.442 136 0.16124 0.426873
253 16 9472 9456 149.468 156 0.185 0.426775
254 16 9506 9490 149.415 136 0.398723 0.426595
255 16 9545 9529 149.44 156 0.183321 0.426623
256 16 9587 9571 149.513 168 0.119509 0.426472
257 16 9625 9609 149.522 152 0.385211 0.42689
258 16 9664 9648 149.547 156 0.109105 0.426757
259 16 9692 9676 149.402 112 0.0799794 0.426747
2020-02-09 13:33:03.181754 min lat: 0.0372422 max lat: 3.3428 avg lat: 0.42702
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 9728 9712 149.381 144 2.10264 0.42702
261 16 9762 9746 149.33 136 2.72966 0.42707
262 16 9787 9771 149.141 100 1.01599 0.427525
263 16 9818 9802 149.046 124 0.136691 0.427962
264 16 9860 9844 149.117 168 2.22855 0.427719
265 16 9894 9878 149.068 136 0.178066 0.427983
266 16 9923 9907 148.944 116 0.157223 0.428009
267 16 9971 9955 149.105 192 0.193973 0.427733
268 16 10017 10001 149.234 184 0.330592 0.427638
269 16 10056 10040 149.259 156 1.13164 0.427578
270 16 10090 10074 149.21 136 0.0725154 0.427934
271 16 10125 10109 149.176 140 0.0797251 0.427801
272 16 10156 10140 149.084 124 0.353268 0.427621
273 16 10193 10177 149.08 148 0.10865 0.427679
274 16 10233 10217 149.119 160 2.20318 0.428075
275 16 10265 10249 149.043 128 0.200518 0.428026
276 16 10300 10284 149.01 140 0.343428 0.42797
277 16 10329 10313 148.89 116 0.865722 0.427923
278 16 10356 10340 148.743 108 0.14778 0.42834
279 16 10386 10370 148.64 120 0.0793715 0.428726
2020-02-09 13:33:23.185357 min lat: 0.0372422 max lat: 3.3428 avg lat: 0.429261
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 10409 10393 148.438 92 2.88688 0.429261
281 16 10434 10418 148.265 100 2.72532 0.429702
282 16 10456 10440 148.052 88 0.16576 0.429962
283 16 10504 10488 148.207 192 0.258442 0.430629
284 16 10548 10532 148.305 176 0.32889 0.430555
285 16 10583 10567 148.275 140 0.20307 0.430647
286 16 10619 10603 148.26 144 0.312288 0.430365
287 16 10664 10648 148.37 180 0.084919 0.430287
288 16 10697 10681 148.313 132 0.97315 0.430161
289 16 10735 10719 148.326 152 0.147487 0.430278
290 16 10764 10748 148.215 116 0.0726217 0.430693
291 16 10808 10792 148.31 176 0.125523 0.430387
292 16 10848 10832 148.35 160 1.10635 0.430431
293 16 10883 10867 148.321 140 0.79381 0.430366
294 16 10931 10915 148.47 192 0.0885528 0.430116
295 16 10972 10956 148.522 164 0.460296 0.429907
296 16 11009 10993 148.521 148 1.27302 0.429746
297 16 11049 11033 148.559 160 0.178288 0.42945
298 16 11087 11071 148.571 152 2.30158 0.429519
299 16 11129 11113 148.636 168 0.185823 0.429486
2020-02-09 13:33:43.188854 min lat: 0.0371554 max lat: 3.3428 avg lat: 0.429263
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 11172 11156 148.713 172 0.167343 0.429263
301 16 11210 11194 148.724 152 0.160433 0.429062
302 16 11242 11226 148.656 128 0.212049 0.429122
303 16 11287 11271 148.759 180 0.311421 0.429094
304 16 11330 11314 148.835 172 0.235107 0.428883
305 16 11372 11356 148.898 168 0.445686 0.428719
306 16 11402 11386 148.804 120 0.397662 0.42869
307 16 11440 11424 148.814 152 0.117302 0.4287
308 16 11483 11467 148.889 172 0.171616 0.428745
309 16 11518 11502 148.86 140 0.53991 0.42876
310 16 11555 11539 148.858 148 0.168095 0.428708
311 16 11593 11577 148.868 152 0.128028 0.428571
312 16 11632 11616 148.89 156 1.94476 0.428652
313 16 11665 11649 148.836 132 0.122182 0.428571
314 16 11696 11680 148.757 124 0.112441 0.428616
315 16 11721 11705 148.603 100 0.126284 0.42908
316 16 11757 11741 148.588 144 0.31772 0.429367
317 16 11799 11783 148.649 168 0.0836561 0.429329
318 16 11829 11813 148.559 120 1.79385 0.429552
319 16 11865 11849 148.544 144 0.122584 0.429648
2020-02-09 13:34:03.191367 min lat: 0.0371554 max lat: 3.3428 avg lat: 0.429959
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 11901 11885 148.53 144 0.0860015 0.429959
321 16 11939 11923 148.541 152 0.0772369 0.429541
322 16 11972 11956 148.49 132 2.15216 0.429568
323 16 12015 11999 148.562 172 0.169144 0.429662
324 16 12060 12044 148.659 180 0.315283 0.429562
325 16 12109 12093 148.805 196 0.323831 0.429345
326 16 12145 12129 148.79 144 1.02612 0.429222
327 16 12192 12176 148.91 188 0.0861337 0.428925
328 16 12234 12218 148.968 168 1.51141 0.428639
329 16 12263 12247 148.864 116 0.191959 0.42866
330 16 12295 12279 148.801 128 0.083131 0.428944
331 16 12333 12317 148.81 152 0.26636 0.428935
332 16 12378 12362 148.904 180 0.0726336 0.428572
333 16 12413 12397 148.877 140 1.60141 0.428564
334 16 12446 12430 148.827 132 0.352465 0.428788
335 16 12487 12471 148.872 164 0.239609 0.428926
336 16 12508 12492 148.679 84 0.124293 0.428794
337 16 12524 12508 148.428 64 0.222529 0.429045
338 16 12549 12533 148.284 100 0.0383554 0.429388
339 16 12570 12554 148.095 84 0.136781 0.429586
2020-02-09 13:34:23.202129 min lat: 0.0371554 max lat: 4.06014 avg lat: 0.430108
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 12597 12581 147.977 108 0.691739 0.430108
341 16 12620 12604 147.813 92 0.119019 0.431232
342 16 12662 12646 147.872 168 2.43334 0.431444
343 16 12697 12681 147.849 140 0.298401 0.431646
344 16 12730 12714 147.802 132 0.221078 0.43167
345 16 12764 12748 147.768 136 0.69765 0.431912
346 16 12792 12776 147.665 112 0.137977 0.432159
347 16 12818 12802 147.539 104 0.879846 0.432386
348 16 12856 12840 147.552 152 0.125793 0.432321
349 16 12890 12874 147.519 136 0.0863322 0.432704
350 16 12920 12904 147.44 120 0.0381378 0.432904
351 16 12962 12946 147.498 168 1.71075 0.432837
352 16 13008 12992 147.602 184 0.149601 0.43253
353 16 13045 13029 147.603 148 0.21414 0.432497
354 16 13082 13066 147.604 148 0.20458 0.43238
355 16 13116 13100 147.571 136 0.138245 0.432553
356 16 13137 13121 147.393 84 0.800156 0.432668
357 16 13167 13151 147.316 120 0.402946 0.43294
358 16 13203 13187 147.307 144 0.199824 0.433138
359 16 13236 13220 147.264 132 0.169808 0.433227
2020-02-09 13:34:43.205412 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.43338
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 13274 13258 147.277 152 0.20655 0.43338
361 16 13300 13284 147.157 104 0.139172 0.433395
362 16 13329 13313 147.071 116 0.240137 0.433706
363 16 13367 13351 147.084 152 0.167571 0.433971
364 16 13416 13400 147.219 196 0.537863 0.433818
365 16 13464 13448 147.341 192 0.152654 0.433449
366 16 13510 13494 147.441 184 1.4697 0.433321
367 16 13547 13531 147.443 148 0.129573 0.433134
368 16 13585 13569 147.455 152 0.961605 0.433204
369 16 13633 13617 147.576 192 0.463153 0.43295
370 16 13675 13659 147.631 168 0.180883 0.432731
371 16 13703 13687 147.535 112 0.247103 0.43261
372 16 13724 13708 147.364 84 0.590489 0.432958
373 16 13751 13735 147.258 108 1.29877 0.433341
374 16 13787 13771 147.25 144 0.0733255 0.433393
375 16 13828 13812 147.294 164 0.134886 0.433677
376 16 13875 13859 147.402 188 0.437709 0.433403
377 16 13916 13900 147.446 164 0.909453 0.433289
378 16 13962 13946 147.543 184 0.118898 0.432914
379 16 14000 13984 147.555 152 0.2024 0.432853
2020-02-09 13:35:03.208344 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.432938
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 14035 14019 147.535 140 0.188379 0.432938
381 16 14077 14061 147.588 168 0.187073 0.432745
382 16 14105 14089 147.495 112 0.161368 0.432808
383 16 14142 14126 147.496 148 0.623856 0.432817
384 16 14176 14160 147.466 136 0.875272 0.432911
385 16 14206 14190 147.395 120 0.159145 0.433174
386 16 14237 14221 147.334 124 0.87072 0.433067
387 16 14261 14245 147.202 96 0.350214 0.433437
388 16 14311 14295 147.338 200 0.214412 0.433347
389 16 14347 14331 147.329 144 0.258318 0.433424
390 16 14389 14373 147.382 168 1.04781 0.433427
391 16 14423 14407 147.353 136 0.72955 0.433336
392 16 14462 14446 147.375 156 0.136917 0.433231
393 16 14500 14484 147.386 152 0.453865 0.433165
394 16 14521 14505 147.225 84 0.288692 0.433401
395 16 14565 14549 147.298 176 0.276463 0.433538
396 16 14598 14582 147.259 132 0.199 0.433735
397 16 14635 14619 147.261 148 0.310182 0.433533
398 16 14668 14652 147.223 132 0.878727 0.433452
399 16 14700 14684 147.175 128 0.128098 0.433778
2020-02-09 13:35:23.212174 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.433684
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 14736 14720 147.167 144 0.0787955 0.433684
401 16 14777 14761 147.209 164 0.295965 0.433589
402 16 14814 14798 147.211 148 0.238222 0.433627
403 16 14845 14829 147.153 124 0.0379297 0.433697
404 16 14887 14871 147.204 168 0.120115 0.433773
405 16 14936 14920 147.325 196 0.174135 0.433583
406 16 14973 14957 147.326 148 0.348998 0.43344
407 16 15004 14988 147.269 124 0.103221 0.433579
408 16 15032 15016 147.183 112 0.438134 0.433809
409 16 15065 15049 147.145 132 0.0854022 0.433849
410 16 15097 15081 147.099 128 0.0913248 0.433897
411 16 15133 15117 147.091 144 0.256522 0.433963
412 16 15155 15139 146.948 88 2.12797 0.434136
413 16 15190 15174 146.931 140 0.162838 0.434406
414 16 15214 15198 146.808 96 1.47845 0.434799
415 16 15252 15236 146.82 152 2.02248 0.434969
416 16 15302 15286 146.948 200 1.46194 0.434657
417 16 15338 15322 146.941 144 1.50939 0.43469
418 16 15389 15373 147.077 204 1.12122 0.434352
419 16 15428 15412 147.098 156 0.118907 0.43418
2020-02-09 13:35:43.215705 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.433971
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 15466 15450 147.11 152 0.825257 0.433971
421 16 15495 15479 147.036 116 0.341953 0.434172
422 16 15532 15516 147.038 148 0.180862 0.434288
423 16 15575 15559 147.097 172 1.90854 0.434159
424 16 15611 15595 147.09 144 0.0796344 0.434075
425 16 15639 15623 147.007 112 2.04723 0.434116
426 16 15675 15659 147 144 0.14453 0.4344
427 16 15712 15696 147.003 148 0.303808 0.434444
428 16 15741 15725 146.93 116 0.14508 0.43437
429 16 15773 15757 146.886 128 2.3327 0.434543
430 16 15804 15788 146.833 124 0.105307 0.434564
431 16 15849 15833 146.91 180 0.448747 0.434632
432 16 15879 15863 146.847 120 0.545647 0.434767
433 16 15903 15887 146.73 96 0.147497 0.434895
434 16 15938 15922 146.714 140 0.170654 0.434879
435 16 15965 15949 146.625 108 1.06293 0.435043
436 16 16007 15991 146.674 168 0.141577 0.435458
437 16 16050 16034 146.732 172 0.0860765 0.435429
438 16 16092 16076 146.781 168 0.0846797 0.435217
439 16 16150 16134 146.975 232 0.506564 0.434777
2020-02-09 13:36:03.218220 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.434446
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 16192 16176 147.022 168 0.116958 0.434446
441 16 16225 16209 146.988 132 0.166428 0.434419
442 16 16257 16241 146.945 128 1.74999 0.434407
443 16 16288 16272 146.893 124 0.554927 0.434489
444 16 16315 16299 146.806 108 0.156289 0.434705
445 16 16344 16328 146.737 116 0.387818 0.434832
446 16 16375 16359 146.686 124 0.108814 0.435006
447 16 16407 16391 146.644 128 0.294962 0.435501
448 16 16430 16414 146.522 92 0.0737029 0.435745
449 16 16463 16447 146.489 132 0.238807 0.435815
450 16 16511 16495 146.59 192 0.204294 0.435873
451 16 16548 16532 146.594 148 0.192584 0.435602
452 16 16581 16565 146.561 132 0.106459 0.435719
453 16 16624 16608 146.617 172 0.0717083 0.435704
454 16 16670 16654 146.7 184 0.25675 0.435577
455 16 16702 16686 146.658 128 0.153033 0.435571
456 16 16738 16722 146.652 144 0.0997808 0.435646
457 16 16785 16769 146.743 188 0.264005 0.435227
458 16 16832 16816 146.833 188 0.160897 0.434993
459 16 16874 16858 146.879 168 0.125864 0.434873
2020-02-09 13:36:23.221646 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.434652
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 16916 16900 146.925 168 1.53706 0.434652
461 16 16950 16934 146.901 136 0.201656 0.434849
462 16 17000 16984 147.016 200 0.439712 0.434664
463 16 17039 17023 147.035 156 0.709505 0.434492
464 16 17081 17065 147.08 168 0.312886 0.434325
465 16 17121 17105 147.108 160 0.132298 0.434227
466 16 17169 17153 147.204 192 0.238188 0.434038
467 16 17202 17186 147.172 132 0.26168 0.434034
468 16 17241 17225 147.191 156 0.109434 0.433921
469 16 17272 17256 147.141 124 0.134454 0.433852
470 16 17306 17290 147.117 136 2.16778 0.433897
471 16 17345 17329 147.136 156 0.099281 0.433875
472 16 17387 17371 147.18 168 0.123631 0.433757
473 16 17423 17407 147.174 144 0.0376549 0.433794
474 16 17471 17455 147.268 192 0.144803 0.433684
475 16 17521 17505 147.379 200 0.110836 0.433521
476 16 17568 17552 147.464 188 0.12726 0.433235
477 16 17605 17589 147.465 148 0.0792458 0.433075
478 16 17631 17615 147.374 104 0.0791533 0.433028
479 16 17664 17648 147.342 132 0.0786209 0.433127
2020-02-09 13:36:43.224242 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.433168
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 17699 17683 147.327 140 0.428667 0.433168
481 16 17740 17724 147.362 164 1.94898 0.433473
482 16 17775 17759 147.346 140 0.083168 0.433377
483 16 17811 17795 147.339 144 0.186749 0.433481
484 16 17851 17835 147.365 160 0.835302 0.433621
485 16 17903 17887 147.49 208 0.357078 0.433234
486 16 17954 17938 147.607 204 0.220655 0.432858
487 16 18000 17984 147.681 184 0.174906 0.432674
488 16 18043 18027 147.731 172 0.136553 0.432389
489 16 18088 18072 147.797 180 0.077572 0.432149
490 16 18123 18107 147.781 140 0.282586 0.432009
491 16 18161 18145 147.79 152 0.803411 0.432241
492 16 18199 18183 147.798 152 0.654894 0.43224
493 16 18231 18215 147.758 128 0.050921 0.432242
494 16 18264 18248 147.726 132 0.232159 0.432271
495 16 18283 18267 147.581 76 0.207232 0.432398
496 16 18308 18292 147.485 100 2.29834 0.432985
497 16 18343 18327 147.47 140 0.252852 0.433067
498 16 18377 18361 147.447 136 0.162768 0.433042
499 16 18423 18407 147.52 184 2.11463 0.433013
2020-02-09 13:37:03.226717 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.432696
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 18475 18459 147.641 208 0.349235 0.432696
501 16 18513 18497 147.65 152 0.225278 0.432492
502 16 18550 18534 147.65 148 0.319968 0.432492
503 16 18586 18570 147.643 144 0.189793 0.432397
504 16 18618 18602 147.604 128 0.125309 0.432459
505 16 18648 18632 147.549 120 0.623812 0.432631
506 16 18688 18672 147.574 160 0.30456 0.432762
507 16 18733 18717 147.638 180 0.207914 0.432615
508 16 18770 18754 147.639 148 1.45656 0.432581
509 16 18810 18794 147.663 160 0.151149 0.432756
510 16 18844 18828 147.64 136 0.437577 0.43261
511 16 18892 18876 147.727 192 0.277336 0.432416
512 16 18935 18919 147.774 172 0.202626 0.432224
513 16 18981 18965 147.845 184 0.216221 0.432172
514 16 19031 19015 147.946 200 0.323121 0.43195
515 16 19061 19045 147.892 120 0.437502 0.431928
516 16 19097 19081 147.884 144 0.266963 0.431934
517 16 19143 19127 147.954 184 0.158258 0.431787
518 16 19179 19163 147.946 144 0.285614 0.431802
519 16 19227 19211 148.031 192 0.633673 0.431557
2020-02-09 13:37:23.230160 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.431264
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 19278 19262 148.138 204 0.25527 0.431264
521 16 19326 19310 148.222 192 0.62698 0.430973
522 16 19368 19352 148.26 168 0.100481 0.430859
523 16 19407 19391 148.275 156 0.588045 0.43076
524 16 19453 19437 148.343 184 0.0795811 0.430628
525 16 19501 19485 148.426 192 0.159733 0.430423
526 16 19548 19532 148.502 188 0.122866 0.430144
527 16 19581 19565 148.47 132 0.136998 0.430003
528 16 19611 19595 148.416 120 0.118945 0.430175
529 16 19651 19635 148.438 160 0.128759 0.430271
530 16 19674 19658 148.332 92 0.199943 0.43042
531 16 19707 19691 148.301 132 0.134716 0.430539
532 16 19743 19727 148.293 144 0.12066 0.430661
533 16 19780 19764 148.292 148 0.169253 0.430572
534 16 19807 19791 148.217 108 0.12908 0.430593
535 16 19837 19821 148.164 120 0.633963 0.430943
536 16 19880 19864 148.208 172 2.01449 0.430868
537 16 19911 19895 148.163 124 0.785238 0.431099
538 16 19960 19944 148.252 196 0.190768 0.430953
539 16 20000 19984 148.274 160 0.165297 0.430835
2020-02-09 13:37:43.233537 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.4307
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 20049 20033 148.362 196 0.185534 0.4307
541 16 20096 20080 148.435 188 0.327209 0.430413
542 16 20139 20123 148.479 172 0.140309 0.430267
543 16 20178 20162 148.492 156 0.40302 0.430212
544 16 20231 20215 148.609 212 0.40325 0.43004
545 16 20277 20261 148.674 184 0.189831 0.42974
546 16 20324 20308 148.746 188 0.121609 0.42961
547 16 20380 20364 148.883 224 0.24303 0.429192
548 16 20423 20407 148.926 172 0.115516 0.428976
549 16 20457 20441 148.902 136 0.148409 0.429049
550 16 20496 20480 148.915 156 0.170918 0.429071
551 16 20519 20503 148.811 92 0.881173 0.429094
552 16 20553 20537 148.788 136 1.9161 0.429276
553 16 20589 20573 148.779 144 0.0777044 0.429302
554 16 20624 20608 148.764 140 2.00163 0.429387
555 16 20667 20651 148.805 172 0.535448 0.429334
556 16 20718 20702 148.905 204 0.52049 0.429135
557 16 20750 20734 148.867 128 0.432513 0.429163
558 16 20789 20773 148.88 156 0.606554 0.429139
559 16 20820 20804 148.835 124 0.18241 0.429114
2020-02-09 13:38:03.238318 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.429272
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 20858 20842 148.84 152 0.150577 0.429272
561 16 20893 20877 148.825 140 0.524481 0.429042
562 16 20930 20914 148.823 148 0.450568 0.429132
563 16 20979 20963 148.907 196 0.0823911 0.429005
564 16 21023 21007 148.955 176 0.277439 0.428965
565 16 21076 21060 149.066 212 0.152311 0.428674
566 16 21115 21099 149.079 156 0.294167 0.428444
567 16 21154 21138 149.091 156 0.079664 0.428444
568 16 21186 21170 149.054 128 0.92728 0.428489
569 16 21221 21205 149.038 140 0.152299 0.428494
570 16 21262 21246 149.064 164 0.155614 0.428482
571 16 21302 21286 149.083 160 0.0882518 0.428503
572 16 21336 21320 149.06 136 0.176632 0.428527
573 16 21391 21375 149.184 220 0.256822 0.428336
574 16 21443 21427 149.286 208 0.43319 0.428055
575 16 21488 21472 149.34 180 0.163608 0.427911
576 16 21524 21508 149.33 144 0.551366 0.42787
577 16 21560 21544 149.321 144 0.173311 0.427742
578 16 21599 21583 149.333 156 0.179705 0.427757
579 16 21644 21628 149.386 180 1.35689 0.427686
2020-02-09 13:38:23.240958 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.427513
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 21695 21679 149.48 204 0.220089 0.427513
581 16 21731 21715 149.47 144 0.144685 0.427329
582 16 21754 21738 149.371 92 0.0904589 0.427396
583 16 21790 21774 149.362 144 0.435873 0.427551
584 16 21823 21807 149.332 132 1.57209 0.427718
585 16 21856 21840 149.303 132 0.164843 0.427771
586 16 21889 21873 149.273 132 0.154224 0.427897
587 16 21923 21907 149.251 136 1.71284 0.427955
588 16 21948 21932 149.167 100 0.443082 0.428027
589 16 21981 21965 149.138 132 0.149436 0.428017
590 16 22023 22007 149.17 168 0.0753 0.42812
591 16 22054 22038 149.127 124 0.138113 0.428243
592 16 22091 22075 149.125 148 1.31866 0.428397
593 16 22130 22114 149.137 156 0.153368 0.428261
594 16 22161 22145 149.094 124 0.675096 0.428256
595 16 22183 22167 148.992 88 0.119917 0.428565
596 16 22218 22202 148.976 140 0.125538 0.428624
597 16 22245 22229 148.908 108 0.0840039 0.428937
598 16 22287 22271 148.94 168 0.313115 0.428859
599 16 22311 22295 148.851 96 0.250167 0.428968
2020-02-09 13:38:43.243589 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.429032
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 16 22341 22325 148.803 120 2.24232 0.429032
Total time run: 601.086899
Total reads made: 22341
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 148.671
Average IOPS: 37
Stddev IOPS: 7
Max IOPS: 58
Min IOPS: 16
Average Latency(s): 0.429814
Max latency(s): 4.37235
Min latency(s): 0.0371554
Cleanup
[root@ceph-mon-01 ~]# rados -p testbench cleanup
Removed 11790 objects
Ceph RBD Performance Measurements
Prepare libvirt
Since one can use libvirt with Ceph RBD, I created 2 vols on the libvirt side. Note that I set up libvirt with a pool.
root@hypervisor ~ # virsh vol-list --pool Ceph-HouseNet-libvirt-pool
Name Path
------------------------------------------------------------------------------
banana libvirt-pool/banana
banana2 libvirt-pool/banana2
Prepare Test VM
Attached the 500 GiB RBD (banana2) to an existing VM as a virtio block device, formatted it with XFS, mounted it and ran fio.
[root@centos-stream ~]# dmesg
[...]
[11506.366077] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)
[11506.371877] virtio-pci 0000:00:0b.0: virtio_pci: leaving for legacy driver
[11506.379137] virtio_blk virtio6: [vdc] 1048576000 512-byte logical blocks (537 GB/500 GiB)
[root@centos-stream ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 1G 0 disk
└─vda1 252:1 0 1023M 0 part /boot
vdb 252:16 0 30G 0 disk
└─vdb1 252:17 0 30G 0 part
├─VG_cs-LV_root 253:0 0 8G 0 lvm /
├─VG_cs-LV_swap 253:1 0 3.1G 0 lvm [SWAP]
├─VG_cs-LV_home 253:2 0 1G 0 lvm /home
└─VG_cs-LV_var_log 253:3 0 1G 0 lvm /var/log
vdc 252:32 0 500G 0 disk
[root@centos-stream ~]# mkfs.xfs /dev/vdc
meta-data=/dev/vdc isize=512 agcount=4, agsize=32768000 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=131072000, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=64000, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@centos-stream ~]# mount /dev/vdc /mnt/
[root@centos-stream ~]# df -h /mnt/
Filesystem Size Used Avail Use% Mounted on
/dev/vdc 500G 3.6G 497G 1% /mnt
Sequential Read, 4k Blocksize
READ: bw=84.9MiB/siops: avg=21742.01
fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4k
readtest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
readtest: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [R(1)][99.9%][r=74.1MiB/s,w=0KiB/s][r=18.0k,w=0 IOPS][eta 00m:01s]
readtest: (groupid=0, jobs=1): err= 0: pid=32571: Sat Feb 8 17:45:56 2020
read: IOPS=21.7k, BW=84.9MiB/s (89.1MB/s)(160GiB/1928858msec)
clat (nsec): min=1040, max=602104k, avg=44325.67, stdev=1608935.76
lat (nsec): min=1390, max=602105k, avg=44692.57, stdev=1608936.85
clat percentiles (nsec):
| 1.00th=[ 1128], 5.00th=[ 1144], 10.00th=[ 1176],
| 20.00th=[ 1208], 30.00th=[ 1240], 40.00th=[ 1288],
| 50.00th=[ 1400], 60.00th=[ 2040], 70.00th=[ 3824],
| 80.00th=[ 3984], 90.00th=[ 4256], 95.00th=[ 4640],
| 99.00th=[ 6560], 99.50th=[ 11712], 99.90th=[ 7897088],
| 99.95th=[34340864], 99.99th=[78118912]
bw ( KiB/s): min= 4096, max=143360, per=99.99%, avg=86968.07, stdev=17899.79, samples=3857
iops : min= 1024, max=35840, avg=21742.01, stdev=4474.94, samples=3857
lat (usec) : 2=59.88%, 4=20.96%, 10=18.54%, 20=0.44%, 50=0.05%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.02%, 50=0.04%
lat (msec) : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=1.75%, sys=8.46%, ctx=51940, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=41943040,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=84.9MiB/s (89.1MB/s), 84.9MiB/s-84.9MiB/s (89.1MB/s-89.1MB/s), io=160GiB (172GB), run=1928858-1928858msec
Disk stats (read/write):
vdc: ios=171695/7, merge=1/0, ticks=19105195/1381, in_queue=19020794, util=10.11%
Sequential Read, 4m Blocksize
READ: bw=74.7MiB/siops: avg=18.65
fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4m
readtest: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=76.1MiB/s,w=0KiB/s][r=19,w=0 IOPS][eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=10007: Sun Feb 9 10:39:59 2020
read: IOPS=18, BW=74.7MiB/s (78.3MB/s)(160GiB/2193474msec)
clat (usec): min=425, max=672398, avg=53545.47, stdev=63278.39
lat (usec): min=425, max=672398, avg=53546.45, stdev=63278.33
clat percentiles (usec):
| 1.00th=[ 515], 5.00th=[ 775], 10.00th=[ 1336], 20.00th=[ 2089],
| 30.00th=[ 2835], 40.00th=[ 4359], 50.00th=[ 24773], 60.00th=[ 49021],
| 70.00th=[ 79168], 80.00th=[111674], 90.00th=[147850], 95.00th=[177210],
| 99.00th=[233833], 99.50th=[263193], 99.90th=[320865], 99.95th=[354419],
| 99.99th=[492831]
bw ( KiB/s): min=16351, max=147456, per=99.99%, avg=76480.58, stdev=18909.26, samples=4386
iops : min= 3, max= 36, avg=18.65, stdev= 4.61, samples=4386
lat (usec) : 500=0.84%, 750=3.94%, 1000=2.20%
lat (msec) : 2=11.67%, 4=20.31%, 10=4.90%, 20=4.20%, 50=12.20%
lat (msec) : 100=15.90%, 250=23.17%, 500=0.65%, 750=0.01%
cpu : usr=0.02%, sys=4.33%, ctx=41070, majf=0, minf=524
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=40960,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=74.7MiB/s (78.3MB/s), 74.7MiB/s-74.7MiB/s (78.3MB/s-78.3MB/s), io=160GiB (172GB), run=2193474-2193474msec
Disk stats (read/write):
vdc: ios=174951/0, merge=17/0, ticks=21141738/0, in_queue=21054323, util=9.14%
Sequential Write, 4k Blocksize
WRITE: bw=31.4MiB/siops: avg=8041.64
fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
writetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
writetest: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=30.3MiB/s][r=0,w=7750 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=505: Sat Feb 8 19:15:35 2020
write: IOPS=8041, BW=31.4MiB/s (32.9MB/s)(160GiB/5215542msec)
clat (usec): min=2, max=101237, avg=122.62, stdev=1027.96
lat (usec): min=2, max=101237, avg=123.03, stdev=1027.96
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3],
| 30.00th=[ 3], 40.00th=[ 3], 50.00th=[ 3], 60.00th=[ 3],
| 70.00th=[ 4], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 8],
| 99.00th=[ 6783], 99.50th=[ 9372], 99.90th=[11600], 99.95th=[11600],
| 99.99th=[11600]
bw ( KiB/s): min= 8976, max=612672, per=100.00%, avg=32166.57, stdev=12683.70, samples=10431
iops : min= 2244, max=153168, avg=8041.64, stdev=3170.93, samples=10431
lat (usec) : 4=70.58%, 10=26.51%, 20=1.40%, 50=0.11%, 100=0.01%
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=1.09%, 20=0.29%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=0.63%, sys=3.84%, ctx=579301, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,41943040,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=31.4MiB/s (32.9MB/s), 31.4MiB/s-31.4MiB/s (32.9MB/s-32.9MB/s), io=160GiB (172GB), run=5215542-5215542msec
Disk stats (read/write):
vdc: ios=0/175534, merge=0/4, ticks=0/998233048, in_queue=998145387, util=0.50%
Sequential Write, 4m Blocksize
WRITE: bw=57.5MiB/siops: avg=14.34
fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
writetest: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=56.1MiB/s][r=0,w=14 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=10351: Sun Feb 9 11:28:38 2020
write: IOPS=14, BW=57.5MiB/s (60.3MB/s)(160GiB/2851095msec)
clat (usec): min=1410, max=155763, avg=69470.49, stdev=11483.13
lat (usec): min=1482, max=155890, avg=69601.41, stdev=11480.40
clat percentiles (msec):
| 1.00th=[ 41], 5.00th=[ 51], 10.00th=[ 60], 20.00th=[ 63],
| 30.00th=[ 64], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 72],
| 70.00th=[ 72], 80.00th=[ 81], 90.00th=[ 82], 95.00th=[ 88],
| 99.00th=[ 90], 99.50th=[ 100], 99.90th=[ 102], 99.95th=[ 110],
| 99.99th=[ 136]
bw ( KiB/s): min=32768, max=2105344, per=99.99%, avg=58839.67, stdev=29089.37, samples=5702
iops : min= 8, max= 514, avg=14.34, stdev= 7.11, samples=5702
lat (msec) : 2=0.62%, 4=0.18%, 10=0.01%, 50=4.55%, 100=94.55%
lat (msec) : 250=0.10%
cpu : usr=0.23%, sys=3.49%, ctx=347036, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,40960,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=57.5MiB/s (60.3MB/s), 57.5MiB/s-57.5MiB/s (60.3MB/s-60.3MB/s), io=160GiB (172GB), run=2851095-2851095msec
Disk stats (read/write):
vdc: ios=0/177589, merge=0/1, ticks=0/544638715, in_queue=544549875, util=0.91%
Sequential, 80% Read, 20% Write, 4k Blocksize
READ: bw=65.0MiB/siops: avg=16775.73WRITE: bw=16.2MiB/siops: avg=4194.47
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k
read80write20test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=84.0MiB/s,w=21.1MiB/s][r=21.5k,w=5403 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=3708: Sat Feb 8 23:32:33 2020
read: IOPS=16.6k, BW=65.0MiB/s (68.2MB/s)(128GiB/2016416msec)
clat (nsec): min=1060, max=6496.7M, avg=56898.90, stdev=2814777.43
lat (nsec): min=1400, max=6496.7M, avg=57266.32, stdev=2814777.63
clat percentiles (nsec):
| 1.00th=[ 1128], 5.00th=[ 1160], 10.00th=[ 1176],
| 20.00th=[ 1224], 30.00th=[ 1256], 40.00th=[ 1304],
| 50.00th=[ 1400], 60.00th=[ 1704], 70.00th=[ 3792],
| 80.00th=[ 3984], 90.00th=[ 4256], 95.00th=[ 4576],
| 99.00th=[ 6624], 99.50th=[ 11840], 99.90th=[ 7766016],
| 99.95th=[ 38010880], 99.99th=[102236160]
bw ( KiB/s): min= 32, max=143360, per=100.00%, avg=67102.92, stdev=29982.08, samples=3999
iops : min= 8, max=35840, avg=16775.73, stdev=7495.52, samples=3999
write: IOPS=4159, BW=16.2MiB/s (17.0MB/s)(31.0GiB/2016416msec)
clat (nsec): min=1800, max=11521k, avg=4234.74, stdev=11813.18
lat (usec): min=2, max=11521, avg= 4.64, stdev=11.83
clat percentiles (nsec):
| 1.00th=[ 2064], 5.00th=[ 2352], 10.00th=[ 2448], 20.00th=[ 2544],
| 30.00th=[ 2608], 40.00th=[ 2736], 50.00th=[ 2992], 60.00th=[ 3792],
| 70.00th=[ 5472], 80.00th=[ 5856], 90.00th=[ 6688], 95.00th=[ 7840],
| 99.00th=[14144], 99.50th=[18304], 99.90th=[28800], 99.95th=[33536],
| 99.99th=[44288]
bw ( KiB/s): min= 784, max=35936, per=100.00%, avg=16777.90, stdev=7500.81, samples=3998
iops : min= 196, max= 8984, avg=4194.47, stdev=1875.20, samples=3998
lat (usec) : 2=50.45%, 4=26.89%, 10=21.75%, 20=0.67%, 50=0.13%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
cpu : usr=1.79%, sys=8.47%, ctx=40855, majf=0, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=33555281,8387759,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=65.0MiB/s (68.2MB/s), 65.0MiB/s-65.0MiB/s (68.2MB/s-68.2MB/s), io=128GiB (137GB), run=2016416-2016416msec
WRITE: bw=16.2MiB/s (17.0MB/s), 16.2MiB/s-16.2MiB/s (17.0MB/s-17.0MB/s), io=31.0GiB (34.4GB), run=2016416-2016416msec
Disk stats (read/write):
vdc: ios=133994/34031, merge=0/10, ticks=18502416/22797554, in_queue=41215836, util=8.15%
Sequential, 80% Read, 20% Write, 4m Blocksize
READ: bw=55.0MiB/siops: avg=14.46WRITE: bw=14.2MiB/siops: avg=4.84
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
read80write20test: (g=0): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=80.1MiB/s,w=32.0MiB/s][r=20,w=8 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=10903: Sun Feb 9 12:19:34 2020
read: IOPS=13, BW=55.0MiB/s (58.7MB/s)(128GiB/2335537msec)
clat (usec): min=435, max=1555.6k, avg=70791.35, stdev=107394.86
lat (usec): min=436, max=1555.6k, avg=70792.35, stdev=107394.84
clat percentiles (usec):
| 1.00th=[ 537], 5.00th=[ 758], 10.00th=[ 1205],
| 20.00th=[ 1893], 30.00th=[ 2704], 40.00th=[ 9372],
| 50.00th=[ 35914], 60.00th=[ 58983], 70.00th=[ 86508],
| 80.00th=[ 121111], 90.00th=[ 175113], 95.00th=[ 238027],
| 99.00th=[ 541066], 99.50th=[ 675283], 99.90th=[1027605],
| 99.95th=[1132463], 99.99th=[1300235]
bw ( KiB/s): min= 8143, max=131072, per=100.00%, avg=59278.21, stdev=26040.27, samples=4517
iops : min= 1, max= 32, avg=14.46, stdev= 6.36, samples=4517
write: IOPS=3, BW=14.2MiB/s (14.9MB/s)(32.3GiB/2335537msec)
clat (usec): min=840, max=7833, avg=2477.47, stdev=929.71
lat (usec): min=885, max=7933, avg=2570.40, stdev=932.81
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1369], 10.00th=[ 1450], 20.00th=[ 1582],
| 30.00th=[ 1778], 40.00th=[ 2024], 50.00th=[ 2311], 60.00th=[ 2638],
| 70.00th=[ 2966], 80.00th=[ 3326], 90.00th=[ 3785], 95.00th=[ 4178],
| 99.00th=[ 4817], 99.50th=[ 4948], 99.90th=[ 5604], 99.95th=[ 5800],
| 99.99th=[ 7832]
bw ( KiB/s): min= 8159, max=106496, per=100.00%, avg=19894.69, stdev=12748.10, samples=3406
iops : min= 1, max= 26, avg= 4.84, stdev= 3.11, samples=3406
lat (usec) : 500=0.44%, 750=3.48%, 1000=2.49%
lat (msec) : 2=18.60%, 4=23.11%, 10=4.19%, 20=2.71%, 50=10.22%
lat (msec) : 100=14.03%, 250=17.16%, 500=2.67%, 750=0.66%, 1000=0.17%
cpu : usr=0.05%, sys=3.77%, ctx=32499, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=32688,8272,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=55.0MiB/s (58.7MB/s), 55.0MiB/s-55.0MiB/s (58.7MB/s-58.7MB/s), io=128GiB (137GB), run=2335537-2335537msec
WRITE: bw=14.2MiB/s (14.9MB/s), 14.2MiB/s-14.2MiB/s (14.9MB/s-14.9MB/s), io=32.3GiB (34.7GB), run=2335537-2335537msec
Disk stats (read/write):
vdc: ios=145237/37789, merge=12/0, ticks=22152096/27634303, in_queue=49694677, util=7.44%
Conclusions
- As expected, Fedora works fine on the hardware.
- As expected, a quick Ceph smoke test was fine.
- The Ceph performance is already a bit better than with ODROID-HC2 OSDs only.
So, apart from budget considerations, I am ready to buy 3 more of these and a 10Gig switch. And then remove the ODROID-HC2 boards from my cluster.
Addendum 2020-02-14
Added three SATA HDD to the F5-422 (1x 1TB, 2x 160G. Those were the last old drives I had on the shelf),
removed the old HDD only osd.7,
created 4 OSDs each with 1 physical SATA HDD and 1 LV on the SATA SSD.
Volume Groups
[root@f5-422-01 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
VG_Ceph_HDD_00 1 1 0 wz--n- <1.82t 0
VG_Ceph_HDD_01 1 1 0 wz--n- <931.51g 0
VG_Ceph_HDD_02 1 1 0 wz--n- <149.05g 0
VG_Ceph_HDD_03 1 1 0 wz--n- <149.05g 0
VG_Ceph_SSD_01 1 4 0 wz--n- <399.17g <223.17g
VG_f5-422-01 1 5 0 wz--n- 65.00g 39.00g
Logical Volumes
[root@f5-422-01 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
LV_Ceph_HDD_OSD_00 VG_Ceph_HDD_00 -wi-ao---- <1.82t
LV_Ceph_HDD_OSD_01 VG_Ceph_HDD_01 -wi-ao---- <931.51g
LV_Ceph_HDD_OSD_02 VG_Ceph_HDD_02 -wi-ao---- <149.05g
LV_Ceph_HDD_OSD_03 VG_Ceph_HDD_03 -wi-ao---- <149.05g
LV_Ceph_SSD_OSD_00 VG_Ceph_SSD_01 -wi-ao---- 80.00g
LV_Ceph_SSD_OSD_01 VG_Ceph_SSD_01 -wi-ao---- 80.00g
LV_Ceph_SSD_OSD_02 VG_Ceph_SSD_01 -wi-ao---- 8.00g
LV_Ceph_SSD_OSD_03 VG_Ceph_SSD_01 -wi-ao---- 8.00g
LV_home VG_f5-422-01 -wi-ao---- 4.00g
LV_root VG_f5-422-01 -wi-ao---- 8.00g
LV_swap VG_f5-422-01 -wi-ao---- 4.00g
LV_var VG_f5-422-01 -wi-ao---- 8.00g
LV_var_log VG_f5-422-01 -wi-ao---- 2.00g
Physical Volumes
[root@f5-422-01 ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 VG_f5-422-01 lvm2 a-- 65.00g 39.00g
/dev/sda4 VG_Ceph_SSD_01 lvm2 a-- <399.17g <223.17g
/dev/sdb1 VG_Ceph_HDD_01 lvm2 a-- <931.51g 0
/dev/sdc1 VG_Ceph_HDD_00 lvm2 a-- <1.82t 0
/dev/sdd1 VG_Ceph_HDD_02 lvm2 a-- <149.05g 0
/dev/sde1 VG_Ceph_HDD_03 lvm2 a-- <149.05g 0
[root@f5-422-01 ~]# ceph-volume lvm create --data VG_Ceph_HDD_00/LV_Ceph_HDD_OSD_00 --block.db VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_00 --dmcrypt --bluestore
[root@f5-422-01 ~]# ceph-volume lvm create --data VG_Ceph_HDD_01/LV_Ceph_HDD_OSD_01 --block.db VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_01 --dmcrypt --bluestore
[root@f5-422-01 ~]# ceph-volume lvm create --data VG_Ceph_HDD_02/LV_Ceph_HDD_OSD_02 --block.db VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_02 --dmcrypt --bluestore
[root@f5-422-01 ~]# ceph-volume lvm create --data VG_Ceph_HDD_03/LV_Ceph_HDD_OSD_03 --block.db VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_03 --dmcrypt --bluestore
System load, memory use and io wait were still OK when it was sorting misplaced objects.
FIXME: the above really should used better LV naming.
See https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#devices
AFAIK the WAL is placed with the DB, if not specified separately.
so no need to add --block.wal <wal-device>.
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/administration_guide/osd-bluestore
snapshot of `top
top - 17:52:20 up 29 min, 1 user, load average: 0.67, 0.50, 0.36
Tasks: 201 total, 2 running, 199 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.9 us, 7.0 sy, 0.0 ni, 76.0 id, 8.7 wa, 1.5 hi, 3.8 si, 0.0 st
MiB Mem : 11831.6 total, 10763.2 free, 744.4 used, 324.0 buff/cache
MiB Swap: 4096.0 total, 4096.0 free, 0.0 used. 10828.4 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6887 ceph 20 0 860548 228292 30420 S 12.3 1.9 1:10.03 ceph-osd
1993 ceph 20 0 858528 210648 29928 S 11.3 1.7 1:40.37 ceph-osd
7498 root 0 -20 0 0 0 I 2.3 0.0 0:03.31 kworker/u9:5-kcryptd/253:17
4738 root 0 -20 0 0 0 I 2.0 0.0 0:09.87 kworker/u9:0-kcryptd/253:14
4739 root 0 -20 0 0 0 I 2.0 0.0 0:07.08 kworker/u9:3-kcryptd/253:14
1732 root 0 -20 0 0 0 I 1.7 0.0 0:13.80 kworker/u9:1-kcryptd/253:14
1214 root 20 0 269468 28304 13812 S 0.7 0.2 0:01.95 tuned
[...]
ceph osd tree
[root@ceph-mon-01 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 13.02634 root default
-17 3.02019 host f5-422-01
7 hdd 1.81929 osd.7 up 1.00000 1.00000
8 hdd 0.90970 osd.8 up 1.00000 1.00000
9 hdd 0.14560 osd.9 up 1.00000 1.00000
10 hdd 0.14560 osd.10 up 1.00000 1.00000
-13 0.90959 host odroid-hc2-00
6 hdd 0.90959 osd.6 up 1.00000 1.00000
-11 0.90970 host odroid-hc2-01
0 hdd 0.90970 osd.0 up 1.00000 1.00000
-9 0.90970 host odroid-hc2-02
4 hdd 0.90970 osd.4 up 1.00000 1.00000
-15 1.81929 host odroid-hc2-03
5 hdd 1.81929 osd.5 up 1.00000 1.00000
-3 1.81929 host odroid-hc2-04
1 hdd 1.81929 osd.1 up 1.00000 1.00000
-5 1.81929 host odroid-hc2-05
2 hdd 1.81929 osd.2 up 1.00000 1.00000
-7 1.81929 host odroid-hc2-06
3 hdd 1.81929 osd.3 up 1.00000 1.00000
ceph-volume lvm list
[root@f5-422-01 ~]# ceph-volume lvm list
====== osd.9 =======
[block] /dev/VG_Ceph_HDD_02/LV_Ceph_HDD_OSD_02
type block
osd id 9
cluster fsid 154b75c2-1767-45c5-b7f6-ba7e3a36fc12
cluster name ceph
osd fsid ebb82241-6760-421f-9715-577b04046089
db device /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_02
encrypted 1
db uuid EmDgfU-hhKz-1Pe5-ua8x-Hsl3-F0TH-ythz10
cephx lockbox secret [REDACTED]
block uuid ibDiUQ-frjH-TvUt-th4C-tbwB-dYwS-9yIYJP
block device /dev/VG_Ceph_HDD_02/LV_Ceph_HDD_OSD_02
vdo 0
crush device class None
devices /dev/sdd1
[ db] /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_02
type db
osd id 9
cluster fsid 154b75c2-1767-45c5-b7f6-ba7e3a36fc12
cluster name ceph
osd fsid ebb82241-6760-421f-9715-577b04046089
db device /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_02
encrypted 1
db uuid EmDgfU-hhKz-1Pe5-ua8x-Hsl3-F0TH-ythz10
cephx lockbox secret [REDACTED]
block uuid ibDiUQ-frjH-TvUt-th4C-tbwB-dYwS-9yIYJP
block device /dev/VG_Ceph_HDD_02/LV_Ceph_HDD_OSD_02
vdo 0
crush device class None
devices /dev/sda4
====== osd.8 =======
[block] /dev/VG_Ceph_HDD_01/LV_Ceph_HDD_OSD_01
type block
osd id 8
cluster fsid 154b75c2-1767-45c5-b7f6-ba7e3a36fc12
cluster name ceph
osd fsid 78f5d058-b0b5-4a7b-988f-3f96332b1246
db device /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_01
encrypted 1
db uuid q8tPpn-1wSg-ais7-7cac-k5eB-HLd1-UItqkV
cephx lockbox secret [REDACTED]
block uuid z1rEFf-Xx9B-81FV-sYuG-x21C-S9hW-TN752D
block device /dev/VG_Ceph_HDD_01/LV_Ceph_HDD_OSD_01
vdo 0
crush device class None
devices /dev/sdb1
[ db] /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_01
type db
osd id 8
cluster fsid 154b75c2-1767-45c5-b7f6-ba7e3a36fc12
cluster name ceph
osd fsid 78f5d058-b0b5-4a7b-988f-3f96332b1246
db device /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_01
encrypted 1
db uuid q8tPpn-1wSg-ais7-7cac-k5eB-HLd1-UItqkV
cephx lockbox secret [REDACTED]
block uuid z1rEFf-Xx9B-81FV-sYuG-x21C-S9hW-TN752D
block device /dev/VG_Ceph_HDD_01/LV_Ceph_HDD_OSD_01
vdo 0
crush device class None
devices /dev/sda4
====== osd.7 =======
[block] /dev/VG_Ceph_HDD_00/LV_Ceph_HDD_OSD_00
type block
osd id 7
cluster fsid 154b75c2-1767-45c5-b7f6-ba7e3a36fc12
cluster name ceph
osd fsid 30b5fe84-7b76-4fcb-9eae-af7a375779b0
db device /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_00
encrypted 1
db uuid YGRkoK-BgOK-CqU7-GUih-Jc6m-gvjH-qw593H
cephx lockbox secret [REDACTED]
block uuid kqcXz2-VDHF-KNCu-9kXc-vgeP-doYB-6d6fEs
block device /dev/VG_Ceph_HDD_00/LV_Ceph_HDD_OSD_00
vdo 0
crush device class None
devices /dev/sdc1
[ db] /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_00
type db
osd id 7
cluster fsid 154b75c2-1767-45c5-b7f6-ba7e3a36fc12
cluster name ceph
osd fsid 30b5fe84-7b76-4fcb-9eae-af7a375779b0
db device /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_00
encrypted 1
db uuid YGRkoK-BgOK-CqU7-GUih-Jc6m-gvjH-qw593H
cephx lockbox secret [REDACTED]
block uuid kqcXz2-VDHF-KNCu-9kXc-vgeP-doYB-6d6fEs
block device /dev/VG_Ceph_HDD_00/LV_Ceph_HDD_OSD_00
vdo 0
crush device class None
devices /dev/sda4
====== osd.10 ======
[block] /dev/VG_Ceph_HDD_03/LV_Ceph_HDD_OSD_03
type block
osd id 10
cluster fsid 154b75c2-1767-45c5-b7f6-ba7e3a36fc12
cluster name ceph
osd fsid 9c00d813-7676-4083-a1d4-d64b76f8f4ab
db device /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_03
encrypted 1
db uuid jtXDYp-nG6y-bKsp-okzW-5Aj3-zgJQ-5cDWz1
cephx lockbox secret [REDACTED]
block uuid Jptyt5-RmI5-1KRD-fXaO-h5OH-dqLa-g7f9nN
block device /dev/VG_Ceph_HDD_03/LV_Ceph_HDD_OSD_03
vdo 0
crush device class None
devices /dev/sde1
[ db] /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_03
type db
osd id 10
cluster fsid 154b75c2-1767-45c5-b7f6-ba7e3a36fc12
cluster name ceph
osd fsid 9c00d813-7676-4083-a1d4-d64b76f8f4ab
db device /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_03
encrypted 1
db uuid jtXDYp-nG6y-bKsp-okzW-5Aj3-zgJQ-5cDWz1
cephx lockbox secret [REDACTED]
block uuid Jptyt5-RmI5-1KRD-fXaO-h5OH-dqLa-g7f9nN
block device /dev/VG_Ceph_HDD_03/LV_Ceph_HDD_OSD_03
vdo 0
crush device class None
devices /dev/sda4
Again, all rados bench tests were run on ceph-mon-01.
Sequential Write Test
Total time run: 600.652078
Total writes made: 11790
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 78.5147
Stddev Bandwidth: 13.7158
Max bandwidth (MB/sec): 116
Min bandwidth (MB/sec): 28
Average IOPS: 19
Stddev IOPS: 3
Max IOPS: 29
Min IOPS: 7
Average Latency(s): 0.815131
Stddev Latency(s): 0.339359
Max latency(s): 2.53383
Min latency(s): 0.21279
rados bench -p testbench 600 write --no-cleanup | tee --append $(date +%F)-rados-test-write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_ceph-mon-01.internal.pcfe.net_20282
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 16 16 0 0 0 - 0
1 16 26 10 39.9614 40 0.934416 0.70472
2 16 48 32 63.9641 88 0.506183 0.782824
3 16 72 56 74.6361 96 0.763491 0.77637
4 16 93 77 76.9744 84 0.410092 0.760004
5 16 110 94 75.1785 68 0.427527 0.769987
6 16 125 109 72.6482 60 0.744231 0.778053
7 16 145 129 73.6971 80 0.284302 0.793663
8 16 165 149 74.4837 80 0.468672 0.7996
9 16 178 162 71.9852 52 1.37299 0.818159
10 16 195 179 71.5857 68 0.405016 0.853723
11 16 212 196 71.2591 68 0.420537 0.852709
12 16 234 218 72.6533 88 0.358446 0.847286
13 16 247 231 71.0643 52 0.374069 0.843661
14 16 268 252 71.9876 84 0.473908 0.860645
15 16 285 269 71.7212 68 0.483529 0.856597
16 16 301 285 71.2383 64 1.59227 0.856457
17 16 317 301 70.8121 64 0.442328 0.85968
18 16 331 315 69.989 56 0.58816 0.88181
19 16 353 337 70.9364 88 1.3939 0.88399
2020-02-14 22:41:13.556889 min lat: 0.249889 max lat: 2.26579 avg lat: 0.883883
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 369 353 70.589 64 0.312343 0.883883
21 16 384 368 70.0839 60 1.36119 0.885663
22 16 401 385 69.9887 68 0.295768 0.888053
23 16 424 408 70.945 92 0.712418 0.891446
24 16 437 421 70.1554 52 0.493172 0.887942
25 16 458 442 70.7088 84 0.698938 0.893216
26 16 473 457 70.2967 60 0.76582 0.891686
27 16 495 479 70.952 88 0.556024 0.886044
28 16 512 496 70.8463 68 0.574363 0.880923
29 16 531 515 71.0237 76 1.3633 0.887491
30 16 548 532 70.9227 68 0.569357 0.891555
31 16 572 556 71.7312 96 0.495083 0.88095
32 16 587 571 71.3645 60 0.661303 0.881039
33 16 608 592 71.7471 84 0.289608 0.878793
34 16 629 613 72.1072 84 0.588759 0.880177
35 16 648 632 72.2182 76 0.540896 0.874191
36 16 672 656 72.8785 96 0.648527 0.867963
37 16 690 674 72.8543 72 0.686326 0.864152
38 16 712 696 73.2526 88 0.554647 0.864158
39 16 724 708 72.605 48 0.292921 0.857932
2020-02-14 22:41:33.559446 min lat: 0.247503 max lat: 2.26579 avg lat: 0.862447
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 730 714 71.3899 24 1.6527 0.862447
41 16 754 738 71.9898 96 0.93162 0.8824
42 16 772 756 71.9899 72 0.823795 0.879136
43 16 794 778 72.362 88 0.72404 0.874639
44 16 813 797 72.4445 76 0.768016 0.871235
45 16 835 819 72.7899 88 0.696 0.867428
46 16 856 840 73.0334 84 0.860067 0.86868
47 16 873 857 72.9262 68 0.473534 0.866583
48 16 889 873 72.7401 64 1.48436 0.868305
49 16 913 897 73.2145 96 0.486223 0.866772
50 16 929 913 73.0301 64 0.630302 0.863038
51 16 954 938 73.5587 100 0.732801 0.864161
52 16 972 956 73.5286 72 0.550323 0.863083
53 16 994 978 73.8015 88 0.902776 0.860538
54 16 1012 996 73.7679 72 1.03256 0.86029
55 16 1032 1016 73.8809 80 0.569524 0.858338
56 16 1049 1033 73.7758 68 1.4483 0.859223
57 16 1067 1051 73.7444 72 0.658675 0.85949
58 16 1091 1075 74.128 96 0.401844 0.857796
59 16 1113 1097 74.3628 88 0.798294 0.855516
2020-02-14 22:41:53.561836 min lat: 0.223072 max lat: 2.5726 avg lat: 0.852987
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 1134 1118 74.5233 84 0.959303 0.852987
61 16 1154 1138 74.6129 80 0.744574 0.851366
62 16 1173 1157 74.6351 76 0.867636 0.850559
63 16 1192 1176 74.6567 76 0.873523 0.849874
64 16 1214 1198 74.865 88 0.588496 0.847659
65 16 1239 1223 75.2515 100 0.79085 0.846227
66 16 1258 1242 75.2627 76 0.794235 0.843374
67 16 1279 1263 75.393 84 0.826382 0.843282
68 16 1298 1282 75.4018 76 0.896996 0.842269
69 16 1321 1305 75.6422 92 0.673371 0.841708
70 16 1339 1323 75.59 72 0.437782 0.840853
71 16 1364 1348 75.9337 100 0.391643 0.838934
72 16 1378 1362 75.6568 56 0.34302 0.839368
73 16 1402 1386 75.9353 96 0.350094 0.840248
74 16 1417 1401 75.7199 60 1.05723 0.839819
75 16 1440 1424 75.9368 92 0.321802 0.838751
76 16 1463 1447 76.148 92 0.853417 0.835822
77 16 1479 1463 75.9902 64 0.913123 0.835253
78 16 1501 1485 76.144 88 0.824506 0.836111
79 16 1524 1508 76.3446 92 0.513667 0.834243
2020-02-14 22:42:13.564122 min lat: 0.223072 max lat: 2.5726 avg lat: 0.834991
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 1540 1524 76.1901 64 0.917013 0.834991
81 16 1557 1541 76.0889 68 1.44861 0.835767
82 16 1571 1555 75.8438 56 0.345051 0.836941
83 16 1587 1571 75.7011 64 0.55372 0.838406
84 16 1608 1592 75.7988 84 0.930096 0.840216
85 16 1621 1605 75.5187 52 1.38569 0.841744
86 16 1636 1620 75.3382 60 0.351551 0.841674
87 16 1656 1640 75.3917 80 0.502591 0.842249
88 16 1676 1660 75.4439 80 0.671843 0.843994
89 16 1694 1678 75.4052 72 0.690537 0.844436
90 16 1711 1695 75.3228 68 0.314688 0.843794
91 16 1728 1712 75.2422 68 0.284384 0.845079
92 16 1738 1722 74.8591 40 1.92035 0.845002
93 16 1749 1733 74.5273 44 2.23773 0.852311
94 16 1766 1750 74.4577 68 0.27145 0.853821
95 16 1786 1770 74.516 80 0.688935 0.854932
96 16 1802 1786 74.4063 64 0.415568 0.854633
97 16 1819 1803 74.3402 68 0.371044 0.856848
98 16 1842 1826 74.5203 92 0.440917 0.855608
99 16 1861 1845 74.5352 76 0.297912 0.854583
2020-02-14 22:42:33.567498 min lat: 0.223072 max lat: 2.86714 avg lat: 0.852756
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 1883 1867 74.6697 88 0.795727 0.852756
101 16 1905 1889 74.8016 88 0.453008 0.852565
102 16 1921 1905 74.6957 64 1.09684 0.85211
103 16 1936 1920 74.5529 60 0.404247 0.85102
104 16 1955 1939 74.5668 76 0.381559 0.854327
105 16 1975 1959 74.6184 80 0.333144 0.855177
106 16 1991 1975 74.5182 64 1.1419 0.854361
107 16 2011 1995 74.5693 80 0.892053 0.854394
108 16 2029 2013 74.5455 72 0.879552 0.855386
109 16 2051 2035 74.6688 88 0.868937 0.853935
110 16 2074 2058 74.8263 92 0.741671 0.851699
111 16 2092 2076 74.8007 72 0.25916 0.851671
112 16 2110 2094 74.7757 72 0.649942 0.85182
113 16 2135 2119 74.9988 100 0.472506 0.851276
114 16 2149 2133 74.832 56 0.932414 0.850445
115 16 2173 2157 75.016 96 0.887637 0.849942
116 16 2190 2174 74.9554 68 0.399664 0.849994
117 16 2208 2192 74.9301 72 0.293087 0.850524
118 16 2227 2211 74.9391 76 0.832736 0.851101
119 16 2244 2228 74.8807 68 1.51699 0.851084
2020-02-14 22:42:53.569832 min lat: 0.223072 max lat: 2.86714 avg lat: 0.851065
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 2264 2248 74.9233 80 0.741264 0.851065
121 16 2284 2268 74.9652 80 0.682165 0.851271
122 16 2301 2285 74.908 68 0.316093 0.85003
123 16 2323 2307 75.0144 88 0.556807 0.850731
124 16 2343 2327 75.0545 80 0.522033 0.849193
125 16 2360 2344 74.998 68 0.541304 0.849782
126 16 2384 2368 75.1646 96 0.297488 0.848757
127 16 2399 2383 75.0451 60 1.33596 0.84915
128 16 2420 2404 75.115 84 0.56683 0.848436
129 16 2436 2420 75.0288 64 1.37316 0.850204
130 16 2457 2441 75.0977 84 1.38193 0.849683
131 16 2476 2460 75.1045 76 0.464722 0.848705
132 16 2498 2482 75.2021 88 1.15378 0.848154
133 16 2520 2504 75.2983 88 0.39472 0.846382
134 16 2536 2520 75.2139 64 1.19732 0.846436
135 16 2564 2548 75.4863 112 0.72259 0.846479
136 16 2582 2566 75.4606 72 1.17366 0.845311
137 16 2594 2578 75.2601 48 1.28357 0.846153
138 16 2615 2599 75.3233 84 1.31443 0.846796
139 16 2632 2616 75.2706 68 0.473917 0.846719
2020-02-14 22:43:13.572303 min lat: 0.223072 max lat: 2.86714 avg lat: 0.84659
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 2653 2637 75.3329 84 0.456113 0.84659
141 16 2673 2657 75.3659 80 1.27685 0.847157
142 16 2688 2672 75.2576 60 0.392893 0.846492
143 16 2708 2692 75.2907 80 1.48618 0.84626
144 16 2723 2707 75.1844 60 0.490787 0.846729
145 16 2743 2727 75.2176 80 0.836494 0.847327
146 16 2769 2753 75.4146 104 0.679884 0.846507
147 16 2786 2770 75.3641 68 0.597976 0.846749
148 16 2804 2788 75.3413 72 0.517633 0.846419
149 16 2827 2811 75.4531 92 0.650622 0.845517
150 16 2845 2829 75.43 72 1.49757 0.845187
151 16 2863 2847 75.4072 72 0.996583 0.845987
152 16 2888 2872 75.569 100 0.49177 0.845011
153 16 2909 2893 75.624 84 0.826943 0.844609
154 16 2930 2914 75.6779 84 0.614527 0.843205
155 16 2947 2931 75.6283 68 0.290206 0.842965
156 16 2964 2948 75.5793 68 0.619257 0.84366
157 16 2986 2970 75.6584 88 0.506869 0.843539
158 16 3002 2986 75.5846 64 0.265475 0.84354
159 16 3025 3009 75.6877 92 0.741666 0.843646
2020-02-14 22:43:33.575695 min lat: 0.223072 max lat: 2.86714 avg lat: 0.843319
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 3042 3026 75.6396 68 0.671463 0.843319
161 16 3059 3043 75.5921 68 0.379707 0.843159
162 16 3085 3069 75.7674 104 0.360053 0.843197
163 16 3103 3087 75.7443 72 0.491305 0.842371
164 16 3124 3108 75.7945 84 0.861325 0.842085
165 16 3142 3126 75.7715 72 0.686406 0.841676
166 16 3164 3148 75.8451 88 0.92492 0.841545
167 16 3183 3167 75.846 76 0.336833 0.840663
168 16 3197 3181 75.7278 56 0.297921 0.84153
169 16 3217 3201 75.753 80 0.384566 0.842507
170 16 3240 3224 75.8486 92 0.476998 0.842379
171 16 3260 3244 75.8727 80 0.371012 0.841434
172 16 3278 3262 75.8502 72 0.547964 0.841449
173 16 3294 3278 75.7816 64 0.477398 0.841627
174 16 3317 3301 75.8748 92 0.314621 0.841515
175 16 3335 3319 75.8526 72 0.885499 0.841042
176 16 3356 3340 75.8988 84 0.75498 0.840854
177 16 3370 3354 75.7864 56 0.420594 0.841091
178 16 3391 3375 75.8324 84 1.23795 0.842172
179 16 3412 3396 75.878 84 0.651285 0.842088
2020-02-14 22:43:53.578085 min lat: 0.223072 max lat: 2.86714 avg lat: 0.841129
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 3431 3415 75.8786 76 1.23181 0.841129
181 16 3451 3435 75.9013 80 0.443923 0.841411
182 16 3472 3456 75.9458 84 1.29829 0.840641
183 16 3494 3478 76.0116 88 0.411632 0.840323
184 16 3513 3497 76.0115 76 0.933453 0.839895
185 16 3536 3520 76.0979 92 0.834135 0.839512
186 16 3554 3538 76.0758 72 0.843441 0.83922
187 16 3570 3554 76.0112 64 0.516185 0.838325
188 16 3591 3575 76.0536 84 0.640002 0.839774
189 16 3611 3595 76.0744 80 0.342578 0.83935
190 16 3629 3613 76.053 72 1.01039 0.839876
191 16 3642 3626 75.927 52 0.450038 0.839891
192 16 3654 3638 75.7815 48 2.0488 0.840513
193 16 3675 3659 75.824 84 0.494598 0.842343
194 16 3701 3685 75.9692 104 0.698002 0.841863
195 16 3716 3700 75.8873 60 0.476102 0.841201
196 16 3739 3723 75.9694 92 0.279008 0.84099
197 16 3757 3741 75.9492 72 0.507472 0.840353
198 16 3778 3762 75.9898 84 0.882544 0.840057
199 16 3798 3782 76.0099 80 0.620276 0.840043
2020-02-14 22:44:13.580545 min lat: 0.212625 max lat: 2.86714 avg lat: 0.839361
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 3816 3800 75.9898 72 0.475999 0.839361
201 16 3836 3820 76.0097 80 0.696399 0.839865
202 16 3859 3843 76.0888 92 0.871214 0.839276
203 16 3872 3856 75.9701 52 1.38602 0.839964
204 16 3894 3878 76.029 88 0.428129 0.840022
205 16 3912 3896 76.0094 72 0.6636 0.840007
206 16 3935 3919 76.0864 92 0.729706 0.839551
207 16 3957 3941 76.144 88 0.585139 0.83898
208 16 3976 3960 76.1432 76 0.753602 0.838972
209 16 3993 3977 76.1042 68 1.11383 0.838931
210 16 4015 3999 76.1608 88 0.391903 0.838574
211 16 4035 4019 76.179 80 1.09045 0.838659
212 16 4053 4037 76.1592 72 0.915773 0.838531
213 16 4074 4058 76.196 84 0.588361 0.837971
214 16 4092 4076 76.1763 72 0.504845 0.83826
215 16 4110 4094 76.1569 72 0.460312 0.83834
216 16 4133 4117 76.2302 92 0.852069 0.837708
217 16 4144 4128 76.0816 44 1.40087 0.83876
218 16 4164 4148 76.0996 80 1.28114 0.839468
219 16 4181 4165 76.0625 68 0.389401 0.839008
2020-02-14 22:44:33.584182 min lat: 0.212625 max lat: 2.86714 avg lat: 0.839435
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 4201 4185 76.0804 80 0.419191 0.839435
221 16 4219 4203 76.0619 72 0.752501 0.839156
222 16 4240 4224 76.0976 84 0.490634 0.839171
223 16 4258 4242 76.0792 72 0.376712 0.839591
224 16 4277 4261 76.0788 76 0.304429 0.839477
225 16 4300 4284 76.1495 92 0.669913 0.839032
226 16 4315 4299 76.078 60 1.41747 0.838542
227 16 4335 4319 76.0952 80 1.37994 0.839772
228 16 4345 4329 75.9369 40 0.648106 0.839553
229 16 4369 4353 76.0245 96 0.777862 0.840525
230 16 4389 4373 76.0417 80 0.667933 0.840261
231 16 4410 4394 76.0761 84 0.372537 0.83969
232 16 4432 4416 76.1275 88 0.512308 0.839019
233 16 4452 4436 76.1441 80 0.335746 0.838827
234 16 4467 4451 76.0751 60 0.624963 0.8389
235 16 4485 4469 76.0577 72 0.328452 0.838649
236 16 4505 4489 76.0742 80 0.706252 0.839467
237 16 4528 4512 76.1413 92 0.898602 0.839011
238 16 4547 4531 76.1407 76 0.802389 0.838821
239 16 4568 4552 76.1735 84 0.869376 0.838655
2020-02-14 22:44:53.587037 min lat: 0.212625 max lat: 2.86714 avg lat: 0.838695
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 4575 4559 75.9728 28 1.77838 0.838695
241 16 4604 4588 76.1388 116 0.494749 0.839428
242 16 4617 4601 76.0391 52 1.34318 0.839286
243 16 4635 4619 76.0224 72 1.22263 0.840278
244 16 4651 4635 75.9731 64 1.73266 0.840841
245 16 4663 4647 75.8589 48 0.894494 0.841124
246 16 4688 4672 75.957 100 0.828727 0.841557
247 16 4698 4682 75.8114 40 0.855654 0.841427
248 16 4723 4707 75.9089 100 0.358043 0.841654
249 16 4746 4730 75.9735 92 0.638163 0.841106
250 16 4767 4751 76.0056 84 1.12893 0.840728
251 16 4786 4770 76.0055 76 1.28614 0.840777
252 16 4806 4790 76.0213 80 0.886302 0.840207
253 16 4829 4813 76.0844 92 0.786548 0.839753
254 16 4847 4831 76.068 72 1.04593 0.840361
255 16 4860 4844 75.9736 52 0.441618 0.840384
256 16 4876 4860 75.9268 64 0.433653 0.840757
257 16 4894 4878 75.9114 72 1.25133 0.841587
258 16 4913 4897 75.9117 76 0.372412 0.841376
259 16 4930 4914 75.8812 68 1.23798 0.842325
2020-02-14 22:45:13.590482 min lat: 0.212625 max lat: 2.86714 avg lat: 0.842371
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 4949 4933 75.8816 76 0.391742 0.842371
261 16 4970 4954 75.9126 84 0.579967 0.841828
262 16 4992 4976 75.9587 88 0.830126 0.841368
263 16 5013 4997 75.9893 84 0.417882 0.841165
264 16 5031 5015 75.9741 72 0.891979 0.840782
265 16 5052 5036 76.0044 84 0.702783 0.840884
266 16 5071 5055 76.0043 76 0.394352 0.840849
267 16 5091 5075 76.0192 80 0.463335 0.840202
268 16 5112 5096 76.049 84 0.702678 0.839889
269 16 5132 5116 76.0636 80 1.19854 0.839907
270 16 5143 5127 75.9449 44 0.577941 0.839567
271 16 5162 5146 75.9451 76 0.788984 0.841106
272 16 5181 5165 75.9452 76 0.905767 0.841102
273 16 5204 5188 76.004 92 0.626781 0.840968
274 16 5215 5199 75.8872 44 0.881823 0.841126
275 16 5237 5221 75.9312 88 0.576911 0.841937
276 16 5260 5244 75.9894 92 1.02934 0.841472
277 16 5280 5264 76.0038 80 0.88656 0.841138
278 16 5299 5283 76.0037 76 0.431861 0.840377
279 16 5319 5303 76.018 80 0.730097 0.840368
2020-02-14 22:45:33.592903 min lat: 0.212625 max lat: 2.86714 avg lat: 0.840195
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 5342 5326 76.0751 92 0.825522 0.840195
281 16 5362 5346 76.089 80 1.47364 0.839971
282 16 5384 5368 76.1312 88 0.718996 0.839561
283 16 5407 5391 76.1873 92 0.455541 0.838959
284 16 5426 5410 76.1866 76 0.339565 0.838931
285 16 5445 5429 76.1859 76 0.997768 0.838967
286 16 5459 5443 76.1153 56 0.6453 0.839156
287 16 5481 5465 76.1566 88 0.339332 0.839123
288 16 5501 5485 76.1699 80 0.54045 0.838988
289 16 5520 5504 76.1693 76 0.608298 0.838836
290 16 5534 5518 76.0997 56 0.521014 0.83875
291 16 5558 5542 76.1681 96 0.694046 0.838985
292 16 5575 5559 76.1401 68 1.35632 0.839131
293 16 5599 5583 76.2078 96 1.10081 0.838568
294 16 5612 5596 76.1255 52 0.765025 0.838712
295 16 5631 5615 76.125 76 0.899501 0.839244
296 16 5647 5631 76.084 64 0.467947 0.839035
297 16 5671 5655 76.151 96 0.30348 0.839383
298 16 5689 5673 76.1369 72 0.758561 0.839404
299 16 5712 5696 76.1899 92 0.551772 0.838726
2020-02-14 22:45:53.596067 min lat: 0.212625 max lat: 2.86714 avg lat: 0.838389
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 5737 5721 76.2692 100 0.704915 0.838389
301 16 5758 5742 76.2949 84 0.785669 0.837782
302 16 5781 5765 76.3469 92 0.429183 0.837098
303 16 5797 5781 76.3061 64 0.494189 0.837037
304 16 5822 5806 76.384 100 0.708051 0.836388
305 16 5835 5819 76.304 52 0.440783 0.836516
306 16 5844 5828 76.1723 36 2.03004 0.837712
307 16 5859 5843 76.1196 60 0.644313 0.839269
308 16 5879 5863 76.1322 80 0.724387 0.839088
309 16 5897 5881 76.1188 72 0.813836 0.839241
310 16 5917 5901 76.1313 80 0.748287 0.839477
311 16 5939 5923 76.1694 88 0.301734 0.839049
312 16 5960 5944 76.1945 84 0.57288 0.838689
313 16 5976 5960 76.1555 64 0.255404 0.838917
314 16 5983 5967 76.0021 28 0.373497 0.838771
315 16 6001 5985 75.9894 72 0.527997 0.840387
316 16 6020 6004 75.9894 76 0.387321 0.840964
317 16 6035 6019 75.9389 60 0.589032 0.840863
318 16 6051 6035 75.9014 64 1.5921 0.842122
319 16 6062 6046 75.8014 44 1.75864 0.842892
2020-02-14 22:46:13.598325 min lat: 0.212625 max lat: 2.86714 avg lat: 0.843093
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 6075 6059 75.727 52 0.510625 0.843093
321 16 6095 6079 75.7402 80 1.33221 0.843825
322 16 6112 6096 75.7162 68 0.531465 0.843635
323 16 6134 6118 75.7542 88 0.583068 0.843745
324 16 6152 6136 75.7425 72 0.402024 0.843517
325 16 6175 6159 75.7925 92 0.443608 0.843279
326 16 6192 6176 75.7686 68 0.368951 0.842888
327 16 6214 6198 75.806 88 0.594562 0.84354
328 16 6228 6212 75.7456 56 0.412328 0.843309
329 16 6251 6235 75.795 92 0.603614 0.843651
330 16 6267 6251 75.7592 64 0.339706 0.843608
331 16 6282 6266 75.7116 60 0.653862 0.84355
332 16 6297 6281 75.6642 60 1.66479 0.844
333 16 6318 6302 75.6892 84 0.658443 0.844422
334 16 6333 6317 75.6422 60 0.712221 0.844198
335 16 6354 6338 75.6672 84 0.392954 0.844561
336 16 6371 6355 75.6443 68 0.43528 0.844609
337 16 6392 6376 75.6691 84 0.883243 0.844805
338 16 6404 6388 75.5872 48 1.66772 0.845132
339 16 6419 6403 75.5412 60 0.9948 0.845651
2020-02-14 22:46:33.600618 min lat: 0.212625 max lat: 2.86714 avg lat: 0.845656
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 6440 6424 75.5661 84 0.280988 0.845656
341 16 6460 6444 75.579 80 1.12206 0.845706
342 16 6476 6460 75.5452 64 1.01699 0.845659
343 16 6496 6480 75.5581 80 1.05166 0.845939
344 16 6512 6496 75.5245 64 0.784998 0.846013
345 16 6536 6520 75.5838 96 0.910466 0.846027
346 16 6549 6533 75.5156 52 0.561443 0.845922
347 16 6569 6553 75.5285 80 1.53521 0.846466
348 16 6588 6572 75.5299 76 0.852849 0.846206
349 16 6610 6594 75.5656 88 0.747423 0.846219
350 16 6629 6613 75.5668 76 1.00139 0.846136
351 16 6654 6638 75.6363 100 0.330358 0.84538
352 16 6668 6652 75.5805 56 1.05996 0.845172
353 16 6689 6673 75.6044 84 0.453 0.845399
354 16 6710 6694 75.628 84 0.978247 0.845216
355 16 6731 6715 75.6516 84 0.449951 0.844757
356 16 6754 6738 75.6975 92 0.999364 0.844656
357 16 6769 6753 75.6535 60 0.257063 0.844173
358 16 6793 6777 75.7103 96 0.643254 0.84433
359 16 6818 6802 75.7779 100 0.630813 0.843743
2020-02-14 22:46:53.603057 min lat: 0.212625 max lat: 2.86714 avg lat: 0.843784
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 6834 6818 75.7452 64 0.509345 0.843784
361 16 6854 6838 75.7569 80 0.628862 0.844216
362 16 6876 6860 75.7907 88 0.991614 0.843604
363 16 6902 6886 75.8684 104 1.01746 0.842847
364 16 6923 6907 75.8907 84 0.434695 0.842675
365 16 6940 6924 75.8691 68 0.863234 0.842366
366 16 6959 6943 75.8694 76 0.899639 0.8426
367 16 6974 6958 75.8262 60 0.335408 0.842511
368 16 6993 6977 75.8266 76 0.827961 0.843044
369 16 7017 7001 75.8812 96 0.655636 0.842712
370 16 7037 7021 75.8924 80 0.418677 0.842272
371 16 7059 7043 75.925 88 0.827429 0.842055
372 16 7078 7062 75.9251 76 0.624113 0.841904
373 16 7096 7080 75.9146 72 0.276808 0.841758
374 16 7116 7100 75.9255 80 0.334143 0.84183
375 16 7137 7121 75.947 84 0.463178 0.841999
376 16 7157 7141 75.9578 80 0.679909 0.841811
377 16 7181 7165 76.0109 96 0.716504 0.841136
378 16 7195 7179 75.9579 56 0.834678 0.841002
379 16 7214 7198 75.958 76 0.645061 0.841464
2020-02-14 22:47:13.605469 min lat: 0.212625 max lat: 2.86714 avg lat: 0.840743
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 7240 7224 76.0318 104 0.512115 0.840743
381 16 7264 7248 76.0841 96 0.341694 0.84039
382 16 7287 7271 76.1258 92 0.44678 0.840084
383 16 7305 7289 76.115 72 1.15374 0.840066
384 16 7316 7300 76.0313 44 0.719197 0.840029
385 16 7339 7323 76.0728 92 0.342887 0.840672
386 16 7354 7338 76.0311 60 1.17534 0.840714
387 16 7376 7360 76.062 88 0.434123 0.840524
388 16 7397 7381 76.0824 84 0.470167 0.840056
389 16 7418 7402 76.1028 84 0.554076 0.839718
390 16 7442 7426 76.1538 96 0.508361 0.839625
391 16 7460 7444 76.1431 72 0.387758 0.839367
392 16 7477 7461 76.1223 68 0.384375 0.839285
393 16 7493 7477 76.0915 64 1.37253 0.839787
394 16 7520 7504 76.1724 108 0.262682 0.83959
395 16 7535 7519 76.1315 60 0.985886 0.83948
396 16 7549 7533 76.0806 56 1.56169 0.83965
397 16 7562 7546 76.0199 52 1.49123 0.841198
398 16 7577 7561 75.9797 60 0.629233 0.841186
399 16 7600 7584 76.0198 92 0.863336 0.841071
2020-02-14 22:47:33.607833 min lat: 0.212625 max lat: 2.86714 avg lat: 0.841177
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 7611 7595 75.9397 44 1.81077 0.841177
401 16 7630 7614 75.9399 76 0.721655 0.84179
402 16 7649 7633 75.94 76 0.4543 0.841562
403 16 7668 7652 75.9401 76 0.297391 0.841552
404 16 7686 7670 75.9303 72 0.499471 0.841649
405 16 7702 7686 75.9009 64 0.612848 0.842062
406 16 7717 7701 75.8617 60 1.33675 0.842256
407 16 7734 7718 75.8423 68 0.516045 0.842413
408 16 7756 7740 75.8721 88 1.272 0.84252
409 16 7776 7760 75.8822 80 0.536753 0.842253
410 16 7802 7786 75.9507 104 0.358715 0.841835
411 16 7823 7807 75.9703 84 0.989766 0.841568
412 16 7842 7826 75.9703 76 1.33159 0.841513
413 16 7856 7840 75.922 56 0.339558 0.841386
414 16 7874 7858 75.9125 72 1.38033 0.841934
415 16 7892 7876 75.903 72 0.339063 0.842166
416 16 7907 7891 75.8648 60 0.407453 0.842302
417 16 7928 7912 75.8843 84 0.607168 0.842308
418 16 7944 7928 75.8558 64 0.484209 0.841998
419 16 7959 7943 75.818 60 0.357573 0.842675
2020-02-14 22:47:53.610169 min lat: 0.212625 max lat: 2.86714 avg lat: 0.843013
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 7974 7958 75.7803 60 0.399992 0.843013
421 16 7990 7974 75.7523 64 0.436496 0.842993
422 16 8014 7998 75.8003 96 0.271399 0.843754
423 16 8037 8021 75.8385 92 0.542647 0.843353
424 16 8055 8039 75.8295 72 0.847937 0.842989
425 16 8077 8061 75.8581 88 0.444809 0.843029
426 16 8094 8078 75.8396 68 0.372791 0.8428
427 16 8106 8090 75.7744 48 0.43486 0.842745
428 16 8122 8106 75.7469 64 0.567038 0.843194
429 16 8149 8133 75.822 108 0.859567 0.843703
430 16 8170 8154 75.841 84 0.928082 0.843454
431 16 8185 8169 75.8042 60 1.08726 0.843415
432 16 8199 8183 75.7584 56 0.42473 0.843196
433 16 8219 8203 75.7681 80 1.23468 0.843669
434 16 8236 8220 75.7502 68 0.523934 0.843578
435 16 8258 8242 75.7784 88 0.794599 0.843634
436 16 8283 8267 75.8339 100 0.815178 0.843314
437 16 8293 8277 75.7519 40 0.400676 0.843102
438 16 8311 8295 75.7433 72 0.464273 0.843948
439 16 8331 8315 75.753 80 1.0573 0.844153
2020-02-14 22:48:13.612601 min lat: 0.212625 max lat: 2.86714 avg lat: 0.843745
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 8351 8335 75.7626 80 0.615956 0.843745
441 16 8365 8349 75.7178 56 0.407184 0.8441
442 16 8387 8371 75.7455 88 1.13163 0.844343
443 16 8407 8391 75.7551 80 0.680602 0.843787
444 16 8424 8408 75.7376 68 1.1486 0.844123
445 16 8443 8427 75.7382 76 0.997138 0.844205
446 16 8465 8449 75.7656 88 0.391728 0.843891
447 16 8489 8473 75.8109 96 0.633176 0.843387
448 16 8499 8483 75.731 40 0.301932 0.84342
449 16 8520 8504 75.7493 84 0.272483 0.844305
450 16 8537 8521 75.7321 68 0.454225 0.844124
451 16 8561 8545 75.777 96 0.637149 0.843772
452 16 8583 8567 75.804 88 0.62366 0.843568
453 16 8607 8591 75.8486 96 0.407124 0.843282
454 16 8623 8607 75.8225 64 0.637014 0.843114
455 16 8649 8633 75.8844 104 0.931033 0.842717
456 16 8662 8646 75.832 52 0.900576 0.842646
457 16 8678 8662 75.8061 64 0.336761 0.842805
458 16 8697 8681 75.8065 76 0.281588 0.843104
459 16 8717 8701 75.8156 80 0.256536 0.842813
2020-02-14 22:48:33.615049 min lat: 0.212625 max lat: 2.86714 avg lat: 0.842962
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 8741 8725 75.8595 96 0.683358 0.842962
461 16 8761 8745 75.8684 80 0.893701 0.842652
462 16 8782 8766 75.886 84 0.471468 0.842877
463 16 8801 8785 75.8862 76 0.530747 0.842645
464 16 8823 8807 75.9123 88 0.53591 0.842144
465 16 8848 8832 75.9641 100 0.518616 0.842071
466 16 8867 8851 75.9642 76 0.801448 0.841763
467 16 8889 8873 75.9899 88 0.392658 0.841542
468 16 8909 8893 75.9984 80 0.950495 0.841361
469 16 8927 8911 75.9899 72 0.585551 0.841366
470 16 8948 8932 76.0069 84 0.437067 0.84104
471 16 8967 8951 76.0069 76 1.06616 0.841255
472 16 8988 8972 76.0238 84 0.425995 0.840832
473 16 9008 8992 76.0322 80 0.68746 0.840736
474 16 9032 9016 76.0743 96 1.10766 0.840699
475 16 9052 9036 76.0825 80 0.47195 0.840593
476 16 9068 9052 76.0571 64 0.418375 0.840568
477 16 9086 9070 76.0486 72 1.48098 0.840785
478 16 9104 9088 76.0401 72 1.25832 0.840848
479 16 9126 9110 76.065 88 0.491488 0.840799
2020-02-14 22:48:53.617620 min lat: 0.212625 max lat: 2.86714 avg lat: 0.840342
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 9151 9135 76.1149 100 0.698806 0.840342
481 16 9168 9152 76.098 68 0.58216 0.840193
482 16 9188 9172 76.1061 80 0.560368 0.840225
483 16 9207 9191 76.1058 76 0.381203 0.839948
484 16 9225 9209 76.0973 72 0.743648 0.840094
485 16 9244 9228 76.0971 76 0.432193 0.840173
486 16 9265 9249 76.1134 84 0.48358 0.840069
487 16 9284 9268 76.1131 76 0.451657 0.840074
488 16 9299 9283 76.0801 60 1.19697 0.840331
489 16 9319 9303 76.0881 80 0.685949 0.840382
490 16 9340 9324 76.1042 84 1.00618 0.84031
491 16 9357 9341 76.0877 68 1.326 0.84027
492 16 9378 9362 76.1037 84 0.341335 0.84042
493 16 9397 9381 76.1035 76 0.479356 0.840153
494 16 9420 9404 76.1357 92 0.653924 0.839953
495 16 9441 9425 76.1515 84 0.599013 0.839917
496 16 9460 9444 76.1512 76 0.856523 0.839647
497 16 9479 9463 76.1508 76 0.731891 0.83966
498 16 9498 9482 76.1505 76 0.80608 0.839343
499 16 9522 9506 76.1902 96 0.641915 0.839182
2020-02-14 22:49:13.620631 min lat: 0.212625 max lat: 2.86714 avg lat: 0.838865
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 9544 9528 76.2138 88 0.552168 0.838865
501 16 9561 9545 76.1974 68 1.46231 0.838976
502 16 9587 9571 76.2527 104 0.853431 0.83882
503 16 9608 9592 76.2681 84 0.3187 0.83858
504 16 9626 9610 76.2596 72 0.427209 0.838259
505 16 9646 9630 76.267 80 1.44886 0.8386
506 16 9661 9645 76.2349 60 0.894211 0.838683
507 16 9676 9660 76.2028 60 0.732685 0.838561
508 16 9697 9681 76.2181 84 0.523891 0.838956
509 16 9717 9701 76.2256 80 0.606057 0.83892
510 16 9737 9721 76.2329 80 0.495583 0.838771
511 16 9755 9739 76.2246 72 0.638669 0.838589
512 16 9776 9760 76.2398 84 0.98839 0.838794
513 16 9798 9782 76.2627 88 0.460856 0.838662
514 16 9817 9801 76.2622 76 0.904328 0.838565
515 16 9834 9818 76.2461 68 0.515633 0.838494
516 16 9858 9842 76.2844 96 0.672811 0.838266
517 16 9880 9864 76.307 88 0.511715 0.838065
518 16 9901 9885 76.3219 84 0.462689 0.837795
519 16 9920 9904 76.3212 76 0.618191 0.837672
2020-02-14 22:49:33.623101 min lat: 0.212625 max lat: 2.86714 avg lat: 0.837568
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 9940 9924 76.3283 80 0.575968 0.837568
521 16 9963 9947 76.3581 92 1.25087 0.837546
522 16 9986 9970 76.3881 92 0.673702 0.837267
523 16 10004 9988 76.3797 72 0.747442 0.83711
524 16 10025 10009 76.3942 84 0.724471 0.836893
525 16 10048 10032 76.4239 92 0.836856 0.836764
526 16 10071 10055 76.4535 92 0.638493 0.836527
527 16 10092 10076 76.4678 84 0.580833 0.836362
528 16 10113 10097 76.482 84 0.916029 0.836109
529 16 10125 10109 76.4282 48 0.319385 0.835826
530 16 10141 10125 76.4047 64 0.295395 0.836421
531 16 10163 10147 76.4266 88 0.254517 0.836641
532 16 10176 10160 76.3806 52 0.516377 0.836685
533 16 10197 10181 76.3949 84 0.345804 0.836966
534 16 10222 10206 76.4391 100 0.843198 0.836562
535 16 10235 10219 76.3934 52 0.934714 0.836863
536 16 10256 10240 76.4076 84 0.75833 0.836771
537 16 10273 10257 76.3919 68 0.509247 0.83682
538 16 10293 10277 76.3986 80 0.427609 0.83696
539 16 10307 10291 76.3607 56 1.6499 0.837218
2020-02-14 22:49:53.626796 min lat: 0.212625 max lat: 2.86714 avg lat: 0.837241
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 10331 10315 76.3971 96 0.799426 0.837241
541 16 10352 10336 76.4111 84 1.18009 0.836969
542 16 10369 10353 76.3956 68 0.841291 0.836936
543 16 10388 10372 76.3948 76 0.700394 0.836782
544 16 10410 10394 76.4161 88 0.569031 0.836636
545 16 10437 10421 76.4741 108 0.767351 0.836383
546 16 10456 10440 76.4732 76 0.423961 0.836316
547 16 10475 10459 76.4723 76 1.09299 0.836371
548 16 10489 10473 76.4349 56 1.18874 0.836428
549 16 10514 10498 76.4778 100 0.866337 0.836398
550 16 10532 10516 76.4697 72 0.357434 0.836193
551 16 10550 10534 76.4616 72 0.474326 0.836106
552 16 10570 10554 76.4679 80 0.640883 0.836397
553 16 10588 10572 76.4599 72 0.969547 0.836349
554 16 10611 10595 76.4879 92 0.332298 0.836141
555 16 10632 10616 76.5014 84 0.391454 0.835754
556 16 10654 10638 76.5221 88 0.583742 0.835846
557 16 10669 10653 76.4924 60 1.04246 0.835872
558 16 10683 10667 76.4556 56 0.848037 0.836048
559 16 10697 10681 76.419 56 1.69668 0.836434
2020-02-14 22:50:13.629206 min lat: 0.212625 max lat: 2.86714 avg lat: 0.836778
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 10719 10703 76.4397 88 0.334409 0.836778
561 16 10734 10718 76.4104 60 0.916444 0.836752
562 16 10752 10736 76.4025 72 0.512184 0.836799
563 16 10775 10759 76.4302 92 0.402044 0.836845
564 16 10796 10780 76.4436 84 1.02154 0.836821
565 16 10811 10795 76.4145 60 0.512833 0.836755
566 16 10828 10812 76.3996 68 0.339695 0.836879
567 16 10849 10833 76.413 84 0.649832 0.836802
568 16 10870 10854 76.4263 84 0.683415 0.836686
569 16 10887 10871 76.4115 68 1.32962 0.836689
570 16 10905 10889 76.4038 72 0.351057 0.836775
571 16 10930 10914 76.4451 100 0.328477 0.836584
572 16 10953 10937 76.4722 92 0.627209 0.836351
573 16 10972 10956 76.4714 76 1.06608 0.83628
574 16 10990 10974 76.4636 72 1.6052 0.836164
575 16 11007 10991 76.4489 68 0.43336 0.836122
576 16 11028 11012 76.462 84 1.04019 0.836395
577 16 11042 11026 76.4265 56 0.513065 0.836387
578 16 11067 11051 76.4673 100 0.889956 0.836545
579 16 11089 11073 76.4872 88 0.477137 0.836189
2020-02-14 22:50:33.631481 min lat: 0.197662 max lat: 2.86714 avg lat: 0.835974
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 11110 11094 76.5001 84 0.498484 0.835974
581 16 11129 11113 76.4992 76 0.602696 0.835825
582 16 11152 11136 76.5258 92 0.785831 0.835818
583 16 11165 11149 76.4838 52 0.564088 0.835714
584 16 11181 11165 76.4624 64 0.354155 0.835988
585 16 11194 11178 76.4205 52 0.350508 0.836059
586 16 11209 11193 76.3925 60 0.340539 0.836568
587 16 11236 11220 76.4463 108 0.900205 0.836523
588 16 11255 11239 76.4456 76 0.476522 0.83651
589 16 11276 11260 76.4584 84 0.350573 0.836479
590 16 11292 11276 76.4372 64 0.385901 0.836453
591 16 11307 11291 76.4094 60 0.440175 0.836418
592 16 11326 11310 76.4087 76 0.764558 0.836915
593 16 11342 11326 76.3878 64 0.460326 0.836891
594 16 11360 11344 76.3804 72 1.57908 0.837281
595 16 11371 11355 76.326 44 0.312726 0.83699
596 16 11393 11377 76.3455 88 0.870125 0.837722
597 16 11410 11394 76.3315 68 1.67967 0.838146
598 16 11427 11411 76.3176 68 1.02629 0.838155
599 16 11439 11423 76.2703 48 0.39205 0.837914
2020-02-14 22:50:53.633829 min lat: 0.197662 max lat: 2.86714 avg lat: 0.8383
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 16 11461 11445 76.2898 88 0.395209 0.8383
Total time run: 600.568810
Total writes made: 11462
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 76.341
Stddev Bandwidth: 14.5922
Max bandwidth (MB/sec): 116
Min bandwidth (MB/sec): 24
Average IOPS: 19
Stddev IOPS: 3
Max IOPS: 29
Min IOPS: 6
Average Latency(s): 0.838328
Stddev Latency(s): 0.38524
Max latency(s): 2.86714
Min latency(s): 0.197662
Sequential Read Test
Total time run: 281.951843
Total reads made: 11790
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 167.263
Average IOPS: 41
Stddev IOPS: 8
Max IOPS: 63
Min IOPS: 22
Average Latency(s): 0.382004
Max latency(s): 2.87508
Min latency(s): 0.0520804
rados bench -p testbench 600 seq | tee --append $(date +%F)-rados-test-read-seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 16 16 0 0 0 - 0
1 16 57 41 163.762 164 0.129241 0.231886
2 16 97 81 161.869 160 0.184329 0.292812
3 16 151 135 179.847 216 0.125822 0.307539
4 16 198 182 181.878 188 0.9544 0.32508
5 16 249 233 186.295 204 0.683037 0.320846
6 16 300 284 189.24 204 0.699804 0.325389
7 16 346 330 188.489 184 0.302995 0.32117
8 16 384 368 183.927 152 1.46503 0.322451
9 16 421 405 179.935 148 0.150797 0.329013
10 16 461 445 177.94 160 0.37658 0.344374
11 16 515 499 181.396 216 0.116553 0.345734
12 16 571 555 184.944 224 0.0848365 0.338793
13 16 621 605 186.101 200 0.0890021 0.337184
14 16 664 648 185.086 172 0.311093 0.332959
15 16 714 698 186.079 200 0.125842 0.332219
16 16 766 750 187.436 208 0.237308 0.331745
17 16 814 798 187.702 192 0.0555165 0.331155
18 16 857 841 186.828 172 1.47429 0.332085
19 16 910 894 188.151 212 0.123422 0.331608
2020-02-14 22:52:16.780349 min lat: 0.0536279 max lat: 2.31221 avg lat: 0.332673
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 946 930 185.942 144 0.190618 0.332673
21 16 986 970 184.705 160 1.57822 0.338705
22 16 1029 1013 184.117 172 0.287843 0.339931
23 16 1086 1070 186.013 228 0.134183 0.339682
24 16 1135 1119 186.426 196 0.399202 0.337652
25 16 1173 1157 185.048 152 0.0534487 0.337519
26 16 1214 1198 184.238 164 0.273811 0.339897
27 16 1243 1227 181.711 116 0.0752925 0.34182
28 16 1269 1253 178.936 104 0.0830936 0.346966
29 16 1300 1284 177.04 124 0.119312 0.349288
30 16 1352 1336 178.071 208 0.101779 0.354258
31 16 1425 1409 181.744 292 0.218366 0.3494
32 16 1472 1456 181.939 188 0.184445 0.347525
33 16 1504 1488 180.304 128 0.950406 0.348341
34 16 1542 1526 179.472 152 0.053534 0.35154
35 16 1582 1566 178.915 160 0.352144 0.350598
36 16 1611 1595 177.167 116 0.0530931 0.352296
37 16 1657 1641 177.351 184 0.115921 0.355825
38 16 1700 1684 177.209 172 0.0883738 0.355616
39 16 1728 1712 175.537 112 0.147714 0.357109
2020-02-14 22:52:36.785894 min lat: 0.0530931 max lat: 2.7562 avg lat: 0.358581
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1762 1746 174.549 136 0.118512 0.358581
41 16 1802 1786 174.193 160 0.215248 0.360302
42 16 1841 1825 173.759 156 0.135572 0.362869
43 16 1887 1871 173.997 184 0.227326 0.362961
44 16 1926 1910 173.587 156 0.210927 0.364243
45 16 1966 1950 173.285 160 0.114966 0.364115
46 16 2007 1991 173.083 164 0.0544832 0.364528
47 16 2038 2022 172.038 124 0.232185 0.366951
48 16 2081 2065 172.036 172 1.43975 0.368035
49 16 2126 2110 172.198 180 1.22513 0.368547
50 16 2171 2155 172.354 180 0.0792847 0.366669
51 16 2217 2201 172.562 184 0.195685 0.36519
52 16 2247 2231 171.551 120 0.203033 0.366924
53 16 2292 2276 171.71 180 1.89268 0.368939
54 16 2330 2314 171.345 152 0.110699 0.368449
55 16 2365 2349 170.774 140 0.140208 0.369659
56 16 2411 2395 171.01 184 0.171001 0.370589
57 16 2461 2445 171.518 200 0.0850895 0.368717
58 16 2503 2487 171.457 168 0.145698 0.369726
59 16 2544 2528 171.33 164 0.223245 0.36941
2020-02-14 22:52:56.794747 min lat: 0.0527629 max lat: 2.7562 avg lat: 0.368929
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 2587 2571 171.341 172 0.106952 0.368929
61 16 2626 2610 171.089 156 0.176093 0.369504
62 16 2669 2653 171.104 172 0.113683 0.370376
63 16 2719 2703 171.562 200 0.279275 0.370493
64 16 2756 2740 171.194 148 0.378959 0.369076
65 16 2805 2789 171.575 196 0.172054 0.370365
66 16 2838 2822 170.975 132 0.205098 0.370206
67 16 2870 2854 170.333 128 1.84009 0.371148
68 16 2908 2892 170.063 152 0.11592 0.371699
69 16 2933 2917 169.048 100 0.529488 0.372645
70 16 2969 2953 168.69 144 0.117464 0.373865
71 16 3020 3004 169.187 204 0.0532694 0.374888
72 16 3059 3043 169.003 156 0.660874 0.374974
73 16 3112 3096 169.592 212 0.578829 0.373939
74 16 3151 3135 169.408 156 0.208941 0.374238
75 16 3195 3179 169.495 176 0.0873402 0.374424
76 16 3230 3214 169.107 140 0.0862061 0.374801
77 16 3265 3249 168.728 140 1.30653 0.376305
78 16 3320 3304 169.385 220 0.106878 0.37544
79 16 3372 3356 169.874 208 0.170799 0.374659
2020-02-14 22:53:16.797724 min lat: 0.0527051 max lat: 2.93153 avg lat: 0.373459
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 3418 3402 170.05 184 0.0943033 0.373459
81 16 3469 3453 170.469 204 0.0885231 0.37343
82 16 3506 3490 170.194 148 0.0960379 0.373806
83 16 3558 3542 170.648 208 0.0745853 0.372642
84 16 3590 3574 170.141 128 0.131342 0.373684
85 16 3627 3611 169.88 148 0.167081 0.373973
86 16 3666 3650 169.718 156 0.0833341 0.373552
87 16 3713 3697 169.928 188 0.0763592 0.373354
88 16 3759 3743 170.088 184 0.370167 0.373698
89 16 3796 3780 169.839 148 0.126625 0.374097
90 16 3829 3813 169.419 132 0.152632 0.374099
91 16 3863 3847 169.051 136 0.10849 0.37519
92 16 3902 3886 168.909 156 0.647038 0.375851
93 16 3930 3914 168.297 112 0.0834804 0.377148
94 16 3966 3950 168.039 144 0.0715648 0.376628
95 16 4002 3986 167.785 144 0.148089 0.378237
96 16 4026 4010 167.038 96 0.557916 0.379146
97 16 4054 4038 166.47 112 2.42487 0.380597
98 16 4084 4068 165.996 120 0.7027 0.381661
99 16 4107 4091 165.249 92 2.55571 0.383561
2020-02-14 22:53:36.800824 min lat: 0.052318 max lat: 2.93153 avg lat: 0.384145
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 4145 4129 165.116 152 2.55698 0.384145
101 16 4172 4156 164.55 108 0.109822 0.386184
102 16 4209 4193 164.388 148 0.0825119 0.386484
103 16 4234 4218 163.763 100 0.0544984 0.386711
104 16 4267 4251 163.457 132 0.118032 0.387648
105 16 4305 4289 163.348 152 2.48676 0.388984
106 16 4336 4320 162.977 124 0.0748831 0.389095
107 16 4380 4364 163.098 176 2.18509 0.389482
108 16 4422 4406 163.143 168 0.191207 0.390398
109 16 4461 4445 163.078 156 0.0549118 0.389534
110 16 4493 4477 162.759 128 0.109156 0.389802
111 16 4532 4516 162.697 156 0.0811559 0.39023
112 16 4570 4554 162.602 152 0.0779582 0.390543
113 16 4621 4605 162.968 204 0.2167 0.390451
114 16 4664 4648 163.047 172 0.352362 0.390079
115 16 4714 4698 163.368 200 0.326338 0.389573
116 16 4748 4732 163.132 136 0.467122 0.389743
117 16 4763 4747 162.25 60 0.576711 0.390551
118 16 4802 4786 162.197 156 0.0927156 0.391904
119 16 4846 4830 162.313 176 0.053048 0.391875
2020-02-14 22:53:56.803423 min lat: 0.0522791 max lat: 3.33647 avg lat: 0.392533
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 4880 4864 162.094 136 0.204178 0.392533
121 16 4924 4908 162.209 176 0.143887 0.392196
122 16 4958 4942 161.994 136 0.9467 0.392456
123 16 5008 4992 162.302 200 0.183683 0.392599
124 16 5055 5039 162.509 188 0.0553233 0.391997
125 16 5086 5070 162.201 124 0.0835045 0.392034
126 16 5129 5113 162.279 172 0.0974787 0.392105
127 16 5172 5156 162.355 172 0.0898608 0.391403
128 16 5222 5206 162.649 200 0.114492 0.391675
129 16 5265 5249 162.721 172 0.18769 0.391338
130 16 5302 5286 162.608 148 0.125773 0.39083
131 16 5334 5318 162.343 128 1.33508 0.392092
132 16 5366 5350 162.083 128 2.01918 0.392181
133 16 5405 5389 162.037 156 0.176213 0.392289
134 16 5434 5418 161.694 116 0.171706 0.392735
135 16 5466 5450 161.444 128 0.173966 0.393531
136 16 5512 5496 161.61 184 0.14199 0.394112
137 16 5543 5527 161.335 124 0.16925 0.394203
138 16 5586 5570 161.412 172 0.158254 0.394102
139 16 5611 5595 160.97 100 0.218431 0.39481
2020-02-14 22:54:16.806208 min lat: 0.0522791 max lat: 3.33647 avg lat: 0.395208
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 5641 5625 160.677 120 0.379187 0.395208
141 16 5695 5679 161.07 216 0.837289 0.396139
142 16 5742 5726 161.259 188 0.149888 0.395531
143 16 5787 5771 161.39 180 0.0804189 0.394532
144 16 5826 5810 161.353 156 0.244229 0.394356
145 16 5865 5849 161.315 156 0.0596529 0.39442
146 16 5907 5891 161.361 168 0.246574 0.394459
147 16 5954 5938 161.542 188 0.128342 0.394449
148 16 5994 5978 161.531 160 2.04655 0.394368
149 16 6026 6010 161.306 128 0.0537127 0.394071
150 16 6043 6027 160.684 68 2.22473 0.395402
151 16 6071 6055 160.362 112 0.0753619 0.396245
152 16 6107 6091 160.254 144 0.0830832 0.397646
153 16 6156 6140 160.487 196 0.181396 0.396943
154 16 6210 6194 160.847 216 0.165789 0.396172
155 16 6247 6231 160.764 148 1.41704 0.396191
156 16 6295 6279 160.965 192 0.1358 0.395222
157 16 6346 6330 161.238 204 0.131862 0.395398
158 16 6393 6377 161.408 188 1.85512 0.394768
159 16 6423 6407 161.147 120 0.0604151 0.394635
2020-02-14 22:54:36.809585 min lat: 0.0522791 max lat: 3.33647 avg lat: 0.395652
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 6447 6431 160.739 96 0.20415 0.395652
161 16 6485 6469 160.685 152 0.389969 0.395521
162 16 6517 6501 160.482 128 0.082259 0.396498
163 16 6561 6545 160.578 176 1.56243 0.396758
164 16 6606 6590 160.696 180 0.448176 0.396284
165 16 6658 6642 160.982 208 0.257093 0.395946
166 16 6695 6679 160.904 148 0.145277 0.39643
167 16 6751 6735 161.281 224 0.19578 0.395502
168 16 6805 6789 161.607 216 0.201983 0.394765
169 16 6854 6838 161.81 196 0.26439 0.394122
170 16 6896 6880 161.846 168 0.163156 0.393913
171 16 6941 6925 161.952 180 0.191756 0.393503
172 16 6987 6971 162.08 184 0.383641 0.393252
173 16 7025 7009 162.022 152 0.056629 0.393576
174 16 7065 7049 162.01 160 0.151143 0.393185
175 16 7105 7089 161.999 160 0.0823962 0.393213
176 16 7142 7126 161.919 148 0.164627 0.393362
177 16 7184 7168 161.953 168 0.162608 0.393676
178 16 7223 7207 161.92 156 0.122373 0.393718
179 16 7267 7251 161.998 176 0.183036 0.393734
2020-02-14 22:54:56.813178 min lat: 0.0522791 max lat: 3.33647 avg lat: 0.393105
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 7317 7301 162.209 200 0.192651 0.393105
181 16 7348 7332 161.998 124 0.456634 0.393201
182 16 7380 7364 161.811 128 0.228963 0.393637
183 16 7409 7393 161.561 116 0.0783955 0.393641
184 16 7458 7442 161.748 196 0.314484 0.394076
185 16 7508 7492 161.954 200 0.252043 0.393712
186 16 7549 7533 161.965 164 0.153378 0.393834
187 16 7583 7567 161.826 136 0.963346 0.393743
188 16 7606 7590 161.455 92 0.0742535 0.394345
189 16 7635 7619 161.214 116 0.117178 0.394577
190 16 7679 7663 161.292 176 2.24898 0.394879
191 16 7715 7699 161.201 144 0.0540177 0.395185
192 16 7755 7739 161.195 160 0.0695364 0.39562
193 16 7792 7776 161.127 148 0.0838847 0.395552
194 16 7830 7814 161.079 152 2.13427 0.395695
195 16 7884 7868 161.361 216 0.172549 0.395417
196 16 7922 7906 161.313 152 0.129746 0.394778
197 16 7968 7952 161.428 184 0.0835599 0.395084
198 16 8016 8000 161.582 192 0.449698 0.394556
199 16 8065 8049 161.755 196 0.299544 0.394278
2020-02-14 22:55:16.815903 min lat: 0.0521896 max lat: 3.33647 avg lat: 0.394235
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 8100 8084 161.646 140 0.0802843 0.394235
201 16 8133 8117 161.499 132 2.46951 0.394181
202 16 8169 8153 161.412 144 2.26804 0.394491
203 16 8206 8190 161.346 148 0.112469 0.394644
204 16 8244 8228 161.3 152 0.185016 0.395308
205 16 8291 8275 161.43 188 0.324219 0.395135
206 16 8330 8314 161.403 156 0.117121 0.3951
207 16 8373 8357 161.455 172 0.189234 0.394841
208 16 8403 8387 161.255 120 0.0831808 0.39493
209 16 8448 8432 161.345 180 0.0805557 0.395204
210 16 8496 8480 161.491 192 0.291093 0.395131
211 16 8547 8531 161.691 204 0.135836 0.394552
212 16 8592 8576 161.778 180 0.0848451 0.394114
213 16 8624 8608 161.619 128 0.164053 0.394098
214 16 8649 8633 161.331 100 0.667972 0.39441
215 16 8693 8677 161.399 176 0.221579 0.394985
216 16 8745 8729 161.615 208 0.159074 0.394847
217 16 8786 8770 161.626 164 0.0703386 0.394809
218 16 8832 8816 161.728 184 1.15555 0.394505
219 16 8880 8864 161.866 192 0.161807 0.394192
2020-02-14 22:55:36.819200 min lat: 0.0521896 max lat: 3.33647 avg lat: 0.394078
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 8910 8894 161.676 120 0.196688 0.394078
221 16 8939 8923 161.469 116 0.572676 0.394209
222 16 8988 8972 161.625 196 0.167857 0.394582
223 16 9034 9018 161.725 184 0.167536 0.394446
224 16 9078 9062 161.789 176 1.086 0.394315
225 16 9105 9089 161.549 108 0.598772 0.394075
226 16 9138 9122 161.419 132 0.132201 0.394585
227 16 9173 9157 161.324 140 2.24971 0.395072
228 16 9214 9198 161.336 164 0.0542014 0.395063
229 16 9256 9240 161.365 168 0.0648941 0.395354
230 16 9287 9271 161.202 124 0.392404 0.395527
231 16 9320 9304 161.076 132 0.0854787 0.395517
232 16 9366 9350 161.174 184 0.0846252 0.395867
233 16 9407 9391 161.187 164 0.0947143 0.395694
234 16 9442 9426 161.096 140 0.0752536 0.395713
235 16 9483 9467 161.108 164 0.36338 0.395553
236 16 9525 9509 161.137 168 0.055651 0.395424
237 16 9566 9550 161.149 164 0.1813 0.396044
238 16 9609 9593 161.195 172 0.0759368 0.39573
239 16 9654 9638 161.273 180 0.723772 0.395581
2020-02-14 22:55:56.822604 min lat: 0.0521896 max lat: 3.33647 avg lat: 0.395281
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 9698 9682 161.334 176 0.0660823 0.395281
241 16 9734 9718 161.262 144 0.306338 0.39526
242 16 9766 9750 161.125 128 0.119929 0.395607
243 16 9804 9788 161.087 152 0.140319 0.395859
244 16 9838 9822 160.984 136 0.303154 0.396034
245 16 9879 9863 160.996 164 0.0816013 0.396404
246 16 9919 9903 160.992 160 0.149237 0.396281
247 16 9953 9937 160.891 136 0.267473 0.39654
248 16 10003 9987 161.048 200 0.361956 0.396246
249 16 10039 10023 160.98 144 0.358183 0.396181
250 16 10077 10061 160.944 152 0.145689 0.396173
251 16 10110 10094 160.829 132 1.91914 0.396246
252 16 10138 10122 160.635 112 0.103371 0.39651
253 16 10191 10175 160.838 212 0.253354 0.396717
254 16 10224 10208 160.724 132 0.13338 0.396788
255 16 10255 10239 160.58 124 0.786144 0.397227
256 16 10283 10267 160.39 112 0.304134 0.39737
257 16 10324 10308 160.404 164 2.21629 0.397522
258 16 10368 10352 160.464 176 0.194805 0.397428
259 16 10401 10385 160.354 132 0.142079 0.397412
2020-02-14 22:56:16.825355 min lat: 0.0521896 max lat: 3.33647 avg lat: 0.397512
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 10434 10418 160.245 132 0.165367 0.397512
261 16 10472 10456 160.214 152 0.291335 0.397748
262 16 10514 10498 160.243 168 0.111663 0.397835
263 16 10550 10534 160.181 144 0.175979 0.398148
264 16 10579 10563 160.014 116 0.746378 0.398033
265 16 10615 10599 159.954 144 0.118493 0.398358
266 16 10642 10626 159.758 108 1.27238 0.399306
267 16 10683 10667 159.774 164 0.116568 0.399076
268 16 10717 10701 159.685 136 0.361172 0.398987
269 16 10742 10726 159.463 100 0.344828 0.399539
270 16 10785 10769 159.51 172 2.06475 0.399972
271 16 10832 10816 159.615 188 1.56799 0.399788
272 16 10884 10868 159.792 208 0.208132 0.39934
273 16 10933 10917 159.925 196 0.0763828 0.398955
274 16 10970 10954 159.881 148 1.65138 0.399037
275 16 11005 10989 159.809 140 0.19707 0.399036
276 16 11045 11029 159.81 160 0.185064 0.399161
277 16 11076 11060 159.68 124 0.132775 0.398999
278 16 11116 11100 159.681 160 0.379721 0.399561
279 16 11159 11143 159.726 172 0.232735 0.399369
2020-02-14 22:56:36.828003 min lat: 0.0521896 max lat: 3.33647 avg lat: 0.399405
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 11200 11184 159.741 164 0.111902 0.399405
281 16 11236 11220 159.685 144 0.0533658 0.399468
282 16 11280 11264 159.742 176 0.0548492 0.399446
283 16 11326 11310 159.828 184 0.118763 0.399322
284 16 11358 11342 159.715 128 2.06554 0.399361
285 16 11395 11379 159.674 148 0.379412 0.399419
286 16 11438 11422 159.717 172 0.186498 0.399545
287 7 11462 11455 159.62 132 1.25933 0.399789
Total time run: 287.165268
Total reads made: 11462
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 159.657
Average IOPS: 39
Stddev IOPS: 8
Max IOPS: 73
Min IOPS: 15
Average Latency(s): 0.400031
Max latency(s): 3.33647
Min latency(s): 0.0521896
Random Read Test
Total time run: 600.851506
Total reads made: 23638
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 157.363
Average IOPS: 39
Stddev IOPS: 7
Max IOPS: 63
Min IOPS: 17
Average Latency(s): 0.406183
Max latency(s): 3.04659
Min latency(s): 0.0372216
rados bench -p testbench 600 rand | tee --append $(date +%F)-rados-test-read-rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 16 16 0 0 0 - 0
1 16 60 44 175.852 176 0.123451 0.284632
2 16 98 82 163.903 152 0.863719 0.297597
3 16 139 123 163.925 164 0.0834313 0.332324
4 16 165 149 148.945 104 0.122863 0.37049
5 16 208 192 153.527 172 0.128988 0.386854
6 16 250 234 155.867 168 0.150761 0.378545
7 16 302 286 163.296 208 0.0558394 0.375426
8 16 345 329 164.38 172 1.35706 0.367942
9 16 403 387 171.886 232 0.37122 0.361504
10 16 440 424 169.497 148 0.0809844 0.356292
11 16 486 470 170.812 184 0.0809962 0.351863
12 16 529 513 170.91 172 0.150925 0.359943
13 16 586 570 175.297 228 0.0743829 0.357099
14 16 629 613 175.06 172 0.0376657 0.352007
15 16 672 656 174.855 172 0.186661 0.350654
16 16 704 688 171.926 128 0.435999 0.356971
17 16 746 730 171.694 168 1.55275 0.362453
18 16 782 766 170.155 144 0.205478 0.363302
19 16 825 809 170.251 172 0.164933 0.362713
2020-02-14 22:57:40.315788 min lat: 0.0373792 max lat: 2.35095 avg lat: 0.368081
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 858 842 168.337 132 1.64804 0.368081
21 16 907 891 169.651 196 0.244302 0.369134
22 16 948 932 169.393 164 0.0379065 0.367461
23 16 980 964 167.593 128 0.0777448 0.370489
24 16 1011 995 165.777 124 0.142145 0.374253
25 16 1053 1037 165.865 168 0.126759 0.377375
26 16 1078 1062 163.332 100 0.0824319 0.381133
27 16 1127 1111 164.541 196 0.131991 0.382737
28 16 1167 1151 164.378 160 0.321278 0.382344
29 16 1200 1184 163.261 132 0.0375643 0.380588
30 16 1248 1232 164.218 192 0.274339 0.378903
31 16 1289 1273 164.208 164 0.0375938 0.380145
32 16 1326 1310 163.7 148 0.189664 0.384006
33 16 1369 1353 163.95 172 0.103673 0.384065
34 16 1406 1390 163.481 148 0.143072 0.382786
35 16 1448 1432 163.609 168 0.313507 0.381893
36 16 1481 1465 162.731 132 0.11719 0.386529
37 16 1510 1494 161.468 116 0.112235 0.387758
38 16 1549 1533 161.323 156 0.160397 0.387332
39 16 1583 1567 160.674 136 0.146137 0.388776
2020-02-14 22:58:00.319211 min lat: 0.0373792 max lat: 3.3282 avg lat: 0.391134
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1622 1606 160.556 156 0.365622 0.391134
41 16 1669 1653 161.225 188 2.06127 0.392502
42 16 1706 1690 160.91 148 0.45716 0.391878
43 16 1737 1721 160.051 124 0.180832 0.392935
44 16 1774 1758 159.777 148 0.123429 0.394191
45 16 1815 1799 159.87 164 0.0928125 0.393497
46 16 1852 1836 159.612 148 0.134454 0.396411
47 16 1909 1893 161.066 228 0.105589 0.393923
48 16 1949 1933 161.043 160 0.109947 0.392866
49 16 1994 1978 161.43 180 0.761015 0.393902
50 16 2042 2026 162.041 192 0.154462 0.391935
51 16 2084 2068 162.157 168 0.0373792 0.389744
52 16 2116 2100 161.5 128 2.10556 0.390347
53 16 2146 2130 160.717 120 0.230573 0.391941
54 16 2192 2176 161.147 184 0.189078 0.39364
55 16 2232 2216 161.126 160 1.12757 0.394596
56 16 2278 2262 161.534 184 0.370259 0.392672
57 16 2332 2316 162.489 216 0.164873 0.391194
58 16 2364 2348 161.894 128 1.65333 0.390781
59 16 2408 2392 162.133 176 0.273545 0.391056
2020-02-14 22:58:20.321771 min lat: 0.0373792 max lat: 3.3282 avg lat: 0.390399
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 2460 2444 162.897 208 0.187673 0.390399
61 16 2500 2484 162.849 160 0.118758 0.388975
62 16 2536 2520 162.543 144 0.133707 0.388743
63 16 2586 2570 163.137 200 0.432153 0.388444
64 16 2614 2598 162.338 112 1.94405 0.389075
65 16 2646 2630 161.81 128 1.9782 0.390416
66 16 2691 2675 162.085 180 1.96478 0.391036
67 16 2742 2726 162.71 204 0.221265 0.390446
68 16 2806 2790 164.081 256 0.246395 0.388779
69 16 2853 2837 164.428 188 0.0392827 0.38626
70 16 2890 2874 164.193 148 0.0791218 0.385767
71 16 2937 2921 164.528 188 0.201076 0.385815
72 16 2968 2952 163.965 124 0.183829 0.386282
73 16 3007 2991 163.855 156 0.144999 0.386893
74 16 3043 3027 163.587 144 2.01545 0.387724
75 16 3084 3068 163.592 164 0.0382203 0.388117
76 16 3121 3105 163.384 148 0.796388 0.387997
77 16 3158 3142 163.184 148 0.158849 0.38875
78 16 3189 3173 162.682 124 0.529617 0.389487
79 16 3227 3211 162.546 152 0.151977 0.389479
2020-02-14 22:58:40.325971 min lat: 0.0373792 max lat: 3.3282 avg lat: 0.391062
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 3263 3247 162.314 144 0.144937 0.391062
81 16 3306 3290 162.434 172 0.204456 0.390011
82 16 3342 3326 162.209 144 1.52041 0.391783
83 16 3387 3371 162.423 180 0.0884655 0.391357
84 16 3439 3423 162.965 208 0.213099 0.39052
85 16 3480 3464 162.977 164 0.0870987 0.390159
86 16 3517 3501 162.803 148 0.150084 0.389389
87 16 3548 3532 162.357 124 2.00885 0.390456
88 16 3579 3563 161.917 124 1.47279 0.392481
89 16 3627 3611 162.254 192 0.240604 0.391945
90 16 3671 3655 162.407 176 1.42438 0.391685
91 16 3718 3702 162.688 188 0.112953 0.39162
92 16 3761 3745 162.787 172 1.60579 0.390767
93 16 3802 3786 162.8 164 0.259297 0.391066
94 16 3837 3821 162.557 140 0.824924 0.390758
95 16 3879 3863 162.614 168 0.191065 0.390873
96 16 3908 3892 162.129 116 0.222965 0.391379
97 16 3944 3928 161.942 144 1.70787 0.392335
98 16 3993 3977 162.289 196 0.135496 0.391873
99 16 4037 4021 162.427 176 0.263083 0.392134
2020-02-14 22:59:00.331314 min lat: 0.0373792 max lat: 3.3282 avg lat: 0.391696
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 4079 4063 162.483 168 0.456687 0.391696
101 16 4107 4091 161.983 112 0.122295 0.391489
102 16 4140 4124 161.689 132 0.334466 0.392019
103 16 4174 4158 161.439 136 0.652835 0.392803
104 16 4205 4189 161.079 124 0.644701 0.3945
105 16 4252 4236 161.335 188 0.164368 0.394728
106 16 4290 4274 161.247 152 0.258417 0.394372
107 16 4326 4310 161.086 144 0.197363 0.394643
108 16 4375 4359 161.409 196 1.30522 0.394651
109 16 4411 4395 161.249 144 0.533143 0.39413
110 16 4452 4436 161.274 164 0.309631 0.393722
111 16 4499 4483 161.514 188 2.58867 0.394114
112 16 4542 4526 161.606 172 0.112219 0.394117
113 16 4589 4573 161.84 188 0.259756 0.393166
114 16 4621 4605 161.543 128 0.130345 0.393359
115 16 4662 4646 161.564 164 0.110149 0.393394
116 16 4694 4678 161.274 128 0.130513 0.393399
117 16 4733 4717 161.229 156 0.243465 0.394524
118 16 4770 4754 161.117 148 0.46596 0.394992
119 16 4817 4801 161.342 188 0.225602 0.394268
2020-02-14 22:59:20.334784 min lat: 0.0373792 max lat: 3.3282 avg lat: 0.39362
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 4875 4859 161.931 232 0.200754 0.39362
121 16 4918 4902 162.014 172 0.0860285 0.39267
122 16 4954 4938 161.866 144 0.0937885 0.392753
123 16 4981 4965 161.428 108 0.128134 0.393598
124 16 5019 5003 161.352 152 0.109069 0.393893
125 16 5056 5040 161.244 148 0.106514 0.394211
126 16 5094 5078 161.171 152 0.15851 0.3945
127 16 5131 5115 161.067 148 0.150916 0.394999
128 16 5173 5157 161.121 168 0.0402077 0.394969
129 16 5201 5185 160.74 112 1.70641 0.39541
130 16 5226 5210 160.273 100 0.0372624 0.394885
131 16 5255 5239 159.935 116 0.1341 0.395679
132 16 5287 5271 159.693 128 0.142474 0.396775
133 16 5339 5323 160.056 208 0.144811 0.39812
134 16 5379 5363 160.055 160 0.119604 0.397925
135 16 5410 5394 159.788 124 0.213036 0.398429
136 16 5454 5438 159.907 176 0.143864 0.398316
137 16 5509 5493 160.346 220 0.111212 0.397563
138 16 5554 5538 160.488 180 0.224602 0.396967
139 16 5608 5592 160.887 216 0.138324 0.396405
2020-02-14 22:59:40.337965 min lat: 0.0372624 max lat: 3.7977 avg lat: 0.395409
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 5659 5643 161.194 204 0.125527 0.395409
141 16 5709 5693 161.469 200 0.0801818 0.394802
142 16 5751 5735 161.515 168 0.0769544 0.394184
143 16 5790 5774 161.477 156 0.25902 0.393683
144 16 5830 5814 161.466 160 1.29159 0.394733
145 16 5871 5855 161.484 164 0.728232 0.394127
146 16 5908 5892 161.391 148 0.451801 0.394311
147 16 5955 5939 161.572 188 0.283553 0.394551
148 16 5997 5981 161.615 168 0.295431 0.394385
149 16 6047 6031 161.873 200 0.132618 0.393483
150 16 6077 6061 161.593 120 0.5996 0.393727
151 16 6110 6094 161.397 132 0.863301 0.393567
152 16 6136 6120 161.02 104 0.179123 0.395037
153 16 6177 6161 161.039 164 0.22298 0.395764
154 16 6225 6209 161.24 192 0.200634 0.395399
155 16 6262 6246 161.154 148 0.108137 0.395019
156 16 6293 6277 160.916 124 1.84742 0.395523
157 16 6323 6307 160.655 120 0.205733 0.396288
158 16 6367 6351 160.752 176 0.0844001 0.396248
159 16 6409 6393 160.798 168 0.50952 0.395807
2020-02-14 23:00:00.340584 min lat: 0.0372624 max lat: 3.7977 avg lat: 0.395747
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 6454 6438 160.918 180 0.0794568 0.395747
161 16 6498 6482 161.011 176 0.149318 0.395461
162 16 6530 6514 160.807 128 2.30604 0.395735
163 16 6568 6552 160.753 152 0.0762198 0.395978
164 16 6610 6594 160.797 168 2.47345 0.395859
165 16 6656 6640 160.938 184 0.231167 0.39583
166 16 6693 6677 160.86 148 0.925372 0.39579
167 16 6733 6717 160.854 160 0.124948 0.396171
168 16 6784 6768 161.111 204 0.0710534 0.395863
169 16 6831 6815 161.27 188 0.322445 0.395155
170 16 6870 6854 161.238 156 0.161645 0.395137
171 16 6919 6903 161.441 196 0.0847008 0.394955
172 16 6965 6949 161.572 184 0.228725 0.394588
173 16 6996 6980 161.355 124 0.31388 0.394539
174 16 7029 7013 161.186 132 1.94047 0.394871
175 16 7068 7052 161.155 156 0.493038 0.39515
176 16 7104 7088 161.058 144 1.74579 0.395366
177 16 7149 7133 161.165 180 1.46472 0.395465
178 16 7189 7173 161.158 160 0.338431 0.395736
179 16 7232 7216 161.218 172 0.310278 0.395447
2020-02-14 23:00:20.345500 min lat: 0.037177 max lat: 3.7977 avg lat: 0.395254
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 7283 7267 161.456 204 0.899701 0.395254
181 16 7320 7304 161.381 148 0.890878 0.394698
182 16 7347 7331 161.088 108 0.091821 0.395509
183 16 7380 7364 160.929 132 0.385622 0.39591
184 16 7416 7400 160.837 144 0.181629 0.396281
185 16 7444 7428 160.573 112 0.143413 0.396619
186 16 7480 7464 160.483 144 0.145263 0.396627
187 16 7522 7506 160.523 168 0.204332 0.396952
188 16 7567 7551 160.626 180 0.0373661 0.396876
189 16 7598 7582 160.433 124 0.235743 0.39709
190 16 7631 7615 160.283 132 0.14132 0.397291
191 16 7678 7662 160.428 188 0.126972 0.397361
192 16 7726 7710 160.592 192 1.82952 0.397006
193 16 7775 7759 160.775 196 0.128593 0.396893
194 16 7818 7802 160.833 172 0.0381462 0.39635
195 16 7873 7857 161.136 220 0.228944 0.396338
196 16 7919 7903 161.253 184 0.168294 0.395813
197 16 7975 7959 161.571 224 0.720571 0.394958
198 16 8025 8009 161.765 200 0.235503 0.394653
199 16 8068 8052 161.816 172 0.130741 0.394229
2020-02-14 23:00:40.349181 min lat: 0.037177 max lat: 3.7977 avg lat: 0.394002
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 8109 8093 161.827 164 1.74702 0.394002
201 16 8147 8131 161.778 152 0.081515 0.393947
202 16 8189 8173 161.809 168 0.190654 0.394281
203 16 8227 8211 161.76 152 1.1193 0.394449
204 16 8269 8253 161.791 168 0.0375981 0.3943
205 16 8313 8297 161.86 176 0.363053 0.39391
206 16 8355 8339 161.89 168 0.321838 0.393566
207 16 8390 8374 161.784 140 2.76282 0.393724
208 16 8429 8413 161.756 156 0.286283 0.393847
209 16 8467 8451 161.709 152 1.8951 0.394293
210 16 8503 8487 161.625 144 0.322017 0.394304
211 16 8527 8511 161.314 96 0.158668 0.394721
212 16 8561 8545 161.194 136 0.371475 0.395156
213 16 8596 8580 161.095 140 0.221386 0.395608
214 16 8648 8632 161.314 208 0.257262 0.395427
215 16 8690 8674 161.345 168 0.173525 0.39527
216 16 8730 8714 161.338 160 0.155188 0.395224
217 16 8780 8764 161.516 200 0.0805348 0.394928
218 16 8823 8807 161.564 172 0.241537 0.394776
219 16 8865 8849 161.594 168 0.288854 0.394875
2020-02-14 23:01:00.351723 min lat: 0.037177 max lat: 3.7977 avg lat: 0.394669
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 8904 8888 161.568 156 1.67448 0.394669
221 16 8954 8938 161.742 200 2.06527 0.394348
222 16 8990 8974 161.662 144 0.109834 0.394328
223 16 9024 9008 161.547 136 0.0802081 0.394393
224 16 9066 9050 161.576 168 0.121965 0.39467
225 16 9109 9093 161.622 172 0.13277 0.394397
226 16 9154 9138 161.703 180 1.82424 0.394558
227 16 9194 9178 161.695 160 0.163132 0.394204
228 16 9242 9226 161.828 192 0.141296 0.394206
229 16 9280 9264 161.785 152 0.297481 0.394122
230 16 9322 9306 161.812 168 0.193375 0.393794
231 16 9362 9346 161.804 160 0.44246 0.394038
232 16 9404 9388 161.831 168 0.116987 0.394084
233 16 9447 9431 161.874 172 0.169001 0.394086
234 16 9482 9466 161.781 140 0.108825 0.394071
235 16 9517 9501 161.688 140 0.1079 0.394095
236 16 9555 9539 161.647 152 0.422795 0.394104
237 16 9591 9575 161.572 144 0.238435 0.394421
238 16 9629 9613 161.532 152 0.297895 0.394591
239 16 9677 9661 161.659 192 0.127349 0.394341
2020-02-14 23:01:20.354282 min lat: 0.037177 max lat: 3.7977 avg lat: 0.39472
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 9704 9688 161.436 108 0.119046 0.39472
241 16 9742 9726 161.397 152 0.242698 0.395238
242 16 9786 9770 161.457 176 0.131604 0.395315
243 16 9824 9808 161.418 152 0.124007 0.395359
244 16 9864 9848 161.412 160 1.03336 0.395257
245 16 9902 9886 161.373 152 0.0845803 0.395326
246 16 9927 9911 161.124 100 0.0842903 0.395676
247 16 9956 9940 160.941 116 0.126562 0.39554
248 16 10000 9984 161.002 176 2.64659 0.395845
249 16 10029 10013 160.821 116 0.750484 0.396193
250 16 10070 10054 160.833 164 0.0835146 0.396892
251 16 10105 10089 160.75 140 0.834029 0.396741
252 16 10135 10119 160.589 120 0.245787 0.396793
253 16 10170 10154 160.507 140 0.108867 0.397467
254 16 10203 10187 160.395 132 0.160558 0.397559
255 16 10228 10212 160.158 100 0.172588 0.397845
256 16 10258 10242 160.001 120 0.145178 0.398206
257 16 10299 10283 160.016 164 0.122252 0.39863
258 16 10345 10329 160.109 184 0.117493 0.398606
259 16 10393 10377 160.232 192 0.0877103 0.398321
2020-02-14 23:01:40.358503 min lat: 0.037177 max lat: 3.7977 avg lat: 0.398218
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 10427 10411 160.138 136 0.190907 0.398218
261 16 10472 10456 160.214 180 0.2608 0.398185
262 16 10526 10510 160.427 216 0.310256 0.398138
263 16 10566 10550 160.425 160 0.144514 0.397781
264 16 10587 10571 160.136 84 0.0781688 0.397929
265 16 10627 10611 160.135 160 0.144636 0.398461
266 16 10656 10640 159.969 116 2.08071 0.398472
267 16 10684 10668 159.79 112 0.0776477 0.398869
268 16 10728 10712 159.85 176 0.254517 0.398997
269 16 10769 10753 159.865 164 0.209268 0.399161
270 16 10800 10784 159.733 124 0.0794802 0.399231
271 16 10846 10830 159.822 184 0.208712 0.399197
272 16 10887 10871 159.837 164 0.0382633 0.399228
273 16 10925 10909 159.808 152 0.111541 0.399158
274 16 10967 10951 159.838 168 0.24793 0.39922
275 16 11016 11000 159.97 196 0.14281 0.3991
276 16 11060 11044 160.028 176 0.164773 0.398994
277 16 11097 11081 159.984 148 0.0851005 0.398718
278 16 11141 11125 160.042 176 0.253983 0.398676
279 16 11187 11171 160.127 184 0.118874 0.398613
2020-02-14 23:02:00.361371 min lat: 0.037177 max lat: 3.7977 avg lat: 0.398341
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 11235 11219 160.241 192 0.44555 0.398341
281 16 11264 11248 160.084 116 0.23781 0.39855
282 16 11301 11285 160.041 148 0.111666 0.398319
283 16 11340 11324 160.026 156 0.113925 0.398296
284 16 11367 11351 159.843 108 0.695603 0.398577
285 16 11397 11381 159.703 120 0.0756108 0.399306
286 16 11428 11412 159.578 124 1.59796 0.399759
287 16 11460 11444 159.468 128 0.668478 0.399606
288 16 11482 11466 159.22 88 0.290673 0.399763
289 16 11527 11511 159.292 180 0.233183 0.400323
290 16 11568 11552 159.308 164 0.181735 0.400387
291 16 11612 11596 159.365 176 0.219351 0.400347
292 16 11656 11640 159.422 176 0.392674 0.400289
293 16 11701 11685 159.492 180 0.2016 0.400232
294 16 11743 11727 159.521 168 0.300506 0.4001
295 16 11782 11766 159.509 156 0.137937 0.400092
296 16 11815 11799 159.416 132 0.121194 0.400266
297 16 11853 11837 159.391 152 1.11964 0.400313
298 16 11871 11855 159.098 72 0.0792226 0.40053
299 16 11906 11890 159.034 140 0.114346 0.40103
2020-02-14 23:02:20.364637 min lat: 0.037177 max lat: 3.7977 avg lat: 0.40089
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 11961 11945 159.237 220 0.0830606 0.40089
301 16 12006 11990 159.306 180 0.108408 0.400556
302 16 12058 12042 159.467 208 0.191387 0.400489
303 16 12099 12083 159.482 164 0.128356 0.400154
304 16 12134 12118 159.418 140 0.0962723 0.400069
305 16 12164 12148 159.288 120 0.127027 0.400342
306 16 12194 12178 159.16 120 0.16311 0.400608
307 16 12233 12217 159.15 156 0.416693 0.400806
308 16 12277 12261 159.204 176 1.86024 0.400724
309 16 12313 12297 159.155 144 0.13534 0.400972
310 16 12365 12349 159.312 208 0.167827 0.40068
311 16 12414 12398 159.43 196 0.0956161 0.400265
312 16 12467 12451 159.598 212 0.16342 0.399962
313 16 12505 12489 159.574 152 0.0875105 0.399805
314 16 12559 12543 159.753 216 0.134114 0.399705
315 16 12603 12587 159.805 176 0.198315 0.399431
316 16 12649 12633 159.881 184 0.209453 0.399291
317 16 12702 12686 160.046 212 0.0782464 0.398918
318 16 12740 12724 160.02 152 0.0758649 0.398652
319 16 12780 12764 160.02 160 1.07874 0.398634
2020-02-14 23:02:40.368051 min lat: 0.037177 max lat: 3.7977 avg lat: 0.398741
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 12820 12804 160.02 160 0.0793648 0.398741
321 16 12855 12839 159.958 140 0.102653 0.398813
322 16 12899 12883 160.007 176 0.0853117 0.398653
323 16 12940 12924 160.02 164 0.114371 0.398739
324 16 12991 12975 160.155 204 0.355505 0.39867
325 16 13032 13016 160.167 164 0.149489 0.398462
326 16 13079 13063 160.253 188 0.139107 0.398219
327 16 13120 13104 160.264 164 0.422033 0.398165
328 16 13167 13151 160.348 188 0.925685 0.398264
329 16 13216 13200 160.457 196 1.40942 0.397964
330 16 13246 13230 160.334 120 0.0750039 0.397781
331 16 13252 13236 159.922 24 2.59722 0.398091
332 16 13305 13289 160.079 212 0.0846953 0.398721
333 16 13347 13331 160.103 168 0.116368 0.398734
334 16 13384 13368 160.066 148 0.116726 0.398807
335 16 13429 13413 160.126 180 0.179768 0.398707
336 16 13473 13457 160.173 176 0.366427 0.398531
337 16 13498 13482 159.994 100 0.242708 0.398625
338 16 13524 13508 159.829 104 0.262901 0.399015
339 16 13562 13546 159.806 152 1.4873 0.399297
2020-02-14 23:03:00.370616 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399439
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 13599 13583 159.771 148 0.243578 0.399439
341 16 13628 13612 159.642 116 0.122422 0.399548
342 16 13664 13648 159.597 144 0.203605 0.399703
343 16 13702 13686 159.574 152 0.0766511 0.400014
344 16 13747 13731 159.634 180 1.52715 0.399873
345 16 13795 13779 159.727 192 0.366039 0.399474
346 16 13841 13825 159.797 184 0.260386 0.399637
347 16 13889 13873 159.89 192 0.183705 0.399119
348 16 13924 13908 159.833 140 0.150892 0.399158
349 16 13961 13945 159.799 148 0.0381405 0.3995
350 16 13999 13983 159.777 152 0.157954 0.399424
351 16 14048 14032 159.88 196 0.10399 0.399199
352 16 14091 14075 159.914 172 0.726027 0.398989
353 16 14120 14104 159.79 116 0.0818657 0.399228
354 16 14150 14134 159.677 120 0.110338 0.399421
355 16 14197 14181 159.757 188 1.76671 0.399498
356 16 14223 14207 159.6 104 0.477281 0.399529
357 16 14257 14241 159.534 136 0.129476 0.399781
358 16 14288 14272 159.435 124 0.180709 0.40029
359 16 14329 14313 159.448 164 0.0875322 0.400258
2020-02-14 23:03:20.373028 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399981
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 14384 14368 159.616 220 0.146278 0.399981
361 16 14430 14414 159.683 184 0.229665 0.399846
362 16 14464 14448 159.618 136 0.11968 0.400016
363 16 14506 14490 159.641 168 0.391185 0.399769
364 16 14539 14523 159.565 132 0.357392 0.399884
365 16 14581 14565 159.588 168 0.115715 0.399998
366 16 14620 14604 159.578 156 0.441937 0.399781
367 16 14644 14628 159.405 96 0.426861 0.400008
368 16 14682 14666 159.385 152 0.088657 0.400141
369 16 14727 14711 159.44 180 0.56245 0.400236
370 16 14769 14753 159.464 168 0.419094 0.400239
371 16 14808 14792 159.454 156 0.114699 0.400303
372 16 14846 14830 159.434 152 0.397308 0.400321
373 16 14874 14858 159.307 112 0.247276 0.400412
374 16 14911 14895 159.277 148 0.195536 0.400649
375 16 14951 14935 159.278 160 0.121796 0.400688
376 16 14991 14975 159.28 160 0.073619 0.400694
377 16 15028 15012 159.25 148 0.0845578 0.400822
378 16 15065 15049 159.221 148 0.0779144 0.40086
379 16 15109 15093 159.265 176 0.16969 0.400802
2020-02-14 23:03:40.375367 min lat: 0.037177 max lat: 3.7977 avg lat: 0.400692
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 15157 15141 159.351 192 1.84038 0.400692
381 16 15198 15182 159.363 164 1.47763 0.400556
382 16 15236 15220 159.344 152 0.106555 0.400497
383 16 15275 15259 159.335 156 0.414832 0.400508
384 16 15318 15302 159.368 172 0.0423731 0.400559
385 16 15371 15355 159.504 212 0.181143 0.400256
386 16 15411 15395 159.506 160 0.258882 0.400208
387 16 15449 15433 159.486 152 0.118214 0.399985
388 16 15500 15484 159.601 204 0.176247 0.400139
389 16 15544 15528 159.643 176 0.602476 0.40003
390 16 15589 15573 159.695 180 1.69999 0.399822
391 16 15630 15614 159.706 164 0.20906 0.399637
392 16 15690 15674 159.911 240 0.138637 0.399445
393 16 15736 15720 159.972 184 0.12967 0.399093
394 16 15777 15761 159.982 164 0.975968 0.399107
395 16 15820 15804 160.013 172 0.0863053 0.398987
396 16 15855 15839 159.962 140 1.73136 0.399017
397 16 15884 15868 159.851 116 0.0723575 0.399227
398 16 15923 15907 159.841 156 0.170595 0.399218
399 16 15965 15949 159.862 168 0.157549 0.39924
2020-02-14 23:04:00.378050 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399188
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 15998 15982 159.792 132 0.510901 0.399188
401 16 16040 16024 159.813 168 0.0437401 0.399302
402 16 16083 16067 159.843 172 0.168033 0.399238
403 16 16120 16104 159.813 148 0.345317 0.399344
404 16 16159 16143 159.804 156 0.183047 0.399492
405 16 16198 16182 159.795 156 0.0394316 0.399513
406 16 16255 16239 159.962 228 0.219768 0.399318
407 16 16301 16285 160.021 184 0.161034 0.399045
408 16 16340 16324 160.012 156 0.170848 0.398944
409 16 16369 16353 159.904 116 1.5619 0.399074
410 16 16410 16394 159.914 164 0.222773 0.399063
411 16 16443 16427 159.846 132 2.20133 0.39934
412 16 16483 16467 159.846 160 0.125939 0.399433
413 16 16520 16504 159.818 148 0.120945 0.399436
414 16 16552 16536 159.741 128 0.12296 0.399584
415 16 16589 16573 159.712 148 0.584697 0.39955
416 16 16636 16620 159.78 188 0.124498 0.399635
417 16 16683 16667 159.848 188 0.0618612 0.399389
418 16 16717 16701 159.791 136 0.0873612 0.399501
419 16 16765 16749 159.868 192 0.260076 0.399289
2020-02-14 23:04:20.380228 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399332
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 16806 16790 159.877 164 1.90153 0.399332
421 16 16853 16837 159.944 188 1.263 0.399369
422 16 16899 16883 160.001 184 0.148121 0.39918
423 16 16939 16923 160.001 160 0.396006 0.399018
424 16 16967 16951 159.888 112 0.115557 0.399174
425 16 17005 16989 159.869 152 0.127425 0.399208
426 16 17034 17018 159.766 116 0.157153 0.399253
427 16 17079 17063 159.814 180 0.100375 0.399326
428 16 17129 17113 159.907 200 0.329485 0.399365
429 16 17173 17157 159.945 176 0.199182 0.399197
430 16 17220 17204 160.01 188 0.0822289 0.399088
431 16 17258 17242 159.991 152 0.32056 0.398906
432 16 17286 17270 159.88 112 0.517494 0.399051
433 16 17323 17307 159.852 148 0.248073 0.39906
434 16 17361 17345 159.834 152 0.0802191 0.399336
435 16 17394 17378 159.77 132 1.75602 0.3995
436 16 17430 17414 159.734 144 0.121855 0.399523
437 16 17458 17442 159.625 112 0.400538 0.399591
438 16 17500 17484 159.644 168 1.80967 0.399974
439 16 17546 17530 159.699 184 0.272027 0.399943
2020-02-14 23:04:40.383551 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399764
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 17590 17574 159.736 176 1.24767 0.399764
441 16 17628 17612 159.719 152 0.649968 0.399615
442 16 17666 17650 159.701 152 0.112506 0.399624
443 16 17708 17692 159.72 168 1.63763 0.399684
444 16 17732 17716 159.576 96 0.120362 0.399878
445 16 17767 17751 159.532 140 0.150185 0.400019
446 16 17813 17797 159.587 184 0.0470421 0.399966
447 16 17848 17832 159.543 140 0.257707 0.399965
448 16 17884 17868 159.509 144 0.173857 0.400361
449 16 17932 17916 159.581 192 0.193253 0.400293
450 16 17977 17961 159.626 180 0.202328 0.400058
451 16 18016 18000 159.618 156 0.138909 0.400007
452 16 18067 18051 159.716 204 0.106073 0.399858
453 16 18125 18109 159.876 232 0.1988 0.399573
454 16 18177 18161 159.982 208 0.0382571 0.399189
455 16 18215 18199 159.964 152 1.60289 0.399198
456 16 18246 18230 159.885 124 1.1903 0.39943
457 16 18297 18281 159.982 204 0.111288 0.399199
458 16 18334 18318 159.955 148 0.0711721 0.3991
459 16 18379 18363 159.999 180 0.461058 0.399224
2020-02-14 23:05:00.386041 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399022
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 18422 18406 160.025 172 1.1225 0.399022
461 16 18457 18441 159.982 140 0.0806834 0.398942
462 16 18500 18484 160.007 172 0.117183 0.399126
463 16 18545 18529 160.05 180 0.630182 0.398932
464 16 18581 18565 160.016 144 1.54936 0.398989
465 16 18611 18595 159.93 120 0.126494 0.399008
466 16 18651 18635 159.93 160 0.187751 0.399056
467 16 18688 18672 159.904 148 0.449994 0.399037
468 16 18725 18709 159.879 148 0.195858 0.399093
469 16 18770 18754 159.921 180 0.157345 0.399193
470 16 18803 18787 159.862 132 0.190713 0.399338
471 16 18845 18829 159.879 168 1.75981 0.399363
472 16 18875 18859 159.795 120 0.1132 0.39942
473 16 18914 18898 159.787 156 0.249205 0.399319
474 16 18960 18944 159.838 184 0.191665 0.399535
475 16 19012 18996 159.939 208 0.173457 0.399357
476 16 19033 19017 159.779 84 1.84464 0.399301
477 16 19060 19044 159.671 108 0.964735 0.399741
478 16 19094 19078 159.621 136 0.181253 0.400068
479 16 19136 19120 159.638 168 0.334372 0.399976
2020-02-14 23:05:20.391051 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399879
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 19190 19174 159.756 216 0.125889 0.399879
481 16 19230 19214 159.756 160 0.650473 0.3997
482 16 19267 19251 159.732 148 0.480142 0.399799
483 16 19319 19303 159.832 208 0.148037 0.399706
484 16 19376 19360 159.972 228 0.0523986 0.399397
485 16 19425 19409 160.047 196 0.0760418 0.399184
486 16 19485 19469 160.211 240 0.155393 0.398693
487 16 19534 19518 160.285 196 0.345075 0.398532
488 16 19569 19553 160.243 140 0.164762 0.398346
489 16 19604 19588 160.202 140 0.0783712 0.398602
490 16 19648 19632 160.234 176 0.0378586 0.398487
491 16 19678 19662 160.152 120 0.233977 0.398649
492 16 19726 19710 160.216 192 0.0997582 0.398701
493 16 19751 19735 160.094 100 0.60529 0.398624
494 16 19787 19771 160.061 144 0.236845 0.398846
495 16 19821 19805 160.013 136 0.119041 0.399088
496 16 19870 19854 160.085 196 1.86207 0.398926
497 16 19903 19887 160.029 132 0.43073 0.398934
498 16 19948 19932 160.069 180 0.250958 0.398887
499 16 19992 19976 160.101 176 0.236857 0.398962
2020-02-14 23:05:40.394422 min lat: 0.037177 max lat: 3.7977 avg lat: 0.398994
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 20014 19998 159.956 88 0.819045 0.398994
501 16 20055 20039 159.964 164 0.370929 0.399065
502 16 20101 20085 160.012 184 1.82925 0.399049
503 16 20131 20115 159.933 120 1.76174 0.399085
504 16 20167 20151 159.901 144 0.221506 0.399129
505 16 20208 20192 159.909 164 0.303542 0.399263
506 16 20245 20229 159.885 148 0.0892217 0.399339
507 16 20273 20257 159.791 112 0.368673 0.399401
508 16 20305 20289 159.728 128 0.133467 0.399454
509 16 20347 20331 159.744 168 0.151511 0.399908
510 16 20404 20388 159.878 228 0.226145 0.399611
511 16 20460 20444 160.004 224 0.0998638 0.399261
512 16 20517 20501 160.136 228 0.728087 0.39902
513 16 20568 20552 160.222 204 0.654557 0.398659
514 16 20615 20599 160.276 188 0.119261 0.398601
515 16 20659 20643 160.306 176 0.324779 0.398453
516 16 20709 20693 160.383 200 0.0373624 0.398206
517 16 20750 20734 160.39 164 0.530034 0.398125
518 16 20791 20775 160.397 164 0.881272 0.39807
519 16 20837 20821 160.443 184 2.24914 0.397978
2020-02-14 23:06:00.397664 min lat: 0.037177 max lat: 3.7977 avg lat: 0.39792
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 20883 20867 160.488 184 0.308555 0.39792
521 16 20932 20916 160.556 196 0.075896 0.397743
522 16 20963 20947 160.486 124 0.122013 0.397673
523 16 20996 20980 160.431 132 0.155341 0.397848
524 16 21043 21027 160.484 188 1.87746 0.398072
525 16 21089 21073 160.529 184 0.112876 0.397831
526 16 21134 21118 160.565 180 0.176155 0.397811
527 16 21174 21158 160.564 160 0.194761 0.397689
528 16 21213 21197 160.556 156 0.111733 0.397644
529 16 21251 21235 160.54 152 0.156289 0.397799
530 16 21276 21260 160.425 100 0.0980361 0.397866
531 16 21300 21284 160.304 96 0.291603 0.397951
532 16 21334 21318 160.258 136 0.476705 0.398449
533 16 21360 21344 160.153 104 2.18596 0.398563
534 16 21391 21375 160.085 124 2.37675 0.398714
535 16 21439 21423 160.144 192 0.122922 0.398903
536 16 21484 21468 160.181 180 0.189145 0.398757
537 16 21523 21507 160.174 156 1.1346 0.398735
538 16 21560 21544 160.151 148 1.74236 0.398757
539 16 21615 21599 160.262 220 0.209218 0.398616
2020-02-14 23:06:20.401240 min lat: 0.037177 max lat: 3.7977 avg lat: 0.398422
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 21662 21646 160.313 188 0.168682 0.398422
541 16 21692 21676 160.239 120 0.0384589 0.398558
542 16 21730 21714 160.223 152 0.0747618 0.398493
543 16 21757 21741 160.127 108 0.117671 0.398652
544 16 21795 21779 160.112 152 0.206234 0.398827
545 16 21840 21824 160.149 180 0.0733131 0.398636
546 16 21894 21878 160.251 216 0.118838 0.398582
547 16 21924 21908 160.177 120 1.66878 0.398576
548 16 21984 21968 160.323 240 0.125454 0.398488
549 16 22036 22020 160.41 208 0.645959 0.398198
550 16 22070 22054 160.365 136 0.171635 0.398285
551 16 22122 22106 160.452 208 0.121334 0.398098
552 16 22166 22150 160.48 176 0.17879 0.397975
553 16 22209 22193 160.5 172 0.0401149 0.397901
554 16 22250 22234 160.507 164 0.0938098 0.397835
555 16 22302 22286 160.592 208 0.123682 0.397779
556 16 22337 22321 160.555 140 0.0870244 0.397675
557 16 22370 22354 160.504 132 0.177097 0.397892
558 16 22418 22402 160.56 192 0.288839 0.397874
559 16 22467 22451 160.624 196 0.357258 0.397636
2020-02-14 23:06:40.404157 min lat: 0.037177 max lat: 3.7977 avg lat: 0.397569
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 22505 22489 160.608 152 0.184911 0.397569
561 16 22537 22521 160.55 128 2.02169 0.397682
562 16 22566 22550 160.471 116 1.05594 0.397722
563 16 22600 22584 160.427 136 1.48791 0.397947
564 16 22657 22641 160.547 228 0.221685 0.398042
565 16 22702 22686 160.581 180 0.130405 0.397764
566 16 22754 22738 160.665 208 0.193956 0.397543
567 16 22796 22780 160.678 168 0.158087 0.39733
568 16 22834 22818 160.663 152 0.109477 0.397427
569 16 22863 22847 160.584 116 0.126454 0.397471
570 16 22899 22883 160.555 144 0.247076 0.397482
571 16 22945 22929 160.596 184 0.166743 0.397661
572 16 22982 22966 160.574 148 1.47256 0.397634
573 16 23012 22996 160.503 120 0.136346 0.397875
574 16 23052 23036 160.502 160 0.0375928 0.397865
575 16 23108 23092 160.613 224 0.107713 0.397704
576 16 23161 23145 160.702 212 0.150169 0.397591
577 16 23209 23193 160.756 192 0.644658 0.397368
578 16 23242 23226 160.706 132 1.61862 0.397304
579 16 23277 23261 160.67 140 0.248773 0.397355
2020-02-14 23:07:00.406933 min lat: 0.037177 max lat: 3.7977 avg lat: 0.397411
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 23306 23290 160.593 116 0.0796421 0.397411
581 16 23345 23329 160.585 156 0.326844 0.397633
582 16 23389 23373 160.612 176 0.0858893 0.397783
583 16 23436 23420 160.659 188 0.111512 0.397561
584 16 23479 23463 160.678 172 0.0374978 0.397498
585 16 23513 23497 160.636 136 0.144081 0.397596
586 16 23551 23535 160.621 152 0.0732863 0.397519
587 16 23594 23578 160.641 172 0.140598 0.397588
588 16 23646 23630 160.721 208 0.112979 0.397386
589 16 23694 23678 160.774 192 0.173778 0.397287
590 16 23739 23723 160.806 180 0.132361 0.397262
591 16 23794 23778 160.907 220 0.190519 0.397091
592 16 23851 23835 161.02 228 0.183477 0.396797
593 16 23886 23870 160.984 140 0.0813829 0.396705
594 16 23917 23901 160.922 124 0.121268 0.396711
595 16 23946 23930 160.847 116 0.12366 0.396998
596 16 23995 23979 160.905 196 0.46652 0.397142
597 16 24058 24042 161.058 252 0.104585 0.396743
598 16 24097 24081 161.049 156 0.839443 0.396593
599 16 24119 24103 160.927 88 0.175991 0.396742
2020-02-14 23:07:20.410917 min lat: 0.037177 max lat: 3.7977 avg lat: 0.396839
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 16 24147 24131 160.846 112 0.561309 0.396839
Total time run: 600.975391
Total reads made: 24147
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 160.719
Average IOPS: 40
Stddev IOPS: 7
Max IOPS: 64
Min IOPS: 6
Average Latency(s): 0.397626
Max latency(s): 3.7977
Min latency(s): 0.037177
Addendum 2020-02-15
Like before, a 500GB block device used in a CentOS Stream VM.
Sequential Read, 4k Blocksize
READ: bw=81.6MiB/siops: avg=20881.32
fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4k
readtest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
readtest: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=92.8MiB/s,w=0KiB/s][r=23.7k,w=0 IOPS][eta 00m:01s]
readtest: (groupid=0, jobs=1): err= 0: pid=22260: Sat Feb 15 14:35:24 2020
read: IOPS=20.9k, BW=81.6MiB/s (85.5MB/s)(160GiB/2008932msec)
clat (nsec): min=1030, max=664744k, avg=46222.65, stdev=1647229.86
lat (nsec): min=1360, max=664745k, avg=46586.50, stdev=1647230.37
clat percentiles (nsec):
| 1.00th=[ 1096], 5.00th=[ 1144], 10.00th=[ 1192],
| 20.00th=[ 1272], 30.00th=[ 1784], 40.00th=[ 3760],
| 50.00th=[ 3856], 60.00th=[ 3984], 70.00th=[ 4080],
| 80.00th=[ 4320], 90.00th=[ 4640], 95.00th=[ 5088],
| 99.00th=[ 13376], 99.50th=[ 19072], 99.90th=[ 7700480],
| 99.95th=[35389440], 99.99th=[80216064]
bw ( KiB/s): min= 8192, max=143360, per=100.00%, avg=83525.31, stdev=17156.90, samples=4016
iops : min= 2048, max=35840, avg=20881.32, stdev=4289.23, samples=4016
lat (usec) : 2=30.67%, 4=31.84%, 10=35.97%, 20=1.08%, 50=0.30%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.02%, 50=0.04%
lat (msec) : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=1.85%, sys=10.16%, ctx=51916, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=41943040,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=81.6MiB/s (85.5MB/s), 81.6MiB/s-81.6MiB/s (85.5MB/s-85.5MB/s), io=160GiB (172GB), run=2008932-2008932msec
Disk stats (read/write):
vdc: ios=166858/7, merge=0/0, ticks=19324638/895, in_queue=19242019, util=9.64%
Sequential Read, 4m Blocksize
READ: bw=87.2MiB/siops: avg=21.80
fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4m
readtest: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=100MiB/s,w=0KiB/s][r=25,w=0 IOPS][eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=22595: Sat Feb 15 15:08:00 2020
read: IOPS=21, BW=87.2MiB/s (91.4MB/s)(160GiB/1879130msec)
clat (usec): min=410, max=642644, avg=45828.27, stdev=55998.10
lat (usec): min=410, max=642645, avg=45829.46, stdev=55998.01
clat percentiles (usec):
| 1.00th=[ 502], 5.00th=[ 652], 10.00th=[ 947], 20.00th=[ 1795],
| 30.00th=[ 2573], 40.00th=[ 3752], 50.00th=[ 17433], 60.00th=[ 40109],
| 70.00th=[ 66323], 80.00th=[ 96994], 90.00th=[131597], 95.00th=[156238],
| 99.00th=[206570], 99.50th=[227541], 99.90th=[295699], 99.95th=[346031],
| 99.99th=[450888]
bw ( KiB/s): min= 8192, max=147456, per=100.00%, avg=89370.05, stdev=18267.50, samples=3754
iops : min= 2, max= 36, avg=21.80, stdev= 4.46, samples=3754
lat (usec) : 500=1.00%, 750=7.24%, 1000=2.23%
lat (msec) : 2=12.46%, 4=18.10%, 10=5.36%, 20=4.66%, 50=13.10%
lat (msec) : 100=16.89%, 250=18.70%, 500=0.25%, 750=0.01%
cpu : usr=0.03%, sys=4.81%, ctx=39885, majf=0, minf=525
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=40960,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=87.2MiB/s (91.4MB/s), 87.2MiB/s-87.2MiB/s (91.4MB/s-91.4MB/s), io=160GiB (172GB), run=1879130-1879130msec
Disk stats (read/write):
vdc: ios=166878/0, merge=17/0, ticks=17788526/0, in_queue=17705166, util=10.28%
Sequential Write, 4k Blocksize
READ: bw=36.3MiB/siops: avg=9293.92
fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
writetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=32.3MiB/s][r=0,w=8261 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=23010: Sat Feb 15 16:27:47 2020
write: IOPS=9294, BW=36.3MiB/s (38.1MB/s)(160GiB/4512682msec)
clat (usec): min=2, max=56749, avg=105.88, stdev=940.46
lat (usec): min=2, max=56750, avg=106.26, stdev=940.46
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3],
| 30.00th=[ 3], 40.00th=[ 3], 50.00th=[ 3], 60.00th=[ 4],
| 70.00th=[ 6], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 8],
| 99.00th=[ 6718], 99.50th=[ 8586], 99.90th=[11469], 99.95th=[11469],
| 99.99th=[11600]
bw ( KiB/s): min=15960, max=948840, per=100.00%, avg=37175.72, stdev=16605.80, samples=9025
iops : min= 3990, max=237210, avg=9293.92, stdev=4151.45, samples=9025
lat (usec) : 4=66.41%, 10=30.55%, 20=1.59%, 50=0.23%, 100=0.01%
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.96%, 20=0.25%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=0.72%, sys=4.66%, ctx=508702, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,41943040,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=36.3MiB/s (38.1MB/s), 36.3MiB/s-36.3MiB/s (38.1MB/s-38.1MB/s), io=160GiB (172GB), run=4512682-4512682msec
Disk stats (read/write):
vdc: ios=0/166857, merge=0/3, ticks=0/863796689, in_queue=863713179, util=0.57%
Sequential Write, 4m Blocksize
READ: bw=56.5MiB/siops: avg=14.34
fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
writetest: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=60.1MiB/s][r=0,w=15 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=23663: Sat Feb 15 17:16:05 2020
write: IOPS=14, BW=56.5MiB/s (59.3MB/s)(160GiB/2897626msec)
clat (usec): min=1369, max=146898, avg=69494.24, stdev=12401.95
lat (usec): min=1465, max=147023, avg=69620.43, stdev=12400.00
clat percentiles (msec):
| 1.00th=[ 32], 5.00th=[ 50], 10.00th=[ 60], 20.00th=[ 61],
| 30.00th=[ 64], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 72],
| 70.00th=[ 77], 80.00th=[ 81], 90.00th=[ 82], 95.00th=[ 90],
| 99.00th=[ 94], 99.50th=[ 101], 99.90th=[ 110], 99.95th=[ 116],
| 99.99th=[ 132]
bw ( KiB/s): min=32768, max=1785856, per=100.00%, avg=58818.35, stdev=28286.29, samples=5704
iops : min= 8, max= 436, avg=14.34, stdev= 6.91, samples=5704
lat (msec) : 2=0.49%, 4=0.38%, 10=0.04%, 20=0.05%, 50=5.20%
lat (msec) : 100=93.64%, 250=0.21%
cpu : usr=0.13%, sys=3.42%, ctx=344222, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,40960,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=56.5MiB/s (59.3MB/s), 56.5MiB/s-56.5MiB/s (59.3MB/s-59.3MB/s), io=160GiB (172GB), run=2897626-2897626msec
Disk stats (read/write):
vdc: ios=0/166761, merge=0/37, ticks=0/549460466, in_queue=549377072, util=0.90%
Sequential, 80% Read, 20% Write, 4k Blocksize
READ: bw=58.6MiB/siops: avg=15219.35WRITE: bw=14.6MiB/siops: avg=3804.37
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k
read80write20test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
read80write20test: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [M(1)][100.0%][r=76.9MiB/s,w=18.0MiB/s][r=19.7k,w=4856 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=24828: Sat Feb 15 19:11:50 2020
read: IOPS=14.0k, BW=58.6MiB/s (61.4MB/s)(128GiB/2238448msec)
clat (nsec): min=1040, max=6732.9M, avg=63398.09, stdev=3332745.35
lat (nsec): min=1370, max=6732.9M, avg=63756.83, stdev=3332745.67
clat percentiles (nsec):
| 1.00th=[ 1112], 5.00th=[ 1144], 10.00th=[ 1176],
| 20.00th=[ 1208], 30.00th=[ 1256], 40.00th=[ 1320],
| 50.00th=[ 1496], 60.00th=[ 3376], 70.00th=[ 3856],
| 80.00th=[ 4048], 90.00th=[ 4384], 95.00th=[ 4832],
| 99.00th=[ 9792], 99.50th=[ 16512], 99.90th=[ 7634944],
| 99.95th=[ 41680896], 99.99th=[116916224]
bw ( KiB/s): min= 4087, max=131072, per=100.00%, avg=60877.40, stdev=27969.38, samples=4408
iops : min= 1021, max=32768, avg=15219.35, stdev=6992.35, samples=4408
write: IOPS=3747, BW=14.6MiB/s (15.3MB/s)(31.0GiB/2238448msec)
clat (nsec): min=1800, max=9485.3k, avg=4791.46, stdev=7221.28
lat (usec): min=2, max=9485, avg= 5.19, stdev= 7.27
clat percentiles (nsec):
| 1.00th=[ 2024], 5.00th=[ 2352], 10.00th=[ 2448], 20.00th=[ 2512],
| 30.00th=[ 2672], 40.00th=[ 2896], 50.00th=[ 3536], 60.00th=[ 5280],
| 70.00th=[ 5792], 80.00th=[ 6240], 90.00th=[ 7648], 95.00th=[ 9408],
| 99.00th=[19328], 99.50th=[24704], 99.90th=[38656], 99.95th=[45824],
| 99.99th=[68096]
bw ( KiB/s): min= 832, max=32752, per=100.00%, avg=15217.50, stdev=6995.75, samples=4408
iops : min= 208, max= 8188, avg=3804.37, stdev=1748.94, samples=4408
lat (usec) : 2=47.21%, 4=25.51%, 10=25.67%, 20=1.17%, 50=0.33%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
cpu : usr=1.63%, sys=8.15%, ctx=40585, majf=0, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=33555281,8387759,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=58.6MiB/s (61.4MB/s), 58.6MiB/s-58.6MiB/s (61.4MB/s-61.4MB/s), io=128GiB (137GB), run=2238448-2238448msec
WRITE: bw=14.6MiB/s (15.3MB/s), 14.6MiB/s-14.6MiB/s (15.3MB/s-15.3MB/s), io=31.0GiB (34.4GB), run=2238448-2238448msec
Disk stats (read/write):
vdc: ios=133172/33814, merge=0/9, ticks=19969679/24963286, in_queue=44849237, util=7.38%
Sequential, 80% Read, 20% Write, 4m Blocksize
READ: bw=65.2MiB/siops: avg=16.71WRITE: bw=16.5MiB/siops: avg=5.31
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
read80write20test: (g=0): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=32.0MiB/s,w=4100KiB/s][r=8,w=1 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=25164: Sat Feb 15 19:45:16 2020
read: IOPS=16, BW=65.2MiB/s (68.4MB/s)(128GiB/2005834msec)
clat (usec): min=417, max=1571.8k, avg=60611.73, stdev=93993.32
lat (usec): min=417, max=1571.8k, avg=60612.91, stdev=93993.26
clat percentiles (usec):
| 1.00th=[ 494], 5.00th=[ 619], 10.00th=[ 1188],
| 20.00th=[ 1926], 30.00th=[ 2737], 40.00th=[ 7242],
| 50.00th=[ 30016], 60.00th=[ 49546], 70.00th=[ 72877],
| 80.00th=[ 103285], 90.00th=[ 149947], 95.00th=[ 202376],
| 99.00th=[ 467665], 99.50th=[ 616563], 99.90th=[ 851444],
| 99.95th=[ 943719], 99.99th=[1266680]
bw ( KiB/s): min= 8143, max=147456, per=100.00%, avg=68526.64, stdev=29831.76, samples=3907
iops : min= 1, max= 36, avg=16.71, stdev= 7.28, samples=3907
write: IOPS=4, BW=16.5MiB/s (17.3MB/s)(32.3GiB/2005834msec)
clat (usec): min=851, max=13613, avg=2636.18, stdev=1151.90
lat (usec): min=890, max=13834, avg=2727.32, stdev=1157.80
clat percentiles (usec):
| 1.00th=[ 1012], 5.00th=[ 1319], 10.00th=[ 1385], 20.00th=[ 1500],
| 30.00th=[ 1762], 40.00th=[ 2089], 50.00th=[ 2442], 60.00th=[ 2835],
| 70.00th=[ 3228], 80.00th=[ 3687], 90.00th=[ 4228], 95.00th=[ 4621],
| 99.00th=[ 5342], 99.50th=[ 6194], 99.90th=[ 8979], 99.95th=[10683],
| 99.99th=[13566]
bw ( KiB/s): min= 8110, max=106496, per=100.00%, avg=21843.13, stdev=14054.37, samples=3102
iops : min= 1, max= 26, avg= 5.31, stdev= 3.43, samples=3102
lat (usec) : 500=0.96%, 750=4.22%, 1000=1.59%
lat (msec) : 2=17.72%, 4=22.73%, 10=5.93%, 20=3.37%, 50=11.73%
lat (msec) : 100=15.00%, 250=13.97%, 500=2.10%, 750=0.50%, 1000=0.15%
cpu : usr=0.06%, sys=4.60%, ctx=31635, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=32688,8272,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=65.2MiB/s (68.4MB/s), 65.2MiB/s-65.2MiB/s (68.4MB/s-68.4MB/s), io=128GiB (137GB), run=2005834-2005834msec
WRITE: bw=16.5MiB/s (17.3MB/s), 16.5MiB/s-16.5MiB/s (17.3MB/s-17.3MB/s), io=32.3GiB (34.7GB), run=2005834-2005834msec
Disk stats (read/write):
vdc: ios=132985/33922, merge=14/0, ticks=17974508/26677750, in_queue=44568551, util=8.07%
Addendum 2020-02-16
I removed the 2 small HDDs from the chassis.
command log of removing osd.9 and osd.10
[root@ceph-mon-01 ~]# ceph osd out osd.10
[root@ceph-mon-01 ~]# ceph osd out osd.9
[root@ceph-mon-01 ~]# ceph -w # until cluster is healthy again.
[root@ceph-mon-01 ~]# ceph osd tree
[root@f5-422-01 ~]# systemctl stop ceph-osd@9.service
[root@f5-422-01 ~]# systemctl stop ceph-osd@10.service
[root@f5-422-01 ~]# cat /etc/ceph/ceph.conf
[root@ceph-mon-01 ~]# ceph osd purge osd.9 --yes-i-really-mean-it
[root@ceph-mon-01 ~]# ceph osd purge osd.10 --yes-i-really-mean-it
Remove old /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm… symlinks
Addendum 2020-02-17
cleaned remaining HDDs and deleted /dev/sda4
└─sda4 8:4 0 399.2G 0 part
├─VG_Ceph_SSD_01-LV_Ceph_SSD_OSD_01 253:5 0 80G 0 lvm
│ └─q8tPpn-1wSg-ais7-7cac-k5eB-HLd1-UItqkV 253:17 0 80G 0 crypt
├─VG_Ceph_SSD_01-LV_Ceph_SSD_OSD_02 253:6 0 8G 0 lvm
│ └─EmDgfU-hhKz-1Pe5-ua8x-Hsl3-F0TH-ythz10 253:15 0 8G 0 crypt
├─VG_Ceph_SSD_01-LV_Ceph_SSD_OSD_03 253:7 0 8G 0 lvm
│ └─jtXDYp-nG6y-bKsp-okzW-5Aj3-zgJQ-5cDWz1 253:16 0 8G 0 crypt
└─VG_Ceph_SSD_01-LV_Ceph_SSD_OSD_00 253:8 0 80G 0 lvm
└─YGRkoK-BgOK-CqU7-GUih-Jc6m-gvjH-qw593H 253:20 0 80G 0 crypt
then ran playbook terramaster-prep-disks.yml which simply created VGs and LVs for use with Ceph.
I only made a Playbook since I will have to prepare multiple F5-422.
Creating them by hand is also fine, but they must be pre-created for osd_scenario: lvm
if you use ceph-ansible.
I created 2 OSDs, each with one full HDD and a 80GiB LV on the SSD.
[root@f5-422-01 ~]# ceph-volume lvm create \
--data VG_HDD_sl2_Ceph/LV_HDD_sl2_OSD_data \
--block.db VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_01 \
--dmcrypt \
--bluestore
[root@f5-422-01 ~]# ceph-volume lvm create \
--data VG_HDD_sl3_Ceph/LV_HDD_sl3_OSD_data \
--block.db VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_02 \
--dmcrypt \
--bluestore
lsblk
[root@f5-422-01 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 465.8G 0 disk
├─sda1 8:1 0 600M 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
├─sda3 8:3 0 65G 0 part
│ ├─VG_f5--422--01-LV_root 253:0 0 8G 0 lvm /
│ ├─VG_f5--422--01-LV_swap 253:1 0 4G 0 lvm [SWAP]
│ ├─VG_f5--422--01-LV_home 253:5 0 4G 0 lvm /home
│ ├─VG_f5--422--01-LV_var 253:6 0 8G 0 lvm /var
│ └─VG_f5--422--01-LV_var_log 253:7 0 2G 0 lvm /var/log
└─sda4 8:4 0 399.2G 0 part
├─VG_SSD_sl1_Ceph-LV_SSD_sl1_OSD_db_01 253:2 0 80G 0 lvm
│ └─DMKQ8z-zn6W-08os-acCj-mi1v-xD6c-tXMa6F 253:10 0 80G 0 crypt
└─VG_SSD_sl1_Ceph-LV_SSD_sl1_OSD_db_02 253:4 0 80G 0 lvm
└─yJV8Ej-IJNV-Qr8i-pJ5K-ytay-SFXc-brEK8F 253:12 0 80G 0 crypt
sdb 8:16 0 931.5G 0 disk
└─sdb1 8:17 0 931.5G 0 part
└─VG_HDD_sl2_Ceph-LV_HDD_sl2_OSD_data 253:8 0 931.5G 0 lvm
└─XWXbNF-SZ2B-0jRG-0jNY-GUjr-131X-05v16p 253:9 0 931.5G 0 crypt
sdc 8:32 0 1.8T 0 disk
└─sdc1 8:33 0 1.8T 0 part
└─VG_HDD_sl3_Ceph-LV_HDD_sl3_OSD_data 253:3 0 1.8T 0 lvm
└─S7R1hz-KfGQ-dcfe-RhKg-xl0N-EYQj-82orQE 253:11 0 1.8T 0 crypt
ceph osd tree
[root@ceph-mon-01 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 12.73514 root default
-17 2.72899 host f5-422-01
7 hdd 0.90970 osd.7 up 1.00000 1.00000
8 hdd 1.81929 osd.8 up 1.00000 1.00000
-13 0.90959 host odroid-hc2-00
6 hdd 0.90959 osd.6 up 1.00000 1.00000
-11 0.90970 host odroid-hc2-01
0 hdd 0.90970 osd.0 up 1.00000 1.00000
-9 0.90970 host odroid-hc2-02
4 hdd 0.90970 osd.4 up 1.00000 1.00000
-15 1.81929 host odroid-hc2-03
5 hdd 1.81929 osd.5 up 1.00000 1.00000
-3 1.81929 host odroid-hc2-04
1 hdd 1.81929 osd.1 up 1.00000 1.00000
-5 1.81929 host odroid-hc2-05
2 hdd 1.81929 osd.2 up 1.00000 1.00000
-7 1.81929 host odroid-hc2-06
3 hdd 1.81929 osd.3 up 1.00000 1.00000
ceph-volume lvm list
[root@f5-422-01 ~]# ceph-volume lvm list
====== osd.8 =======
[block] /dev/VG_HDD_sl3_Ceph/LV_HDD_sl3_OSD_data
type block
osd id 8
cluster fsid 154b75c2-1767-45c5-b7f6-ba7e3a36fc12
cluster name ceph
osd fsid d2c7d993-a204-43b7-9c87-4bc6331feb8f
db device /dev/VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_02
encrypted 1
db uuid yJV8Ej-IJNV-Qr8i-pJ5K-ytay-SFXc-brEK8F
cephx lockbox secret [REDACTED]
block uuid S7R1hz-KfGQ-dcfe-RhKg-xl0N-EYQj-82orQE
block device /dev/VG_HDD_sl3_Ceph/LV_HDD_sl3_OSD_data
vdo 0
crush device class None
devices /dev/sdc1
[ db] /dev/VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_02
type db
osd id 8
cluster fsid 154b75c2-1767-45c5-b7f6-ba7e3a36fc12
cluster name ceph
osd fsid d2c7d993-a204-43b7-9c87-4bc6331feb8f
db device /dev/VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_02
encrypted 1
db uuid yJV8Ej-IJNV-Qr8i-pJ5K-ytay-SFXc-brEK8F
cephx lockbox secret [REDACTED]
block uuid S7R1hz-KfGQ-dcfe-RhKg-xl0N-EYQj-82orQE
block device /dev/VG_HDD_sl3_Ceph/LV_HDD_sl3_OSD_data
vdo 0
crush device class None
devices /dev/sda4
====== osd.7 =======
[block] /dev/VG_HDD_sl2_Ceph/LV_HDD_sl2_OSD_data
type block
osd id 7
cluster fsid 154b75c2-1767-45c5-b7f6-ba7e3a36fc12
cluster name ceph
osd fsid d3417340-754e-4e99-ade2-1ebe93a274cf
db device /dev/VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_01
encrypted 1
db uuid DMKQ8z-zn6W-08os-acCj-mi1v-xD6c-tXMa6F
cephx lockbox secret [REDACTED]
block uuid XWXbNF-SZ2B-0jRG-0jNY-GUjr-131X-05v16p
block device /dev/VG_HDD_sl2_Ceph/LV_HDD_sl2_OSD_data
vdo 0
crush device class None
devices /dev/sdb1
[ db] /dev/VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_01
type db
osd id 7
cluster fsid 154b75c2-1767-45c5-b7f6-ba7e3a36fc12
cluster name ceph
osd fsid d3417340-754e-4e99-ade2-1ebe93a274cf
db device /dev/VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_01
encrypted 1
db uuid DMKQ8z-zn6W-08os-acCj-mi1v-xD6c-tXMa6F
cephx lockbox secret [REDACTED]
block uuid XWXbNF-SZ2B-0jRG-0jNY-GUjr-131X-05v16p
block device /dev/VG_HDD_sl2_Ceph/LV_HDD_sl2_OSD_data
vdo 0
crush device class None
devices /dev/sda4
Volume Groups
[root@f5-422-01 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
VG_HDD_sl2_Ceph 1 1 0 wz--n- <931.51g 0
VG_HDD_sl3_Ceph 1 1 0 wz--n- <1.82t 0
VG_SSD_sl1_Ceph 1 2 0 wz--n- <399.17g <239.17g
VG_f5-422-01 1 5 0 wz--n- 65.00g 39.00g
Logical Volumes
[root@f5-422-01 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
LV_HDD_sl2_OSD_data VG_HDD_sl2_Ceph -wi-ao---- <931.51g
LV_HDD_sl3_OSD_data VG_HDD_sl3_Ceph -wi-ao---- <1.82t
LV_SSD_sl1_OSD_db_01 VG_SSD_sl1_Ceph -wi-ao---- 80.00g
LV_SSD_sl1_OSD_db_02 VG_SSD_sl1_Ceph -wi-ao---- 80.00g
LV_home VG_f5-422-01 -wi-ao---- 4.00g
LV_root VG_f5-422-01 -wi-ao---- 8.00g
LV_swap VG_f5-422-01 -wi-ao---- 4.00g
LV_var VG_f5-422-01 -wi-ao---- 8.00g
LV_var_log VG_f5-422-01 -wi-ao---- 2.00g
Physical Volumes
[root@f5-422-01 ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 VG_f5-422-01 lvm2 a-- 65.00g 39.00g
/dev/sda4 VG_SSD_sl1_Ceph lvm2 a-- <399.17g <239.17g
/dev/sdb1 VG_HDD_sl2_Ceph lvm2 a-- <931.51g 0
/dev/sdc1 VG_HDD_sl3_Ceph lvm2 a-- <1.82t 0