ODROID-HC2 Nodes After 10 Weeks In My Ceph Cluster
Table of Contents
It’s been about 2½ months since I started adding ODROID-HC2 nodes to my Ceph Luminous cluster.
This is a quick update on how it’s doing so far.
The nodes are still on Fedora 29 armhfp, I’ll have to deal with an upgrade fop F30 or later within the next 6 months.
What This Post is About
Until recently I would adjust my playbooks, roles and fiddle with Ceph, followed by updating previous posts.
While that’s necessary (the updating of posts) since I pull in the latest versions of the Ansible bits, I feel I should record my findings so far.
Current Cluster State
[root@odroid-hc2-00 ~]# ceph -s
cluster:
id: b1a134d0-927c-4eb9-9ae5-2f6902dce55c
health: HEALTH_OK
services:
mon: 3 daemons, quorum odroid-hc2-00,odroid-hc2-01,odroid-hc2-02
mgr: odroid-hc2-00(active), standbys: odroid-hc2-01, odroid-hc2-02
mds: cephfs-1/1/1 up {0=odroid-hc2-03=up:active}
osd: 7 osds: 7 up, 7 in
rgw: 1 daemon active
data:
pools: 9 pools, 208 pgs
objects: 17.50k objects, 66.8GiB
usage: 207GiB used, 9.80TiB / 10.0TiB avail
pgs: 208 active+clean
The Bad
- the nodes without serial console make more work than those with when something goes wrong with the low-cost hardware
- the reboot issue plus having all nodes fed off one PSU is not such a good combination
- while returns of broken hardware are painless in Europe, so far I had these failures
- 3 µSD cards
- 3 ODROID-U3 USB-UART MODULE
- 1 ODROID-HC2
Room for Improvement
- my current performance settings mean that the cluster needs a bit to spool up to speed
(hence the test runs over 10 minutes (600 seconds) further down below)
- Since this is a test cluster, I’d rather leave it that way, no need to nail down everything to max perf all the time, I’d rather save power.
The Good
- the 2 nodes with constantly attached serial console (logging to a conserver) are quite painless
- Ceph’s resilience means that even though the hardware side is too flaky for my taste (in my day job I nearly exclusively deal with enterprise grade x86_64 hardware), I have not lost any data stored on this Ceph cluster.
- bandwidth is now within the levels of what can be expected of this low-cost hardware
Performance Measurements with rados bench
Once again I gathered performance data, see 7. Benchmarking Performance of the RHCS 3 Administration Guide for details.
Write Test
Total time run: 600.691810
Total writes made: 8388
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 55.8556
Stddev Bandwidth: 17.0047
Max bandwidth (MB/sec): 96
Min bandwidth (MB/sec): 0
Average IOPS: 13
Stddev IOPS: 4
Max IOPS: 24
Min IOPS: 0
Average Latency(s): 1.14563
Stddev Latency(s): 0.840153
Max latency(s): 16.5859
Min latency(s): 0.204739
rados bench -p testbench 600 write --no-cleanup
[root@odroid-hc2-00 ~]# rados bench -p testbench 600 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_odroid-hc2-00.internal.pcfe.n_24309
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 20 4 16.0296 16 0.930154 0.855249
2 16 37 21 42.031 68 0.632538 0.953409
3 16 41 25 33.3464 16 0.33715 0.861344
4 16 41 25 25.0047 0 - 0.861344
5 16 41 25 20.0019 0 - 0.861344
6 16 41 25 16.6672 0 - 0.861344
7 16 41 25 14.2854 0 - 0.861344
8 16 41 25 12.4992 0 - 0.861344
9 16 41 25 11.1094 0 - 0.861344
10 16 41 25 9.99799 0 - 0.861344
11 16 41 25 9.08854 0 - 0.861344
12 16 41 25 8.33066 0 - 0.861344
13 16 41 25 7.6897 0 - 0.861344
14 16 41 25 7.14035 0 - 0.861344
15 16 41 25 6.66421 0 - 0.861344
16 16 41 25 6.24756 0 - 0.861344
17 16 50 34 7.99686 2.57143 0.333293 4.21942
18 16 61 45 9.99615 44 1.01069 5.44457
19 16 69 53 11.1537 32 2.69975 5.42909
2019-05-18 16:09:13.371129 min lat: 0.296097 max lat: 16.5859 avg lat: 4.66752
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 82 66 13.1951 52 0.485576 4.66752
21 16 97 81 15.4145 60 1.32828 4.01412
22 16 110 94 17.0758 52 1.49541 3.62898
23 16 128 112 19.4617 72 1.49855 3.23275
24 16 137 121 20.15 36 1.73638 3.06026
25 16 153 137 21.9023 64 0.717952 2.83943
26 16 166 150 23.0588 52 1.74417 2.71984
27 16 186 170 25.1659 80 0.714535 2.49741
28 16 198 182 25.9806 48 0.61696 2.39038
29 16 206 190 26.1877 32 0.328705 2.33554
30 16 219 203 27.0471 52 0.548666 2.30743
31 16 233 217 27.9664 56 0.544651 2.24442
32 16 246 230 28.7163 52 0.79844 2.17362
33 16 264 248 30.0262 72 1.31281 2.09175
34 16 283 267 31.3767 76 1.37788 2.00764
35 16 301 285 32.5349 72 0.608292 1.93916
36 16 309 293 32.5196 32 0.35753 1.91056
37 16 325 309 33.3691 64 0.499244 1.88549
38 16 343 327 34.3842 72 1.20053 1.83698
39 16 353 337 34.5273 40 0.558362 1.80665
2019-05-18 16:09:33.406019 min lat: 0.296097 max lat: 16.5859 avg lat: 1.78314
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 369 353 35.2628 64 0.360414 1.78314
41 16 384 368 35.8647 60 1.73654 1.753
42 15 407 392 37.2946 96 0.481849 1.70109
43 16 418 402 37.3555 40 0.50218 1.67719
44 16 432 416 37.7782 56 0.526682 1.6628
45 16 451 435 38.6257 76 1.07195 1.64098
46 16 462 446 38.7406 44 1.23201 1.62098
47 16 477 461 39.1918 60 0.504165 1.61046
48 16 493 477 39.7079 64 0.602835 1.59419
49 16 514 498 40.6107 84 0.645411 1.55728
50 16 532 516 41.2376 72 1.00432 1.54031
51 16 550 534 41.84 72 1.56522 1.51871
52 16 560 544 41.8044 40 0.964614 1.50549
53 16 576 560 42.2226 64 0.332654 1.49786
54 16 587 571 42.2507 44 1.4148 1.49321
55 16 598 582 42.2822 44 1.26732 1.4859
56 16 611 595 42.4552 52 0.633864 1.48008
57 16 627 611 42.8326 64 0.562681 1.47483
58 16 633 617 42.5079 24 1.51784 1.47024
59 16 646 630 42.6684 52 0.665011 1.47437
2019-05-18 16:09:53.424128 min lat: 0.260421 max lat: 16.5859 avg lat: 1.47003
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 660 644 42.8902 56 0.409845 1.47003
61 16 672 656 42.9735 48 1.44398 1.46796
62 16 692 676 43.5699 80 0.456449 1.45481
63 16 701 685 43.4497 36 1.73998 1.45229
64 16 712 696 43.4529 44 0.493589 1.45176
65 16 724 708 43.5227 48 0.312931 1.44536
66 16 740 724 43.8324 64 0.451251 1.44561
67 16 756 740 44.133 64 0.573374 1.43748
68 16 769 753 44.2484 52 0.365358 1.42875
69 16 791 775 44.8817 88 1.36392 1.4197
70 16 802 786 44.8689 44 0.39191 1.41026
71 16 821 805 45.3068 76 1.64609 1.40038
72 16 837 821 45.566 64 0.622298 1.3894
73 16 854 838 45.8728 68 0.539862 1.38664
74 16 869 853 46.0634 60 1.07408 1.37739
75 16 887 871 46.402 72 1.38983 1.36975
76 16 902 886 46.5806 60 0.533069 1.36304
77 16 914 898 46.5989 48 0.496655 1.35985
78 16 931 915 46.8728 68 0.544218 1.35366
79 16 941 925 46.7858 40 0.708357 1.35069
2019-05-18 16:10:13.447849 min lat: 0.260421 max lat: 16.5859 avg lat: 1.34329
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 964 948 47.3502 92 0.745056 1.34329
81 16 977 961 47.3968 52 0.371952 1.33965
82 16 991 975 47.5001 56 0.678969 1.33616
83 16 1009 993 47.7947 72 0.514251 1.32903
84 16 1024 1008 47.9395 60 1.24164 1.32275
85 16 1041 1025 48.1751 68 0.730028 1.31787
86 16 1057 1041 48.3588 64 0.51351 1.31322
87 16 1067 1051 48.2627 40 1.32062 1.31128
88 16 1078 1062 48.2142 44 0.316048 1.30865
89 16 1091 1075 48.2562 52 2.10825 1.31268
90 16 1105 1089 48.3419 56 1.66736 1.31318
91 16 1122 1106 48.5575 68 1.74501 1.30937
92 16 1136 1120 48.6381 56 0.60998 1.30401
93 16 1148 1132 48.63 48 1.74713 1.30776
94 16 1155 1139 48.4106 28 1.79613 1.30918
95 16 1172 1156 48.6165 68 0.560134 1.30849
96 16 1183 1167 48.5683 44 1.38118 1.30658
97 16 1197 1181 48.6446 56 0.591372 1.30637
98 16 1208 1192 48.5969 44 1.91099 1.3036
99 16 1222 1206 48.6713 56 0.550189 1.30388
2019-05-18 16:10:33.478342 min lat: 0.260421 max lat: 16.5859 avg lat: 1.30363
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 1235 1219 48.7042 52 2.11017 1.30363
101 16 1249 1233 48.7761 56 0.552273 1.30166
102 16 1264 1248 48.8857 60 0.303107 1.29806
103 16 1277 1261 48.9157 52 1.78821 1.29731
104 16 1286 1270 48.7914 36 1.74565 1.29844
105 16 1303 1287 48.974 68 0.631927 1.29911
106 16 1324 1308 49.3038 84 0.885227 1.29361
107 16 1337 1321 49.3286 52 0.475104 1.2892
108 16 1354 1338 49.5011 68 0.626061 1.2853
109 15 1374 1359 49.8171 84 0.799113 1.27858
110 16 1387 1371 49.8002 48 0.8391 1.27591
111 16 1400 1384 49.819 52 1.76957 1.27751
112 16 1413 1397 49.8382 52 0.414058 1.27581
113 16 1425 1409 49.8216 48 1.58055 1.27407
114 16 1434 1418 49.7 36 0.625207 1.2749
115 16 1442 1426 49.546 32 0.573376 1.27731
116 16 1461 1445 49.7734 76 1.54251 1.27927
117 16 1479 1463 49.9629 72 0.615321 1.27566
118 16 1486 1470 49.7769 28 1.40165 1.27336
119 16 1502 1486 49.8961 64 0.686663 1.27543
2019-05-18 16:10:53.491667 min lat: 0.260421 max lat: 16.5859 avg lat: 1.27614
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 1513 1497 49.8469 44 1.68219 1.27614
121 16 1527 1511 49.8971 56 0.935795 1.27465
122 16 1544 1528 50.0451 68 0.666474 1.27167
123 16 1559 1543 50.1254 60 1.46194 1.26863
124 16 1570 1554 50.0758 44 0.32392 1.26717
125 16 1590 1574 50.3148 80 0.875384 1.26823
126 16 1599 1583 50.201 36 1.53562 1.26728
127 16 1614 1598 50.2778 60 1.19224 1.26553
128 16 1629 1613 50.3536 60 0.709749 1.26244
129 16 1647 1631 50.5207 72 1.80161 1.26207
130 16 1660 1644 50.5317 52 1.84457 1.26187
131 16 1672 1656 50.5121 48 0.444286 1.26024
132 16 1689 1673 50.6441 68 0.47536 1.25649
133 16 1710 1694 50.8945 84 0.624648 1.25324
134 16 1722 1706 50.8702 48 1.42011 1.25104
135 16 1735 1719 50.8772 52 1.96487 1.25107
136 16 1748 1732 50.8815 52 1.7548 1.25084
137 16 1762 1746 50.9187 56 0.526288 1.25028
138 16 1778 1762 51.0134 64 0.589687 1.24794
139 16 1795 1779 51.1353 68 0.430793 1.24651
2019-05-18 16:11:13.524136 min lat: 0.260421 max lat: 16.5859 avg lat: 1.24183
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 1819 1803 51.4553 96 0.633373 1.24183
141 16 1831 1815 51.4307 48 0.816879 1.23845
142 16 1841 1825 51.3502 40 1.54713 1.23841
143 16 1858 1842 51.4665 68 1.49424 1.24
144 16 1869 1853 51.4146 44 0.509308 1.23792
145 16 1883 1867 51.4461 56 0.429097 1.23692
146 16 1898 1882 51.5045 60 0.459762 1.23583
147 16 1913 1897 51.5621 60 0.905544 1.23579
148 16 1923 1907 51.4839 40 0.65524 1.23417
149 16 1934 1918 51.4319 44 2.16588 1.23903
150 16 1947 1931 51.4356 52 0.32784 1.23821
151 16 1960 1944 51.4392 52 0.80663 1.23628
152 16 1981 1965 51.6531 84 0.743415 1.23557
153 16 1990 1974 51.5508 36 0.41452 1.2332
154 16 2008 1992 51.6834 72 1.37006 1.23235
155 16 2021 2005 51.6853 52 1.58191 1.23132
156 16 2038 2022 51.7897 68 0.302508 1.231
157 16 2057 2041 51.9437 76 1.13743 1.2295
158 16 2069 2053 51.9187 48 1.10888 1.22656
159 16 2082 2066 51.9191 52 0.460752 1.22603
2019-05-18 16:11:33.534628 min lat: 0.260421 max lat: 16.5859 avg lat: 1.22595
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 2098 2082 51.9945 64 0.311714 1.22595
161 16 2112 2096 52.0192 56 0.636634 1.22468
162 16 2126 2110 52.0437 56 0.451941 1.22366
163 16 2142 2126 52.1169 64 1.5064 1.22373
164 16 2156 2140 52.1405 56 1.52295 1.22367
165 16 2170 2154 52.1638 56 1.28338 1.22162
166 16 2185 2169 52.2109 60 1.47192 1.22065
167 16 2197 2181 52.1856 48 0.647264 1.21922
168 16 2211 2195 52.2081 56 0.645364 1.22003
169 16 2229 2213 52.325 72 0.676861 1.21939
170 16 2248 2232 52.4641 76 1.54151 1.21728
171 16 2255 2239 52.3211 28 0.347513 1.21549
172 16 2269 2253 52.3424 56 0.654888 1.21679
173 16 2284 2268 52.3865 60 1.2619 1.21645
174 16 2300 2284 52.4531 64 1.29193 1.21674
175 16 2314 2298 52.4733 56 1.44989 1.21519
176 16 2326 2310 52.4478 48 1.72853 1.2156
177 16 2337 2321 52.4001 44 1.3404 1.21573
178 16 2350 2334 52.3978 52 0.567191 1.21587
179 16 2360 2344 52.3285 40 1.44666 1.21578
2019-05-18 16:11:53.539663 min lat: 0.260421 max lat: 16.5859 avg lat: 1.21767
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 2373 2357 52.3266 52 0.393848 1.21767
181 16 2386 2370 52.3247 52 0.278133 1.21819
182 16 2397 2381 52.2788 44 0.590739 1.21835
183 16 2409 2393 52.2554 48 1.60516 1.21978
184 16 2424 2408 52.2974 60 0.455627 1.22111
185 16 2435 2419 52.2525 44 1.31594 1.22062
186 16 2444 2428 52.1651 36 0.568799 1.21909
187 16 2466 2450 52.3564 88 0.892549 1.21922
188 16 2478 2462 52.3332 48 0.299418 1.217
189 16 2496 2480 52.4371 72 0.695307 1.21672
190 16 2514 2498 52.5398 72 0.480004 1.21425
191 16 2523 2507 52.4532 36 1.72035 1.21515
192 16 2535 2519 52.43 48 0.605721 1.21559
193 16 2550 2534 52.4691 60 1.38608 1.21557
194 16 2565 2549 52.5078 60 0.268934 1.2137
195 16 2578 2562 52.5051 52 0.476035 1.21393
196 16 2593 2577 52.5432 60 0.459752 1.21394
197 16 2609 2593 52.6013 64 0.515444 1.21261
198 16 2625 2609 52.6587 64 0.431176 1.21234
199 16 2643 2627 52.7531 72 1.17626 1.2102
2019-05-18 16:12:13.556963 min lat: 0.245755 max lat: 16.5859 avg lat: 1.21002
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 2649 2633 52.6092 24 0.381636 1.21002
201 16 2665 2649 52.6657 64 1.97941 1.21109
202 16 2679 2663 52.6821 56 0.433315 1.20983
203 16 2696 2680 52.7573 68 1.30231 1.2103
204 16 2701 2685 52.5968 20 1.44582 1.20977
205 16 2716 2700 52.6328 60 1.39051 1.21112
206 16 2726 2710 52.5715 40 0.531209 1.21193
207 16 2739 2723 52.5686 52 0.513902 1.21093
208 16 2753 2737 52.585 56 1.70676 1.21246
209 16 2775 2759 52.7543 88 1.26044 1.21163
210 16 2788 2772 52.7506 52 1.29441 1.20984
211 16 2802 2786 52.7659 56 0.596399 1.20857
212 16 2818 2802 52.8187 64 0.333844 1.20671
213 16 2836 2820 52.9087 72 1.08406 1.20647
214 16 2850 2834 52.923 56 1.33532 1.20574
215 16 2864 2848 52.9372 56 0.599857 1.20524
216 16 2874 2858 52.8773 40 1.99396 1.20439
217 16 2888 2872 52.8916 56 0.614629 1.20525
218 16 2902 2886 52.9058 56 0.345928 1.20577
219 16 2919 2903 52.9746 68 1.22508 1.2058
2019-05-18 16:12:33.563410 min lat: 0.204739 max lat: 16.5859 avg lat: 1.20445
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 2934 2918 53.0064 60 0.321194 1.20445
221 16 2944 2928 52.9476 40 1.16769 1.20475
222 16 2956 2940 52.9176 48 2.03323 1.2043
223 16 2970 2954 52.9313 56 0.361867 1.20369
224 16 2984 2968 52.9449 56 0.698546 1.20355
225 16 3002 2986 53.0293 72 0.580753 1.2037
226 16 3017 3001 53.0601 60 1.21367 1.20232
227 16 3032 3016 53.0905 60 0.741994 1.20071
228 16 3050 3034 53.1731 72 1.32353 1.20131
229 16 3065 3049 53.2027 60 1.08493 1.20039
230 16 3085 3069 53.319 80 0.63936 1.19817
231 16 3099 3083 53.3305 56 1.45461 1.19674
232 16 3114 3098 53.3591 60 1.07505 1.19661
233 16 3131 3115 53.4218 68 1.21683 1.19473
234 16 3147 3131 53.4669 64 0.563806 1.19456
235 16 3162 3146 53.4945 60 0.489637 1.19357
236 16 3178 3162 53.5389 64 1.05553 1.19238
237 16 3191 3175 53.5323 52 0.544769 1.19196
238 16 3211 3195 53.6433 80 0.705249 1.1902
239 16 3225 3209 53.6531 56 1.09667 1.18922
2019-05-18 16:12:53.604936 min lat: 0.204739 max lat: 16.5859 avg lat: 1.18838
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 3240 3224 53.6794 60 0.312957 1.18838
241 16 3256 3240 53.7221 64 0.482895 1.18821
242 16 3271 3255 53.7479 60 0.640144 1.18715
243 16 3283 3267 53.7242 48 0.507039 1.1874
244 16 3300 3284 53.7826 68 1.41019 1.18698
245 16 3316 3300 53.8241 64 0.321793 1.186
246 16 3334 3318 53.8978 72 0.957751 1.18526
247 16 3348 3332 53.9062 56 0.733105 1.18416
248 16 3365 3349 53.9628 68 0.45977 1.18333
249 16 3373 3357 53.8745 32 1.35831 1.18334
250 16 3392 3376 53.9628 76 0.980545 1.1833
251 16 3410 3394 54.0345 72 0.557226 1.18218
252 16 3417 3401 53.9311 28 0.452422 1.18133
253 16 3441 3425 54.0972 96 0.929698 1.18145
254 15 3452 3437 54.0731 48 1.38442 1.18009
255 16 3467 3451 54.0805 56 0.436901 1.17982
256 16 3488 3472 54.1971 84 1.93081 1.1791
257 16 3502 3486 54.204 56 0.488884 1.17699
258 16 3518 3502 54.2418 64 0.494519 1.17616
259 16 3530 3514 54.2176 48 1.28206 1.17737
2019-05-18 16:13:13.615649 min lat: 0.204739 max lat: 16.5859 avg lat: 1.17626
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 3549 3533 54.3012 76 0.68118 1.17626
261 16 3561 3545 54.277 48 1.36922 1.17597
262 16 3578 3562 54.3293 68 1.13122 1.17618
263 16 3589 3573 54.2899 44 1.15971 1.17496
264 16 3601 3585 54.2661 48 0.333611 1.17498
265 16 3616 3600 54.2876 60 0.711266 1.17525
266 16 3625 3609 54.2188 36 0.252959 1.17601
267 16 3644 3628 54.3001 76 0.80956 1.17609
268 16 3660 3644 54.3362 64 0.485574 1.17565
269 16 3668 3652 54.2531 32 0.358967 1.17428
270 16 3687 3671 54.3335 76 1.86974 1.17627
271 16 3691 3675 54.1921 16 0.433627 1.17573
272 16 3704 3688 54.184 52 0.65242 1.17744
273 16 3717 3701 54.1759 52 0.687427 1.17768
274 16 3726 3710 54.1096 36 1.87406 1.17902
275 16 3737 3721 54.0727 44 0.700663 1.17899
276 16 3751 3735 54.0795 56 0.418646 1.17986
277 16 3765 3749 54.0863 56 1.41936 1.18035
278 16 3782 3766 54.1362 68 0.538819 1.1801
279 16 3798 3782 54.1714 64 0.496792 1.17917
2019-05-18 16:13:33.625596 min lat: 0.204739 max lat: 16.5859 avg lat: 1.17953
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 3809 3793 54.1351 44 1.54345 1.17953
281 16 3819 3803 54.0847 40 1.6929 1.18013
282 16 3833 3817 54.0914 56 0.56953 1.17999
283 16 3854 3838 54.1967 84 0.323926 1.17841
284 16 3871 3855 54.2452 68 1.00298 1.17832
285 16 3885 3869 54.2513 56 1.37687 1.17782
286 16 3900 3884 54.2712 60 1.0587 1.17702
287 16 3912 3896 54.2493 48 1.20156 1.17693
288 16 3931 3915 54.3247 76 1.62476 1.17693
289 16 3949 3933 54.3858 72 0.430894 1.17501
290 16 3962 3946 54.3775 52 1.01582 1.17428
291 16 3976 3960 54.3829 56 1.31065 1.17449
292 16 3992 3976 54.4157 64 0.528849 1.17405
293 16 4004 3988 54.3938 48 0.361629 1.17338
294 16 4023 4007 54.4671 76 0.633131 1.17284
295 16 4037 4021 54.4721 56 1.12174 1.1722
296 16 4054 4038 54.5177 68 1.34113 1.17193
297 16 4066 4050 54.4956 48 0.439699 1.17153
298 16 4083 4067 54.5408 68 0.374661 1.17085
299 16 4097 4081 54.5455 56 1.826 1.17071
2019-05-18 16:13:53.637452 min lat: 0.204739 max lat: 16.5859 avg lat: 1.17008
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 4110 4094 54.5369 52 0.772369 1.17008
301 16 4124 4108 54.5415 56 0.335137 1.16989
302 16 4139 4123 54.5593 60 0.287464 1.17074
303 16 4151 4135 54.5376 48 0.491748 1.17015
304 16 4166 4150 54.5554 60 0.450227 1.17106
305 16 4174 4158 54.4815 32 1.52729 1.17149
306 16 4182 4166 54.408 32 0.558202 1.17201
307 16 4197 4181 54.4261 60 0.573123 1.17342
308 16 4215 4199 54.4831 72 0.455909 1.17276
309 16 4225 4209 54.4362 40 0.407753 1.17227
310 16 4239 4223 54.441 56 1.9277 1.17261
311 16 4252 4236 54.433 52 0.275288 1.17265
312 16 4266 4250 54.438 56 1.66835 1.17265
313 16 4286 4270 54.5195 80 0.742528 1.17161
314 16 4303 4287 54.5622 68 1.20689 1.17167
315 16 4315 4299 54.5394 48 0.430751 1.17066
316 16 4331 4315 54.5692 64 0.575494 1.17055
317 16 4341 4325 54.5232 40 1.93008 1.17083
318 16 4356 4340 54.5403 60 0.642964 1.17099
319 16 4370 4354 54.5448 56 1.47414 1.17033
2019-05-18 16:14:13.661444 min lat: 0.204739 max lat: 16.5859 avg lat: 1.17029
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 4381 4365 54.5118 44 0.312343 1.17029
321 16 4397 4381 54.5412 64 0.511051 1.17074
322 16 4409 4393 54.5208 48 1.6233 1.17184
323 16 4423 4407 54.5253 56 1.30348 1.17186
324 16 4435 4419 54.505 48 0.567373 1.1717
325 16 4446 4430 54.4726 44 1.37418 1.1719
326 16 4463 4447 54.5139 68 0.51342 1.17228
327 16 4468 4452 54.4083 20 2.1888 1.17229
328 16 4481 4465 54.4009 52 0.399453 1.1736
329 16 4497 4481 54.43 64 2.99338 1.17409
330 16 4514 4498 54.4709 68 1.0119 1.173
331 16 4532 4516 54.5236 72 0.609291 1.17212
332 16 4547 4531 54.54 60 1.48135 1.17084
333 16 4564 4548 54.5803 68 0.574407 1.17022
334 16 4578 4562 54.5845 56 1.51042 1.17021
335 16 4590 4574 54.5648 48 1.53674 1.17045
336 16 4604 4588 54.569 56 0.507811 1.17048
337 16 4621 4605 54.6027 68 0.529507 1.16953
338 16 4633 4617 54.583 48 0.332042 1.16911
339 16 4650 4634 54.6224 68 0.567079 1.16926
2019-05-18 16:14:33.712040 min lat: 0.204739 max lat: 16.5859 avg lat: 1.16964
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 4659 4643 54.5676 36 0.314137 1.16964
341 16 4675 4659 54.5951 64 1.90094 1.17062
342 16 4690 4674 54.6108 60 0.699515 1.16977
343 16 4705 4689 54.6264 60 0.376131 1.16905
344 16 4722 4706 54.6652 68 0.410512 1.16831
345 16 4744 4728 54.7616 88 0.735994 1.16676
346 16 4763 4747 54.8229 76 1.18011 1.1661
347 16 4776 4760 54.8147 52 1.03632 1.16539
348 16 4796 4780 54.8868 80 1.11234 1.16441
349 16 4812 4796 54.9127 64 0.423259 1.16305
350 16 4823 4807 54.881 44 0.49099 1.16325
351 16 4844 4828 54.9638 84 0.716123 1.16269
352 16 4858 4842 54.9667 56 0.858849 1.16156
353 16 4878 4862 55.0374 80 0.993311 1.16133
354 16 4886 4870 54.9724 32 1.89012 1.16108
355 16 4895 4879 54.9189 36 2.15176 1.16175
356 16 4914 4898 54.978 76 1.37308 1.16274
357 16 4929 4913 54.992 60 0.292571 1.16173
358 16 4938 4922 54.9389 36 0.360288 1.16201
359 16 4957 4941 54.9975 76 1.28495 1.16154
2019-05-18 16:14:53.726181 min lat: 0.204739 max lat: 16.5859 avg lat: 1.16102
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 4969 4953 54.978 48 0.585598 1.16102
361 16 4987 4971 55.025 72 0.63191 1.1617
362 16 5001 4985 55.0277 56 0.636301 1.1613
363 16 5018 5002 55.0633 68 1.20723 1.16064
364 16 5035 5019 55.0987 68 0.86308 1.15978
365 16 5050 5034 55.1121 60 1.25283 1.15972
366 16 5067 5051 55.1472 68 1.04086 1.15898
367 16 5076 5060 55.095 36 1.86551 1.15882
368 16 5090 5074 55.0974 56 1.92613 1.15843
369 16 5103 5087 55.0889 52 0.733858 1.1588
370 16 5124 5108 55.1669 84 0.901831 1.15829
371 16 5134 5118 55.126 40 1.63097 1.15765
372 16 5142 5126 55.0638 32 2.49766 1.15865
373 16 5157 5141 55.077 60 0.86992 1.15961
374 16 5171 5155 55.0794 56 0.78585 1.15943
375 16 5192 5176 55.1563 84 0.430681 1.15891
376 16 5204 5188 55.1373 48 0.99319 1.15886
377 16 5221 5205 55.1713 68 0.416587 1.15826
378 16 5240 5224 55.2262 76 0.756114 1.15727
379 16 5259 5243 55.2809 76 1.17882 1.15674
2019-05-18 16:15:13.735791 min lat: 0.204739 max lat: 16.5859 avg lat: 1.15574
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 5273 5257 55.2827 56 1.30316 1.15574
381 16 5280 5264 55.211 28 0.421515 1.1554
382 16 5294 5278 55.213 56 0.495295 1.1564
383 16 5309 5293 55.2254 60 0.310589 1.15657
384 16 5329 5313 55.2898 80 0.843848 1.15622
385 16 5345 5329 55.3123 64 1.11149 1.15561
386 16 5362 5346 55.345 68 0.402285 1.15421
387 16 5379 5363 55.3776 68 0.847615 1.1544
388 16 5394 5378 55.3895 60 0.470617 1.15407
389 16 5408 5392 55.391 56 0.421586 1.15373
390 16 5420 5404 55.372 48 0.331792 1.15319
391 16 5437 5421 55.4042 68 0.877509 1.15325
392 16 5453 5437 55.4261 64 1.80872 1.15301
393 16 5467 5451 55.4275 56 0.542076 1.1524
394 16 5482 5466 55.439 60 0.732007 1.15192
395 16 5499 5483 55.4707 68 1.41413 1.15188
396 16 5509 5493 55.4316 40 1.51009 1.15191
397 16 5529 5513 55.4934 80 1.44471 1.15151
398 16 5537 5521 55.4344 32 1.56754 1.15178
399 16 5556 5540 55.4858 76 0.743028 1.15181
2019-05-18 16:15:33.746068 min lat: 0.204739 max lat: 16.5859 avg lat: 1.15114
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 5569 5553 55.477 52 0.468189 1.15114
401 16 5587 5571 55.518 72 0.533789 1.15089
402 16 5603 5587 55.5391 64 1.18352 1.1509
403 16 5613 5597 55.5005 40 1.49849 1.15079
404 16 5632 5616 55.5511 76 0.80678 1.15109
405 16 5646 5630 55.5521 56 0.353304 1.15025
406 16 5666 5650 55.6122 80 0.617229 1.14954
407 16 5676 5660 55.5738 40 1.30539 1.14893
408 16 5682 5666 55.4965 24 2.12495 1.15003
409 16 5695 5679 55.4878 52 0.593519 1.15105
410 16 5705 5689 55.4499 40 0.602261 1.15109
411 16 5720 5704 55.4609 60 1.94311 1.15183
412 16 5734 5718 55.4621 56 0.490515 1.1515
413 16 5751 5735 55.4924 68 0.645214 1.1513
414 16 5765 5749 55.4936 56 0.354147 1.1508
415 16 5780 5764 55.5044 60 0.64927 1.15095
416 16 5792 5776 55.4863 48 0.502865 1.15079
417 16 5808 5792 55.5067 64 0.575211 1.15128
418 16 5822 5806 55.5078 56 0.615406 1.15115
419 16 5835 5819 55.4994 52 0.508867 1.1508
2019-05-18 16:15:53.756481 min lat: 0.204739 max lat: 16.5859 avg lat: 1.15129
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 5854 5838 55.5481 76 1.44501 1.15129
421 16 5863 5847 55.5016 36 1.15826 1.15113
422 16 5879 5863 55.5217 64 0.314032 1.15043
423 16 5894 5878 55.5321 60 0.703674 1.15077
424 16 5908 5892 55.5331 56 1.53015 1.15052
425 16 5923 5907 55.5435 60 0.280521 1.15031
426 16 5946 5930 55.629 92 0.429985 1.14956
427 16 5955 5939 55.583 36 0.384777 1.1492
428 16 5968 5952 55.5746 52 0.721271 1.14907
429 16 5985 5969 55.6035 68 1.74732 1.14977
430 16 5998 5982 55.595 52 1.05115 1.1497
431 16 6010 5994 55.5774 48 2.07382 1.14997
432 16 6022 6006 55.5598 48 1.37511 1.14971
433 16 6034 6018 55.5423 48 1.65097 1.14982
434 16 6050 6034 55.5616 64 1.62414 1.15012
435 16 6066 6050 55.5809 64 0.43617 1.15008
436 16 6069 6053 55.4809 12 0.32026 1.14998
437 16 6084 6068 55.4912 60 1.99363 1.15125
438 16 6094 6078 55.4557 40 0.691585 1.15172
439 16 6109 6093 55.466 60 1.14974 1.15185
2019-05-18 16:16:13.768496 min lat: 0.204739 max lat: 16.5859 avg lat: 1.15192
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 6124 6108 55.4762 60 0.676361 1.15192
441 16 6139 6123 55.4864 60 0.378641 1.15134
442 16 6156 6140 55.5146 68 0.621909 1.15149
443 16 6168 6152 55.4976 48 0.344402 1.15114
444 16 6186 6170 55.5346 72 0.570477 1.15102
445 16 6196 6180 55.4997 40 0.388845 1.15086
446 16 6218 6202 55.5724 88 1.40662 1.15087
447 16 6229 6213 55.5465 44 0.350942 1.15013
448 16 6241 6225 55.5296 48 1.87033 1.1504
449 16 6256 6240 55.5349 60 0.743087 1.15051
450 16 6270 6254 55.5359 56 1.68079 1.15047
451 16 6286 6270 55.5546 64 0.290122 1.14984
452 16 6296 6280 55.5202 40 1.75126 1.15047
453 16 6310 6294 55.5212 56 0.501221 1.15033
454 16 6328 6312 55.5574 72 0.500216 1.15075
455 16 6342 6326 55.5584 56 0.283727 1.15027
456 16 6354 6338 55.5418 48 1.40202 1.15017
457 16 6368 6352 55.5428 56 0.481239 1.15094
458 16 6381 6365 55.535 52 0.355391 1.15059
459 16 6396 6380 55.5447 60 1.24346 1.15015
2019-05-18 16:16:33.813952 min lat: 0.204739 max lat: 16.5859 avg lat: 1.15009
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 6411 6395 55.5543 60 0.391753 1.15009
461 16 6424 6408 55.5466 52 1.98274 1.15033
462 16 6440 6424 55.5648 64 0.590823 1.15035
463 16 6455 6439 55.5744 60 0.237137 1.15004
464 16 6467 6451 55.558 48 0.65687 1.15009
465 16 6487 6471 55.6105 80 0.75469 1.14977
466 16 6501 6485 55.6113 56 0.995754 1.14932
467 16 6515 6499 55.6121 56 1.33373 1.14926
468 16 6523 6507 55.5617 32 1.97651 1.14912
469 16 6537 6521 55.5626 56 1.98675 1.15049
470 16 6553 6537 55.5805 64 0.293269 1.15005
471 16 6568 6552 55.5899 60 0.599057 1.14996
472 16 6581 6565 55.5822 52 0.398278 1.14939
473 16 6603 6587 55.6507 88 0.450805 1.14941
474 16 6620 6604 55.6767 68 0.460914 1.14847
475 16 6629 6613 55.6353 36 0.485028 1.14832
476 16 6638 6622 55.594 36 1.43775 1.149
477 16 6652 6636 55.5948 56 0.553024 1.14981
478 16 6660 6644 55.5453 32 0.319692 1.14934
479 16 6679 6663 55.5874 76 0.531922 1.14985
2019-05-18 16:16:53.825232 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14924
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 6691 6675 55.5716 48 0.512707 1.14924
481 16 6708 6692 55.5973 68 1.51198 1.14926
482 16 6716 6700 55.5483 32 0.526603 1.14942
483 16 6731 6715 55.5573 60 0.843943 1.14994
484 16 6753 6737 55.6242 88 0.793789 1.14939
485 16 6768 6752 55.6331 60 0.312519 1.14866
486 16 6787 6771 55.6749 76 0.963633 1.14875
487 16 6800 6784 55.6673 52 1.03712 1.14818
488 16 6812 6796 55.6516 48 0.380842 1.1478
489 16 6825 6809 55.6441 52 1.41718 1.14846
490 16 6838 6822 55.6365 52 0.614704 1.1482
491 15 6855 6840 55.6697 72 0.306905 1.14796
492 16 6870 6854 55.6704 56 0.671838 1.1481
493 16 6884 6868 55.671 56 0.527099 1.14827
494 16 6892 6876 55.6231 32 0.453514 1.1484
495 16 6902 6886 55.5914 40 2.57987 1.14888
496 16 6924 6908 55.6566 88 0.969073 1.14913
497 16 6936 6920 55.6412 48 1.26388 1.14867
498 16 6952 6936 55.6579 64 1.46033 1.14879
499 16 6970 6954 55.6905 72 0.413916 1.14791
2019-05-18 16:17:13.838553 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14763
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 6984 6968 55.6911 56 0.468279 1.14763
501 16 6999 6983 55.6994 60 1.5795 1.14763
502 16 7013 6997 55.6999 56 1.72081 1.14741
503 16 7035 7019 55.7641 88 0.802843 1.14663
504 16 7040 7024 55.6931 20 1.3077 1.14642
505 16 7057 7041 55.7174 68 0.538387 1.14694
506 16 7067 7051 55.6863 40 0.293349 1.14711
507 16 7081 7065 55.6868 56 0.336351 1.14757
508 16 7097 7081 55.7031 64 0.536487 1.14737
509 16 7111 7095 55.7036 56 0.290292 1.14712
510 16 7125 7109 55.7042 56 0.50036 1.14748
511 16 7133 7117 55.6578 32 1.5987 1.14797
512 16 7148 7132 55.6662 60 0.301433 1.14831
513 16 7160 7144 55.6512 48 0.399621 1.14804
514 16 7175 7159 55.6595 60 1.67203 1.14842
515 16 7192 7176 55.6834 68 0.465412 1.1479
516 16 7208 7192 55.6995 64 0.429258 1.14742
517 16 7227 7211 55.7386 76 0.798461 1.14702
518 16 7240 7224 55.7314 52 1.53195 1.14677
519 16 7250 7234 55.7011 40 1.56924 1.14699
2019-05-18 16:17:33.851722 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14686
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 7267 7251 55.7246 68 0.479032 1.14686
521 16 7282 7266 55.7327 60 0.912604 1.14674
522 16 7297 7281 55.7408 60 0.545733 1.14597
523 16 7309 7293 55.7259 48 1.76437 1.14667
524 16 7321 7305 55.7111 48 1.77764 1.14684
525 16 7337 7321 55.7268 64 0.69031 1.14708
526 16 7356 7340 55.7645 76 0.308001 1.14691
527 16 7369 7353 55.7573 52 1.229 1.14676
528 16 7381 7365 55.7426 48 1.11876 1.14662
529 16 7396 7380 55.7506 60 0.611191 1.1464
530 16 7414 7398 55.7811 72 0.250219 1.14617
531 16 7431 7415 55.804 68 1.07885 1.14592
532 16 7440 7424 55.7667 36 0.669192 1.14532
533 16 7446 7430 55.707 24 0.426968 1.14489
534 16 7461 7445 55.715 60 0.558104 1.14709
535 16 7479 7463 55.7454 72 1.37258 1.14753
536 16 7488 7472 55.7085 36 1.05389 1.14747
537 16 7495 7479 55.6569 28 0.391677 1.14723
538 16 7513 7497 55.6872 72 0.486923 1.14804
539 16 7534 7518 55.7396 84 1.88252 1.14771
2019-05-18 16:17:53.872656 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14745
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 7547 7531 55.7327 52 0.567006 1.14745
541 16 7564 7548 55.7553 68 1.05759 1.14701
542 16 7578 7562 55.7557 56 1.448 1.1469
543 16 7588 7572 55.7266 40 1.3252 1.14678
544 16 7595 7579 55.6756 28 0.361036 1.14646
545 16 7616 7600 55.7275 84 0.712029 1.14786
546 16 7624 7608 55.684 32 1.17169 1.14749
547 16 7640 7624 55.6992 64 0.609843 1.14756
548 16 7661 7645 55.7507 84 0.442766 1.14715
549 16 7678 7662 55.773 68 0.990156 1.14643
550 16 7693 7677 55.7806 60 1.3231 1.14613
551 16 7708 7692 55.7882 60 0.676965 1.14567
552 16 7730 7714 55.8464 88 1.43524 1.14535
553 16 7747 7731 55.8683 68 0.548032 1.14468
554 16 7765 7749 55.8974 72 1.29826 1.14421
555 16 7769 7753 55.8255 16 0.650907 1.14414
556 16 7785 7769 55.8402 64 1.69654 1.14462
557 16 7794 7778 55.8045 36 0.353513 1.14455
558 16 7815 7799 55.8549 84 0.525465 1.14492
559 16 7828 7812 55.8479 52 0.884194 1.1444
2019-05-18 16:18:13.883823 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14433
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 7847 7831 55.8838 76 1.28146 1.14433
561 16 7854 7838 55.8341 28 0.306502 1.14395
562 16 7867 7851 55.8273 52 0.414001 1.14449
563 16 7883 7867 55.8417 64 1.38221 1.14474
564 16 7897 7881 55.8419 56 1.41273 1.14456
565 16 7918 7902 55.8917 84 1.40659 1.14443
566 16 7932 7916 55.8918 56 0.461647 1.14401
567 16 7948 7932 55.9061 64 0.578833 1.14369
568 16 7961 7945 55.8991 52 0.835055 1.14356
569 16 7972 7956 55.8782 44 1.6421 1.14382
570 16 7982 7966 55.8503 40 1.71238 1.1443
571 16 7994 7978 55.8365 48 0.544987 1.14462
572 16 8007 7991 55.8298 52 0.371788 1.14482
573 16 8020 8004 55.823 52 0.486054 1.14488
574 16 8034 8018 55.8233 56 1.4323 1.14495
575 16 8046 8030 55.8097 48 1.78167 1.14499
576 16 8061 8045 55.8169 60 0.566864 1.14516
577 16 8081 8065 55.8587 80 0.79225 1.14484
578 16 8098 8082 55.8796 68 0.518239 1.14419
579 16 8110 8094 55.8659 48 0.259853 1.14349
2019-05-18 16:18:33.894234 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14333
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 8130 8114 55.9075 80 0.873793 1.14333
581 16 8144 8128 55.9076 56 0.45542 1.14289
582 16 8156 8140 55.8939 48 1.56962 1.14336
583 16 8167 8151 55.8735 44 0.32456 1.14353
584 16 8182 8166 55.8804 60 1.7 1.14398
585 16 8200 8184 55.9078 72 1.14439 1.14363
586 16 8212 8196 55.8942 48 0.505604 1.14297
587 16 8231 8215 55.9284 76 1.103 1.14329
588 16 8243 8227 55.9149 48 0.412295 1.14305
589 16 8252 8236 55.881 36 0.444145 1.14288
590 16 8264 8248 55.8676 48 2.05193 1.14416
591 16 8272 8256 55.8272 32 1.83562 1.14378
592 16 8284 8268 55.814 48 2.29641 1.14555
593 16 8288 8272 55.7468 16 2.64731 1.14552
594 16 8310 8294 55.801 88 0.948032 1.14599
595 16 8324 8308 55.8013 56 0.974495 1.14556
596 16 8347 8331 55.862 92 1.60445 1.14496
597 16 8357 8341 55.8354 40 0.696737 1.14438
598 16 8368 8352 55.8155 44 1.53799 1.14499
599 16 8379 8363 55.7958 44 0.72372 1.14497
2019-05-18 16:18:53.907850 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14518
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 14 8388 8374 55.7761 44 1.57904 1.14518
Total time run: 600.691810
Total writes made: 8388
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 55.8556
Stddev Bandwidth: 17.0047
Max bandwidth (MB/sec): 96
Min bandwidth (MB/sec): 0
Average IOPS: 13
Stddev IOPS: 4
Max IOPS: 24
Min IOPS: 0
Average Latency(s): 1.14563
Stddev Latency(s): 0.840153
Max latency(s): 16.5859
Min latency(s): 0.204739
Sequential Read Test
Total time run: 348.446419
Total reads made: 8388
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 96.2903
Average IOPS: 24
Stddev IOPS: 5
Max IOPS: 31
Min IOPS: 0
Average Latency(s): 0.655719
Max latency(s): 20.4885
Min latency(s): 0.0702014
rados bench -p testbench 600 seq
[root@odroid-hc2-00 ~]# rados bench -p testbench 600 seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 34 18 71.9509 72 0.649461 0.465242
2 16 61 45 89.9379 108 0.22313 0.547868
3 16 81 65 86.6131 80 0.211665 0.446539
4 16 82 66 65.9582 4 0.135089 0.44182
5 16 82 66 52.7651 0 - 0.44182
6 16 82 66 43.9723 0 - 0.44182
7 16 82 66 37.6915 0 - 0.44182
8 16 82 66 32.9806 0 - 0.44182
9 16 82 66 29.3167 0 - 0.44182
10 16 82 66 26.3851 0 - 0.44182
11 16 82 66 23.9863 0 - 0.44182
12 16 82 66 21.9879 0 - 0.44182
13 16 82 66 20.2968 0 - 0.44182
14 16 82 66 18.8469 0 - 0.44182
15 16 82 66 17.5905 0 - 0.44182
16 15 86 71 17.7352 1.66667 14.7545 1.24742
17 16 96 80 18.8085 36 0.136803 1.49274
18 16 97 81 17.986 4 0.147446 1.47613
19 16 97 81 17.0392 0 - 1.47613
2019-05-18 16:26:46.043620 min lat: 0.0977557 max lat: 14.9532 avg lat: 1.47613
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 97 81 16.1871 0 - 1.47613
21 16 97 81 15.4162 0 - 1.47613
22 16 106 90 16.3509 9 0.248036 2.12854
23 16 130 114 19.8111 96 20.0951 2.83186
24 16 157 141 23.4825 108 0.478491 2.65431
25 16 182 166 26.5408 100 0.54867 2.34945
26 15 208 193 29.6676 108 1.08236 2.11712
27 15 235 220 32.5602 108 0.244662 1.92329
28 16 263 247 35.2491 108 1.03459 1.77847
29 16 288 272 37.4779 100 0.324194 1.66429
30 16 313 297 39.5596 100 0.256307 1.58032
31 16 338 322 41.507 100 0.409362 1.50381
32 15 362 347 43.3273 100 0.223785 1.44598
33 15 386 371 44.9175 96 0.128119 1.39374
34 16 414 398 46.7704 108 0.140685 1.33956
35 15 438 423 48.2876 100 0.293822 1.30218
36 16 465 449 49.833 104 0.929303 1.26329
37 16 492 476 51.4031 108 1.01234 1.22373
38 16 519 503 52.8906 108 1.16127 1.18722
39 16 547 531 54.4042 112 0.9601 1.15267
2019-05-18 16:27:06.069087 min lat: 0.089375 max lat: 20.4885 avg lat: 1.13349
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 569 553 55.2428 88 1.31021 1.13349
41 16 593 577 56.2355 96 0.271013 1.11265
42 16 621 605 57.5616 112 1.35264 1.09212
43 16 648 632 58.7332 108 0.146986 1.0714
44 16 672 656 59.579 96 1.19001 1.05477
45 16 699 683 60.6536 108 0.548563 1.03666
46 16 724 708 61.5079 100 0.909568 1.02225
47 16 751 735 62.4959 108 0.363091 1.00547
48 15 777 762 63.4405 108 0.507468 0.990405
49 16 802 786 64.104 96 0.407017 0.98194
50 16 827 811 64.821 100 1.23883 0.969793
51 16 856 840 65.8233 116 0.572234 0.957716
52 16 882 866 66.5565 104 0.370699 0.94461
53 16 907 891 67.1865 100 0.241297 0.936921
54 16 932 916 67.7934 100 0.879143 0.928525
55 16 959 943 68.5234 108 0.614999 0.919552
56 16 984 968 69.0848 100 0.229312 0.911014
57 16 1009 993 69.6263 100 0.241872 0.902711
58 16 1036 1020 70.2871 108 0.436083 0.895007
59 15 1062 1047 70.923 108 0.389651 0.888343
2019-05-18 16:27:26.077915 min lat: 0.089375 max lat: 20.4885 avg lat: 0.883441
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 1085 1069 71.2071 88 0.247771 0.883441
61 16 1107 1091 71.4818 88 0.319624 0.880142
62 16 1133 1117 72.0056 104 0.329838 0.873547
63 15 1158 1143 72.5121 104 1.19158 0.866999
64 16 1183 1167 72.8786 96 1.6969 0.862038
65 16 1209 1193 73.3567 104 0.520301 0.857765
66 15 1231 1216 73.6229 92 0.254234 0.854008
67 16 1257 1241 74.016 100 0.747341 0.850056
68 16 1283 1267 74.4562 104 0.651322 0.845709
69 16 1307 1291 74.7679 96 0.528358 0.840703
70 16 1333 1317 75.1847 104 1.0838 0.838281
71 16 1357 1341 75.4743 96 0.277656 0.833992
72 16 1382 1366 75.8143 100 0.554572 0.830625
73 16 1404 1388 75.9808 88 1.0496 0.829183
74 16 1429 1413 76.3048 100 0.905241 0.825251
75 16 1454 1438 76.6202 100 0.728943 0.822016
76 16 1481 1465 77.0325 108 1.16819 0.818149
77 16 1504 1488 77.2262 92 1.41273 0.815016
78 16 1526 1510 77.3639 88 1.45944 0.81292
79 16 1554 1538 77.8017 112 1.53274 0.808983
2019-05-18 16:27:46.118494 min lat: 0.0883547 max lat: 20.4885 avg lat: 0.804803
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 15 1578 1563 78.0614 100 0.0931126 0.804803
81 15 1606 1591 78.4773 112 0.232075 0.802335
82 16 1632 1616 78.7392 100 1.22366 0.799794
83 16 1659 1643 79.091 108 0.184041 0.796709
84 16 1684 1668 79.3394 100 0.314756 0.79322
85 16 1709 1693 79.582 100 0.246481 0.791329
86 16 1733 1717 79.7724 96 0.384007 0.788049
87 15 1757 1742 79.9874 100 0.232566 0.78663
88 16 1785 1769 80.3051 108 1.30851 0.783935
89 16 1810 1794 80.5201 100 0.259362 0.781293
90 15 1837 1822 80.8676 112 0.199282 0.77932
91 16 1862 1846 81.0334 96 0.341418 0.777467
92 16 1887 1871 81.2391 100 1.31535 0.775779
93 16 1912 1896 81.4402 100 0.208707 0.773902
94 16 1937 1921 81.6372 100 0.590208 0.77177
95 16 1964 1948 81.9141 108 0.2059 0.769184
96 15 1987 1972 82.0584 96 0.245602 0.766938
97 16 2011 1995 82.1605 92 0.119004 0.765537
98 15 2037 2022 82.4177 108 1.06405 0.76481
99 15 2063 2048 82.6207 104 1.14823 0.763587
2019-05-18 16:28:06.183096 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.760714
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 15 2088 2073 82.7913 100 1.38304 0.760714
101 15 2115 2100 83.0389 108 1.16035 0.759352
102 15 2140 2125 83.1963 100 0.230518 0.75629
103 16 2169 2153 83.4753 112 1.71676 0.753751
104 16 2190 2174 83.48 84 0.377225 0.7546
105 16 2214 2198 83.5988 96 1.04079 0.753256
106 16 2240 2224 83.7906 104 0.511616 0.752575
107 16 2268 2252 84.0536 112 1.27808 0.75003
108 16 2293 2277 84.2008 100 0.591175 0.748349
109 16 2316 2300 84.2717 92 0.397413 0.747774
110 16 2343 2327 84.4867 108 0.184993 0.744705
111 16 2367 2351 84.59 96 0.304083 0.743798
112 16 2396 2380 84.8679 116 0.303333 0.74207
113 16 2423 2407 85.0721 108 0.248038 0.740364
114 16 2447 2431 85.1677 96 0.525181 0.739478
115 16 2473 2457 85.331 104 0.197812 0.737703
116 16 2501 2485 85.5603 112 0.178938 0.736079
117 15 2527 2512 85.7496 108 0.159411 0.734786
118 15 2550 2535 85.7997 92 0.200366 0.733558
119 16 2573 2557 85.8178 88 0.229232 0.733471
2019-05-18 16:28:26.210712 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.733638
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 2596 2580 85.869 92 0.636454 0.733638
121 16 2621 2605 85.9854 100 1.28439 0.733321
122 16 2649 2633 86.1981 112 0.0995623 0.730976
123 15 2672 2657 86.2593 96 0.120085 0.729898
124 16 2697 2681 86.3375 96 0.95693 0.730524
125 16 2722 2706 86.4464 100 1.06586 0.72968
126 16 2746 2730 86.522 96 0.297313 0.728528
127 16 2773 2757 86.6907 108 0.189163 0.728046
128 15 2797 2782 86.7925 100 0.222701 0.726587
129 16 2822 2806 86.8635 96 1.05778 0.726072
130 16 2845 2829 86.9027 92 0.237288 0.725759
131 16 2871 2855 87.0328 104 0.2232 0.724304
132 16 2894 2878 87.0703 92 0.276495 0.723182
133 16 2919 2903 87.1671 100 1.70854 0.722995
134 15 2946 2931 87.3509 112 0.240077 0.721212
135 16 2974 2958 87.5034 108 0.959884 0.720882
136 16 2998 2982 87.5657 96 0.702406 0.720009
137 16 3025 3009 87.7144 108 1.10943 0.719172
138 16 3047 3031 87.7164 88 1.21191 0.718943
139 15 3070 3055 87.7754 96 0.470351 0.718459
2019-05-18 16:28:46.246905 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.717831
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 3097 3081 87.891 104 0.229593 0.717831
141 15 3124 3109 88.0561 112 1.09959 0.716613
142 16 3152 3136 88.1961 108 0.233637 0.714827
143 15 3174 3159 88.2226 92 0.599847 0.713792
144 15 3200 3185 88.3292 104 0.215048 0.713827
145 15 3227 3212 88.4596 108 0.542062 0.712929
146 16 3257 3241 88.6478 116 0.883503 0.711706
147 16 3283 3267 88.7519 104 0.910553 0.710833
148 16 3309 3293 88.8546 104 0.258162 0.710026
149 16 3336 3320 88.9828 108 1.21196 0.708694
150 16 3358 3342 88.9761 88 1.71717 0.708478
151 16 3383 3367 89.0488 100 0.097676 0.708006
152 16 3409 3393 89.1469 104 0.197949 0.707226
153 16 3433 3417 89.1914 96 0.174553 0.706958
154 16 3458 3442 89.2614 100 0.429104 0.706735
155 16 3485 3469 89.3819 108 0.254237 0.705395
156 16 3509 3493 89.4242 96 0.446979 0.704827
157 16 3533 3517 89.4658 96 1.74394 0.704498
158 16 3560 3544 89.5828 108 0.254862 0.703977
159 16 3585 3569 89.6481 100 0.536047 0.703279
2019-05-18 16:29:06.272821 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.702655
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 3610 3594 89.7126 100 0.499299 0.702655
161 16 3635 3619 89.7761 100 0.282486 0.702283
162 15 3661 3646 89.8807 108 1.1474 0.701903
163 15 3688 3673 89.9794 108 0.388947 0.700798
164 16 3714 3698 90.0403 100 0.347593 0.700595
165 16 3744 3728 90.2215 120 0.457697 0.699318
166 16 3769 3753 90.2801 100 1.15885 0.698339
167 16 3793 3777 90.3141 96 0.275888 0.69741
168 16 3818 3802 90.3715 100 0.22709 0.697647
169 16 3843 3827 90.4282 100 0.226985 0.696795
170 15 3870 3855 90.5524 112 0.329313 0.696379
171 16 3893 3877 90.5373 88 1.58285 0.696692
172 15 3919 3904 90.6368 108 0.704328 0.695854
173 16 3948 3932 90.76 112 1.11327 0.695163
174 16 3974 3958 90.8358 104 0.475799 0.694865
175 16 3997 3981 90.8423 92 0.305772 0.694271
176 15 4021 4006 90.8885 100 1.3461 0.694088
177 16 4049 4033 90.9849 108 0.30406 0.69337
178 15 4075 4060 91.0773 108 0.332814 0.692678
179 16 4101 4085 91.1269 100 0.424896 0.691768
2019-05-18 16:29:26.338383 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.691053
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 4125 4109 91.1538 96 0.29066 0.691053
181 16 4150 4134 91.2024 100 0.306401 0.691114
182 16 4176 4160 91.2724 104 0.417583 0.690919
183 16 4199 4183 91.2763 92 0.360782 0.690085
184 16 4223 4207 91.3018 96 0.295387 0.689989
185 16 4249 4233 91.3702 104 0.318681 0.69005
186 16 4274 4258 91.4164 100 0.740703 0.68992
187 16 4296 4280 91.398 88 0.492648 0.689863
188 16 4323 4307 91.4861 108 0.649533 0.689777
189 16 4350 4334 91.5732 108 1.10171 0.688964
190 16 4375 4359 91.6173 100 0.2567 0.688335
191 16 4398 4382 91.6192 92 1.35683 0.688417
192 16 4425 4409 91.7043 108 0.263076 0.68808
193 15 4448 4433 91.7228 96 0.956792 0.687832
194 16 4475 4459 91.7857 104 0.302509 0.687035
195 15 4499 4484 91.8275 100 0.352269 0.686929
196 16 4526 4510 91.8894 104 0.456594 0.6868
197 16 4554 4538 91.9912 112 1.31374 0.685685
198 16 4579 4563 92.0314 100 0.737832 0.68535
199 16 4602 4586 92.0311 92 1.18962 0.684826
2019-05-18 16:29:46.351818 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.684069
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 4628 4612 92.0908 104 1.85666 0.684069
201 16 4653 4637 92.1298 100 0.349701 0.683663
202 16 4676 4660 92.1291 92 0.116027 0.684061
203 16 4702 4686 92.1859 104 0.436322 0.683872
204 16 4730 4714 92.2826 112 1.68873 0.683235
205 16 4755 4739 92.32 100 0.40087 0.68341
206 16 4780 4764 92.357 100 0.43221 0.68287
207 16 4808 4792 92.4516 112 0.774458 0.682526
208 16 4834 4818 92.5068 104 0.341888 0.681914
209 16 4863 4847 92.6189 116 0.249431 0.681263
210 16 4887 4871 92.6347 96 0.482918 0.680565
211 15 4913 4898 92.6997 108 0.255771 0.680287
212 15 4941 4926 92.7892 112 0.990813 0.680322
213 16 4968 4952 92.8416 104 0.220118 0.679066
214 16 4992 4976 92.8561 96 1.94265 0.678654
215 16 5015 4999 92.852 92 1.84279 0.6788
216 16 5041 5025 92.9033 104 0.366245 0.679139
217 16 5069 5053 92.991 112 0.776837 0.678807
218 16 5095 5079 93.041 104 0.223128 0.678052
219 16 5117 5101 93.0179 88 0.379307 0.678041
2019-05-18 16:30:06.383558 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.6779
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 5146 5130 93.1221 116 1.07424 0.6779
221 16 5170 5154 93.1349 96 0.347358 0.677567
222 16 5198 5182 93.2197 112 1.37466 0.676895
223 16 5221 5205 93.2141 92 0.435221 0.676695
224 15 5249 5234 93.3135 116 0.420222 0.67648
225 15 5276 5261 93.3693 108 0.408896 0.675628
226 16 5303 5287 93.4162 104 0.80939 0.675319
227 15 5327 5312 93.4428 100 1.66228 0.67529
228 16 5354 5338 93.489 104 0.588061 0.675052
229 16 5380 5364 93.5347 104 0.413989 0.674534
230 16 5405 5389 93.5626 100 0.784262 0.674463
231 16 5429 5413 93.573 96 0.446694 0.674204
232 16 5455 5439 93.6177 104 1.06563 0.673836
233 16 5479 5463 93.6279 96 0.151696 0.673185
234 16 5506 5490 93.689 108 0.460371 0.673363
235 16 5533 5517 93.7497 108 0.621171 0.673046
236 15 5557 5542 93.7744 100 0.854384 0.672891
237 15 5584 5569 93.8337 108 1.19965 0.672703
238 16 5611 5595 93.8762 104 0.910699 0.672185
239 16 5636 5620 93.9015 100 0.296045 0.671709
2019-05-18 16:30:26.430199 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.671832
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 15 5658 5643 93.8925 92 1.76473 0.671832
241 15 5682 5667 93.901 96 0.630003 0.672041
242 16 5712 5696 93.9909 116 0.275767 0.671112
243 16 5736 5720 93.9987 96 1.29537 0.670659
244 15 5757 5742 93.9705 88 0.367462 0.670431
245 16 5784 5768 94.0112 104 0.199453 0.670598
246 16 5808 5792 94.019 96 0.380391 0.670767
247 16 5834 5818 94.0591 104 0.274492 0.670727
248 16 5859 5843 94.0827 100 0.288092 0.670483
249 15 5886 5871 94.1537 112 0.902737 0.6704
250 16 5915 5899 94.2247 112 0.1058 0.669707
251 16 5942 5926 94.2794 108 1.2498 0.668788
252 16 5967 5951 94.3019 100 0.0702014 0.6683
253 16 5993 5977 94.3367 104 0.2925 0.66828
254 15 6019 6004 94.3899 108 0.154874 0.668287
255 16 6048 6032 94.4587 112 0.389957 0.668076
256 16 6073 6057 94.4801 100 0.613371 0.667623
257 16 6098 6082 94.5015 100 0.283191 0.667479
258 16 6124 6108 94.5381 104 0.205475 0.666867
259 16 6149 6133 94.559 100 0.439629 0.666714
2019-05-18 16:30:46.464212 min lat: 0.0702014 max lat: 20.4885 avg lat: 0.666356
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 6176 6160 94.6104 108 0.147709 0.666356
261 15 6201 6186 94.6378 104 0.444546 0.666173
262 16 6229 6213 94.6885 108 0.474866 0.665829
263 16 6255 6239 94.7237 104 1.07845 0.665858
264 16 6282 6266 94.7737 108 0.153152 0.665209
265 16 6308 6292 94.8083 104 0.246811 0.664923
266 16 6330 6314 94.7827 88 0.301539 0.664682
267 16 6357 6341 94.8319 108 0.168142 0.664857
268 16 6380 6364 94.8212 92 1.232 0.665078
269 16 6404 6388 94.8253 96 0.331193 0.664991
270 16 6429 6413 94.8444 100 0.791554 0.664882
271 16 6458 6442 94.9222 116 0.421307 0.664248
272 16 6486 6470 94.9847 112 0.158938 0.663664
273 16 6508 6492 94.9591 88 0.705513 0.663375
274 16 6530 6514 94.9335 88 0.567068 0.664039
275 16 6557 6541 94.9807 108 1.49898 0.664032
276 16 6583 6567 95.0132 104 0.34375 0.663265
277 16 6607 6591 95.0166 96 0.370137 0.663109
278 16 6635 6619 95.0775 112 1.62973 0.663323
279 16 6661 6645 95.1093 104 1.09981 0.663127
2019-05-18 16:31:06.500490 min lat: 0.0702014 max lat: 20.4885 avg lat: 0.662423
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 15 6687 6672 95.1536 108 0.181715 0.662423
281 16 6713 6697 95.1707 100 0.241655 0.662289
282 16 6740 6724 95.2159 108 0.502332 0.662137
283 16 6767 6751 95.2609 108 0.523246 0.662163
284 16 6793 6777 95.2915 104 0.222285 0.661771
285 16 6817 6801 95.2939 96 0.97777 0.661658
286 16 6842 6826 95.3101 100 0.46303 0.661258
287 16 6867 6851 95.3251 100 2.06722 0.66074
288 16 6893 6877 95.3536 104 0.357226 0.661123
289 16 6920 6904 95.3972 108 0.231899 0.660786
290 16 6947 6931 95.4405 108 0.375587 0.660537
291 16 6973 6957 95.4693 104 0.295895 0.660247
292 15 6996 6981 95.4656 96 0.562043 0.660555
293 16 7024 7008 95.5082 108 0.71041 0.660282
294 16 7055 7039 95.6028 124 0.276101 0.65944
295 15 7080 7065 95.6306 104 0.185409 0.65949
296 16 7105 7089 95.6317 96 0.146311 0.65949
297 16 7132 7116 95.6732 108 0.795265 0.659381
298 16 7158 7142 95.701 104 0.117077 0.658996
299 16 7182 7166 95.7018 96 0.231621 0.65881
2019-05-18 16:31:26.542031 min lat: 0.0702014 max lat: 20.4885 avg lat: 0.659082
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 7204 7188 95.676 88 0.734178 0.659082
301 16 7229 7213 95.6902 100 0.655116 0.658974
302 15 7255 7240 95.7303 108 0.848148 0.658957
303 16 7282 7266 95.7575 104 0.160946 0.658426
304 15 7310 7295 95.8202 116 0.282947 0.658226
305 15 7335 7320 95.8329 100 1.26983 0.65798
306 16 7363 7347 95.8725 108 0.186013 0.657633
307 16 7389 7373 95.8989 104 0.358382 0.657644
308 16 7417 7401 95.9509 112 0.734936 0.657326
309 16 7439 7423 95.9248 88 1.41027 0.657385
310 15 7462 7447 95.9236 96 0.256498 0.657517
311 16 7487 7471 95.9237 96 0.223205 0.657493
312 16 7516 7500 95.9877 116 0.848565 0.65737
313 16 7543 7527 96.0257 108 0.263861 0.656776
314 16 7565 7549 96.0001 88 0.700857 0.65683
315 16 7587 7571 95.9746 88 0.282148 0.65701
316 15 7615 7600 96.0375 116 0.225031 0.656858
317 16 7641 7625 96.0498 100 0.0870665 0.656489
318 16 7665 7649 96.0495 96 0.350301 0.65656
319 16 7692 7676 96.0867 108 0.946106 0.656574
2019-05-18 16:31:46.573000 min lat: 0.0702014 max lat: 20.4885 avg lat: 0.656309
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 7717 7701 96.0987 100 0.189171 0.656309
321 16 7743 7727 96.1228 104 0.20635 0.656048
322 16 7766 7750 96.1095 92 0.107305 0.656156
323 16 7790 7774 96.1091 96 0.841181 0.656212
324 16 7814 7798 96.1086 96 1.61225 0.655964
325 15 7839 7824 96.1312 104 0.524573 0.655988
326 16 7867 7851 96.1673 108 0.195431 0.655979
327 16 7893 7877 96.1911 104 0.140344 0.655711
328 16 7916 7900 96.1782 92 0.866095 0.655767
329 16 7942 7926 96.2018 104 0.55399 0.655742
330 15 7966 7951 96.213 100 1.10584 0.655765
331 16 7993 7977 96.2364 104 0.271027 0.65548
332 16 8018 8002 96.2477 100 0.185103 0.655168
333 16 8044 8028 96.2707 104 0.326841 0.654999
334 16 8068 8052 96.2697 96 0.229361 0.655127
335 16 8091 8075 96.2568 92 0.276868 0.654857
336 15 8116 8101 96.2778 104 0.546563 0.654517
337 15 8140 8125 96.2701 96 0.60756 0.654944
338 16 8164 8148 96.2574 92 0.79943 0.655273
339 16 8189 8173 96.2683 100 0.213498 0.655065
2019-05-18 16:32:06.620439 min lat: 0.0702014 max lat: 20.4885 avg lat: 0.655098
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 8213 8197 96.2675 96 0.486064 0.655098
341 15 8237 8222 96.2777 100 0.368972 0.654855
342 16 8262 8246 96.2768 96 1.26142 0.654944
343 16 8287 8271 96.2875 100 1.16228 0.655047
344 15 8310 8295 96.2849 96 0.276478 0.654849
345 16 8338 8322 96.3187 108 0.337555 0.654757
346 16 8358 8342 96.2715 80 0.124775 0.654621
347 16 8383 8367 96.2817 100 0.387493 0.655212
Total time run: 348.446419
Total reads made: 8388
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 96.2903
Average IOPS: 24
Stddev IOPS: 5
Max IOPS: 31
Min IOPS: 0
Average Latency(s): 0.655719
Max latency(s): 20.4885
Min latency(s): 0.0702014
Random Read Test
Total time run: 600.630222
Total reads made: 15183
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 101.114
Average IOPS: 25
Stddev IOPS: 1
Max IOPS: 31
Min IOPS: 18
Average Latency(s): 0.624714
Max latency(s): 2.24639
Min latency(s): 0.0160655
rados bench -p testbench 600 rand
[root@odroid-hc2-00 ~]# rados bench -p testbench 600 rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 15 38 23 91.9614 92 0.460823 0.459978
2 16 62 46 91.9588 92 0.102533 0.530199
3 16 89 73 97.2974 108 0.563032 0.559597
4 16 114 98 97.9662 100 0.540406 0.571641
5 16 138 122 97.5686 96 0.143201 0.582627
6 15 163 148 98.6343 104 1.15127 0.594922
7 16 189 173 98.826 100 1.11067 0.604032
8 16 213 197 98.47 96 0.367997 0.606495
9 15 237 222 98.6355 100 0.304864 0.607933
10 16 261 245 97.9697 92 1.28205 0.613282
11 15 289 274 99.6063 116 0.245911 0.603371
12 16 317 301 100.302 108 1.09828 0.609613
13 15 341 326 100.276 100 0.336652 0.60194
14 16 367 351 100.255 100 0.33114 0.605137
15 15 395 380 101.302 116 0.028177 0.596902
16 16 421 405 101.219 100 0.725493 0.603266
17 16 448 432 101.616 108 1.26503 0.605315
18 16 474 458 101.748 104 0.0796884 0.602191
19 16 498 482 101.444 96 0.324958 0.601594
2019-05-18 16:33:22.102095 min lat: 0.0178536 max lat: 1.7715 avg lat: 0.604092
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 525 509 101.77 108 1.62598 0.604092
21 16 551 535 101.875 104 1.12436 0.608823
22 16 577 561 101.97 104 0.332545 0.607396
23 16 603 587 102.058 104 1.07456 0.607911
24 16 626 610 101.637 92 0.523947 0.606173
25 16 650 634 101.411 96 1.35517 0.611286
26 16 675 659 101.355 100 1.25767 0.611671
27 16 700 684 101.304 100 1.21896 0.610098
28 16 725 709 101.257 100 0.278581 0.609667
29 15 753 738 101.762 116 0.152244 0.61278
30 15 780 765 101.969 108 1.20678 0.610946
31 16 805 789 101.776 96 0.238761 0.611217
32 16 832 816 101.97 108 1.43774 0.608713
33 16 855 839 101.667 92 0.275256 0.609295
34 15 879 864 101.617 100 0.506553 0.610091
35 16 903 887 101.341 92 0.42799 0.614838
36 15 929 914 101.525 108 0.246686 0.614385
37 15 954 939 101.483 100 0.660536 0.614812
38 16 978 962 101.233 92 1.19454 0.615563
39 15 1004 989 101.406 108 0.913785 0.616659
2019-05-18 16:33:42.108159 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.615418
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1034 1018 101.77 116 0.642701 0.615418
41 16 1059 1043 101.726 100 0.907593 0.613914
42 16 1084 1068 101.684 100 0.374597 0.614907
43 16 1110 1094 101.737 104 0.96958 0.615492
44 16 1136 1120 101.788 104 0.840982 0.615351
45 16 1161 1145 101.748 100 0.735584 0.615418
46 16 1189 1173 101.97 112 0.720255 0.614002
47 16 1213 1197 101.842 96 1.27825 0.614561
48 16 1242 1226 102.137 116 0.61307 0.614118
49 16 1267 1251 102.093 100 0.225578 0.611559
50 16 1293 1277 102.13 104 0.947885 0.613134
51 16 1317 1301 102.009 96 0.22036 0.612555
52 15 1344 1329 102.201 112 0.233945 0.613195
53 15 1369 1354 102.159 100 0.292725 0.613236
54 16 1396 1380 102.193 104 0.244423 0.612626
55 15 1420 1405 102.152 100 1.50651 0.61284
56 16 1443 1427 101.899 88 0.360348 0.614463
57 16 1469 1453 101.936 104 0.261534 0.615286
58 16 1496 1480 102.04 108 0.364362 0.613428
59 16 1522 1506 102.073 104 0.493277 0.613733
2019-05-18 16:34:02.113333 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.614096
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 1546 1530 101.971 96 0.330609 0.614096
61 16 1572 1556 102.004 104 1.52149 0.615369
62 16 1595 1579 101.842 92 0.318493 0.615996
63 15 1621 1606 101.939 108 0.977447 0.615806
64 16 1648 1632 101.971 104 0.086239 0.616227
65 16 1674 1658 102.002 104 0.663038 0.615419
66 16 1699 1683 101.971 100 0.365909 0.614713
67 16 1727 1711 102.12 112 0.308491 0.614882
68 16 1752 1736 102.075 100 1.05294 0.615594
69 16 1779 1763 102.161 108 0.217732 0.614623
70 16 1806 1790 102.244 108 1.60054 0.614634
71 16 1833 1817 102.325 108 0.232145 0.614549
72 16 1859 1843 102.346 104 0.226481 0.613269
73 16 1887 1871 102.477 112 0.309953 0.612531
74 16 1915 1899 102.605 112 0.294306 0.612399
75 16 1941 1925 102.624 104 0.239048 0.611767
76 15 1964 1949 102.536 96 0.361543 0.612859
77 15 1987 1972 102.398 92 0.600103 0.613286
78 16 2015 1999 102.47 108 0.105284 0.613295
79 15 2043 2028 102.64 116 1.30296 0.612416
2019-05-18 16:34:22.129791 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.612544
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 15 2066 2051 102.507 92 0.378624 0.612544
81 16 2090 2074 102.376 92 0.304344 0.613135
82 15 2114 2099 102.347 100 1.35539 0.61378
83 15 2141 2126 102.415 108 0.727947 0.612929
84 16 2168 2152 102.433 104 1.33487 0.612479
85 16 2193 2177 102.404 100 1.68349 0.612177
86 16 2219 2203 102.423 104 0.614819 0.61321
87 16 2245 2229 102.441 104 1.02461 0.614046
88 16 2270 2254 102.412 100 0.728825 0.614221
89 16 2297 2281 102.475 108 0.500017 0.613279
90 16 2321 2305 102.402 96 0.542202 0.613601
91 16 2344 2328 102.287 92 0.428398 0.614248
92 16 2368 2352 102.219 96 0.261816 0.614744
93 16 2393 2377 102.195 100 1.60576 0.615061
94 16 2417 2401 102.129 96 0.504581 0.615406
95 16 2442 2426 102.106 100 0.238524 0.6155
96 15 2468 2453 102.167 108 0.345518 0.615092
97 16 2494 2478 102.144 100 0.639688 0.614828
98 16 2515 2499 101.959 84 0.339075 0.616186
99 16 2539 2523 101.898 96 1.35541 0.61736
2019-05-18 16:34:42.136600 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.617872
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 2564 2548 101.879 100 1.18894 0.617872
101 16 2588 2572 101.818 96 1.41383 0.617181
102 15 2609 2594 101.682 88 1.41548 0.618115
103 16 2637 2621 101.743 108 1.59457 0.618172
104 15 2660 2645 101.687 96 1.29024 0.618203
105 15 2683 2668 101.595 92 0.271359 0.618088
106 16 2709 2693 101.579 100 0.271561 0.619286
107 16 2736 2720 101.639 108 1.21346 0.619218
108 16 2761 2745 101.623 100 0.23701 0.619419
109 15 2782 2767 101.498 88 1.11943 0.620276
110 16 2803 2787 101.302 80 0.127282 0.620966
111 16 2828 2812 101.29 100 1.19177 0.621593
112 15 2854 2839 101.349 108 0.500804 0.621117
113 15 2883 2868 101.479 116 0.979941 0.621037
114 15 2908 2893 101.465 100 0.595129 0.620529
115 16 2934 2918 101.452 100 0.631809 0.620587
116 16 2958 2942 101.405 96 1.30586 0.620826
117 16 2983 2967 101.392 100 1.43655 0.6207
118 16 3012 2996 101.516 116 0.28437 0.620198
119 15 3037 3022 101.536 104 0.396062 0.620285
2019-05-18 16:35:02.147676 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.620297
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 15 3059 3044 101.423 88 1.20749 0.620297
121 16 3085 3069 101.411 100 0.289698 0.619866
122 16 3109 3093 101.366 96 1.47013 0.62052
123 15 3137 3122 101.484 116 0.539145 0.620032
124 16 3163 3147 101.472 100 0.300984 0.620212
125 16 3187 3171 101.428 96 0.245069 0.620262
126 16 3212 3196 101.416 100 0.382789 0.619751
127 15 3237 3222 101.436 104 0.624405 0.620601
128 15 3265 3250 101.518 112 0.150987 0.620461
129 16 3288 3272 101.413 88 0.328893 0.620595
130 15 3313 3298 101.433 104 0.524091 0.621403
131 16 3339 3323 101.421 100 0.289123 0.620895
132 16 3368 3352 101.531 116 0.395289 0.62023
133 16 3392 3376 101.489 96 1.36399 0.61999
134 16 3419 3403 101.538 108 0.107805 0.619212
135 16 3444 3428 101.526 100 0.380014 0.619493
136 15 3471 3456 101.603 112 0.553851 0.619538
137 16 3497 3481 101.591 100 0.413806 0.619751
138 16 3519 3503 101.492 88 1.57274 0.619977
139 16 3545 3529 101.51 104 0.812276 0.621023
2019-05-18 16:35:22.157837 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.620722
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 3571 3555 101.527 104 0.701561 0.620722
141 16 3595 3579 101.487 96 0.641256 0.620624
142 16 3619 3603 101.448 96 0.406191 0.62071
143 16 3644 3628 101.438 100 0.847267 0.62114
144 15 3668 3653 101.427 100 1.02263 0.62112
145 16 3695 3679 101.445 104 0.690649 0.621159
146 16 3718 3702 101.38 92 1.16805 0.620918
147 16 3744 3728 101.397 104 0.869129 0.62144
148 16 3769 3753 101.388 100 0.553545 0.621529
149 16 3792 3776 101.324 92 0.461667 0.62138
150 16 3818 3802 101.342 104 0.291619 0.621279
151 16 3844 3828 101.359 104 0.973913 0.621405
152 16 3869 3853 101.35 100 0.369249 0.621225
153 16 3895 3879 101.367 104 1.42668 0.621382
154 16 3919 3903 101.332 96 0.238997 0.621625
155 16 3946 3930 101.375 108 1.34762 0.621129
156 16 3969 3953 101.315 92 1.17845 0.621669
157 16 3995 3979 101.332 104 0.473975 0.621859
158 15 4020 4005 101.348 104 0.584256 0.621786
159 15 4043 4028 101.289 92 0.267278 0.621693
2019-05-18 16:35:42.166110 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.621476
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 4071 4055 101.331 108 0.194401 0.621476
161 16 4096 4080 101.322 100 0.315621 0.62117
162 16 4123 4107 101.363 108 0.322259 0.621174
163 16 4148 4132 101.354 100 0.48933 0.621475
164 15 4168 4153 101.248 84 0.285016 0.621883
165 16 4194 4178 101.241 100 1.00267 0.622445
166 16 4222 4206 101.305 112 0.302076 0.622458
167 16 4247 4231 101.297 100 0.607249 0.622223
168 16 4271 4255 101.265 96 1.10633 0.622223
169 15 4295 4280 101.257 100 1.40437 0.622956
170 16 4320 4304 101.226 96 1.29874 0.622788
171 16 4348 4332 101.289 112 0.891543 0.622758
172 16 4373 4357 101.281 100 1.01804 0.622526
173 16 4397 4381 101.25 96 0.314154 0.622338
174 16 4425 4409 101.312 112 0.280333 0.622013
175 16 4453 4437 101.372 112 0.507101 0.621668
176 16 4478 4462 101.364 100 0.369535 0.621579
177 16 4500 4484 101.288 88 0.286021 0.621995
178 16 4524 4508 101.258 96 0.0986942 0.62182
179 16 4550 4534 101.274 104 0.291024 0.622242
2019-05-18 16:36:02.175971 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.622272
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 15 4572 4557 101.222 92 0.134211 0.622272
181 15 4599 4584 101.259 108 1.43393 0.62254
182 16 4625 4609 101.252 100 0.283741 0.622358
183 16 4650 4634 101.245 100 1.54544 0.622435
184 15 4675 4660 101.259 104 0.855797 0.622905
185 15 4699 4684 101.23 96 0.996938 0.623006
186 16 4723 4707 101.18 92 0.508726 0.622852
187 16 4748 4732 101.174 100 1.15762 0.623139
188 16 4773 4757 101.167 100 0.300548 0.622578
189 16 4800 4784 101.203 108 1.6523 0.622247
190 16 4827 4811 101.238 108 0.335725 0.622353
191 16 4853 4837 101.252 104 0.297378 0.62186
192 16 4879 4863 101.266 104 0.378816 0.622332
193 16 4905 4889 101.28 104 0.630036 0.62254
194 16 4929 4913 101.252 96 1.31716 0.62255
195 16 4953 4937 101.225 96 1.44665 0.622514
196 16 4979 4963 101.239 104 1.29166 0.62281
197 16 5002 4986 101.192 92 0.649523 0.622641
198 16 5025 5009 101.145 92 1.40315 0.622802
199 16 5051 5035 101.159 104 1.85447 0.623106
2019-05-18 16:36:22.189333 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.623351
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 5077 5061 101.173 104 0.228415 0.623351
201 16 5102 5086 101.167 100 1.1258 0.623219
202 16 5128 5112 101.181 104 0.144679 0.623165
203 16 5152 5136 101.155 96 0.45859 0.623267
204 15 5176 5161 101.149 100 0.246263 0.623226
205 16 5200 5184 101.104 92 0.15298 0.622968
206 16 5226 5210 101.118 104 0.314363 0.623264
207 16 5255 5239 101.19 116 0.296073 0.622894
208 16 5277 5261 101.126 88 0.123729 0.623258
209 16 5303 5287 101.14 104 0.243795 0.623575
210 16 5326 5310 101.096 92 0.683485 0.623494
211 16 5350 5334 101.071 96 0.250412 0.623411
212 16 5377 5361 101.104 108 1.42768 0.623551
213 16 5401 5385 101.08 96 0.471156 0.623626
214 16 5429 5413 101.13 112 1.12058 0.623706
215 16 5457 5441 101.181 112 0.149699 0.623356
216 15 5482 5467 101.194 104 0.382906 0.623127
217 16 5506 5490 101.151 92 1.43476 0.6236
218 16 5531 5515 101.145 100 0.831024 0.623643
219 15 5557 5542 101.173 108 1.26516 0.62353
2019-05-18 16:36:42.207277 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.623377
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 15 5581 5566 101.149 96 1.41688 0.623377
221 16 5607 5591 101.143 100 1.22669 0.623508
222 16 5635 5619 101.192 112 0.702633 0.623437
223 15 5660 5645 101.204 104 0.658568 0.623323
224 16 5685 5669 101.181 96 0.450759 0.623488
225 16 5712 5696 101.211 108 0.695003 0.623314
226 16 5742 5726 101.294 120 0.289723 0.6227
227 16 5766 5750 101.27 96 0.739123 0.622798
228 15 5790 5775 101.264 100 0.321139 0.622755
229 15 5816 5801 101.276 104 0.998477 0.622826
230 16 5844 5828 101.305 108 0.353511 0.622642
231 16 5868 5852 101.282 96 0.795351 0.62282
232 16 5893 5877 101.276 100 0.849987 0.622851
233 16 5919 5903 101.288 104 0.208041 0.622614
234 15 5944 5929 101.299 104 0.222121 0.622392
235 15 5971 5956 101.327 108 1.20031 0.62237
236 15 5998 5983 101.356 108 0.129616 0.621988
237 16 6024 6008 101.35 100 0.360037 0.622042
238 16 6049 6033 101.344 100 0.266008 0.622015
239 15 6076 6061 101.388 112 0.616195 0.622186
2019-05-18 16:37:02.217008 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.621925
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 6102 6086 101.382 100 0.292008 0.621925
241 16 6127 6111 101.376 100 0.477008 0.621951
242 16 6153 6137 101.387 104 1.52713 0.621981
243 16 6176 6160 101.348 92 1.05464 0.622165
244 15 6199 6184 101.326 96 0.910721 0.622189
245 15 6225 6210 101.337 104 1.07306 0.622479
246 15 6253 6238 101.38 112 0.828068 0.6224
247 16 6277 6261 101.342 92 0.420895 0.622246
248 16 6303 6287 101.352 104 0.214216 0.622144
249 16 6331 6315 101.395 112 0.638923 0.62214
250 16 6356 6340 101.389 100 0.238125 0.621844
251 16 6380 6364 101.367 96 1.45499 0.622043
252 16 6406 6390 101.377 104 0.417803 0.622315
253 15 6431 6416 101.388 104 0.104336 0.622099
254 16 6456 6440 101.366 96 0.278432 0.622136
255 16 6480 6464 101.345 96 0.243936 0.622064
256 15 6506 6491 101.371 108 0.214287 0.621874
257 16 6530 6514 101.334 92 0.267618 0.621983
258 16 6555 6539 101.329 100 0.454341 0.622434
259 15 6579 6564 101.323 100 0.313691 0.622321
2019-05-18 16:37:22.227463 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.621957
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 15 6606 6591 101.349 108 0.0984729 0.621957
261 15 6630 6615 101.328 96 0.186963 0.622095
262 16 6657 6641 101.338 104 0.309167 0.622338
263 16 6684 6668 101.363 108 0.470031 0.622299
264 16 6708 6692 101.343 96 0.357501 0.62221
265 16 6728 6712 101.262 80 0.874509 0.622712
266 15 6751 6736 101.242 96 0.636873 0.623195
267 15 6776 6761 101.237 100 0.811325 0.623353
268 16 6799 6783 101.188 88 0.299508 0.623158
269 15 6825 6810 101.213 108 0.423659 0.623193
270 16 6850 6834 101.193 96 0.320671 0.623018
271 16 6877 6861 101.218 108 0.461819 0.622948
272 15 6905 6890 101.272 116 1.1124 0.622878
273 15 6931 6916 101.282 104 0.653229 0.622811
274 15 6957 6942 101.292 104 0.33856 0.622757
275 16 6983 6967 101.287 100 0.276897 0.622739
276 16 7009 6993 101.297 104 0.917901 0.622597
277 16 7034 7018 101.292 100 1.29641 0.622527
278 16 7060 7044 101.302 104 0.675619 0.622507
279 16 7083 7067 101.268 92 0.48588 0.622615
2019-05-18 16:37:42.236221 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.622716
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 7108 7092 101.264 100 0.250136 0.622716
281 16 7133 7117 101.259 100 0.837257 0.623053
282 16 7158 7142 101.254 100 0.62718 0.622826
283 15 7185 7170 101.292 112 0.141372 0.622738
284 16 7208 7192 101.245 88 0.217206 0.622943
285 16 7233 7217 101.241 100 0.207664 0.622944
286 15 7257 7242 101.236 100 0.33304 0.622862
287 16 7282 7266 101.217 96 0.629921 0.622786
288 15 7305 7290 101.199 96 0.345602 0.622856
289 15 7334 7319 101.25 116 0.473422 0.622672
290 16 7360 7344 101.246 100 0.953368 0.622936
291 16 7381 7365 101.186 84 0.246506 0.622984
292 16 7408 7392 101.21 108 0.325467 0.623197
293 16 7432 7416 101.192 96 0.0971356 0.62308
294 15 7456 7441 101.187 100 0.0976546 0.623063
295 15 7482 7467 101.197 104 0.978687 0.623418
296 16 7506 7490 101.166 92 1.04283 0.623518
297 16 7532 7516 101.175 104 0.452704 0.623388
298 15 7556 7541 101.171 100 0.239914 0.623335
299 16 7580 7564 101.14 92 0.268488 0.623272
2019-05-18 16:38:02.245811 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.62314
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 7604 7588 101.123 96 0.261678 0.62314
301 16 7629 7613 101.119 100 0.345197 0.623453
302 16 7655 7639 101.128 104 1.27369 0.623607
303 16 7681 7665 101.138 104 0.424676 0.623547
304 16 7704 7688 101.108 92 0.489889 0.623969
305 16 7727 7711 101.077 92 0.409788 0.624143
306 16 7756 7740 101.126 116 0.781044 0.623874
307 16 7784 7768 101.161 112 0.308747 0.623663
308 16 7809 7793 101.157 100 0.385183 0.623397
309 16 7836 7820 101.179 108 1.08285 0.623437
310 16 7861 7845 101.176 100 0.219953 0.623419
311 16 7886 7870 101.172 100 0.473606 0.623255
312 16 7913 7897 101.193 108 0.373179 0.623166
313 16 7940 7924 101.215 108 0.299289 0.623072
314 16 7961 7945 101.16 84 0.460902 0.623042
315 16 7989 7973 101.194 112 1.6262 0.623145
316 16 8013 7997 101.177 96 1.03781 0.623491
317 16 8038 8022 101.173 100 0.415992 0.623553
318 15 8067 8052 101.232 120 0.326914 0.623371
319 16 8090 8074 101.191 88 0.715763 0.623521
2019-05-18 16:38:22.256352 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.623574
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 8113 8097 101.162 92 0.642659 0.623574
321 16 8138 8122 101.158 100 0.290379 0.623648
322 16 8164 8148 101.167 104 0.294173 0.623614
323 16 8190 8174 101.175 104 0.283546 0.623708
324 16 8213 8197 101.147 92 0.900547 0.623813
325 15 8237 8222 101.143 100 0.190628 0.623697
326 16 8263 8247 101.14 100 0.259647 0.623555
327 15 8289 8274 101.16 108 0.67046 0.62356
328 16 8316 8300 101.169 104 0.612266 0.623628
329 15 8345 8330 101.226 120 0.10275 0.6233
330 15 8371 8356 101.234 104 0.306141 0.623092
331 15 8398 8383 101.255 108 0.499758 0.622897
332 16 8424 8408 101.251 100 0.334223 0.622795
333 16 8451 8435 101.271 108 0.241176 0.622599
334 16 8478 8462 101.291 108 0.897724 0.622809
335 16 8502 8486 101.275 96 0.610627 0.622765
336 15 8528 8513 101.295 108 0.61575 0.62254
337 16 8555 8539 101.303 104 1.26721 0.622779
338 15 8579 8564 101.299 100 0.453215 0.622726
339 15 8607 8592 101.33 112 0.17955 0.62261
2019-05-18 16:38:42.264516 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.62247
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 8635 8619 101.35 108 0.0334024 0.62247
341 16 8661 8645 101.357 104 0.988156 0.622354
342 16 8689 8673 101.388 112 1.26421 0.62202
343 15 8715 8700 101.407 108 0.312452 0.621784
344 16 8739 8723 101.38 92 1.32538 0.622013
345 16 8763 8747 101.364 96 0.315033 0.622122
346 16 8789 8773 101.372 104 1.32989 0.622242
347 16 8813 8797 101.356 96 0.302235 0.622284
348 16 8837 8821 101.341 96 0.288684 0.62229
349 16 8862 8846 101.337 100 1.31734 0.622471
350 16 8890 8874 101.367 112 0.924812 0.622399
351 16 8916 8900 101.374 104 0.164308 0.622384
352 15 8940 8925 101.37 100 0.333364 0.622447
353 16 8967 8951 101.378 104 0.533656 0.622316
354 15 8994 8979 101.408 112 0.899128 0.622287
355 15 9018 9003 101.392 96 0.108629 0.622123
356 16 9045 9029 101.399 104 1.38919 0.621981
357 16 9069 9053 101.384 96 0.321945 0.622057
358 16 9094 9078 101.38 100 1.39323 0.622078
359 16 9119 9103 101.376 100 0.333002 0.622029
2019-05-18 16:39:02.273571 min lat: 0.0162125 max lat: 2.24639 avg lat: 0.62199
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 9145 9129 101.383 104 0.313679 0.62199
361 16 9173 9157 101.413 112 0.359067 0.621675
362 16 9197 9181 101.398 96 1.97909 0.621929
363 16 9219 9203 101.36 88 0.390756 0.62227
364 16 9245 9229 101.368 104 0.267295 0.622265
365 16 9271 9255 101.375 104 0.398881 0.622302
366 16 9297 9281 101.382 104 0.165758 0.622169
367 16 9320 9304 101.356 92 0.409822 0.622241
368 16 9347 9331 101.374 108 0.319764 0.622099
369 16 9374 9358 101.392 108 0.708846 0.622193
370 16 9398 9382 101.377 96 0.870945 0.622281
371 16 9423 9407 101.373 100 0.016725 0.622284
372 16 9451 9435 101.402 112 0.398917 0.622177
373 16 9477 9461 101.409 104 0.473994 0.622106
374 16 9498 9482 101.362 84 0.709182 0.622427
375 16 9523 9507 101.358 100 0.82152 0.622516
376 16 9547 9531 101.344 96 0.0842975 0.622507
377 16 9569 9553 101.308 88 0.352341 0.62252
378 16 9591 9575 101.273 88 0.222197 0.622671
379 16 9616 9600 101.27 100 1.9463 0.622653
2019-05-18 16:39:22.281595 min lat: 0.0162125 max lat: 2.24639 avg lat: 0.622871
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 15 9641 9626 101.277 104 0.556325 0.622871
381 16 9668 9652 101.284 104 0.271111 0.622929
382 16 9693 9677 101.281 100 0.724258 0.623197
383 15 9719 9704 101.298 108 0.304844 0.622945
384 16 9742 9726 101.263 88 0.181045 0.622843
385 16 9766 9750 101.25 96 0.250694 0.623063
386 16 9791 9775 101.246 100 0.736536 0.623055
387 16 9815 9799 101.233 96 0.352645 0.623219
388 16 9843 9827 101.26 112 0.554207 0.6232
389 16 9870 9854 101.278 108 0.197903 0.623133
390 16 9895 9879 101.274 100 0.143791 0.622853
391 16 9923 9907 101.301 112 1.55711 0.622798
392 16 9946 9930 101.278 92 1.73373 0.622746
393 16 9973 9957 101.295 108 1.0784 0.622856
394 16 9997 9981 101.281 96 0.382286 0.62273
395 16 10024 10008 101.298 108 0.360277 0.622619
396 16 10047 10031 101.274 92 0.435118 0.622975
397 16 10077 10061 101.321 120 0.39789 0.622904
398 15 10101 10086 101.318 100 0.607862 0.622839
399 16 10127 10111 101.314 100 0.325718 0.622647
2019-05-18 16:39:42.289432 min lat: 0.0162125 max lat: 2.24639 avg lat: 0.622821
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 10152 10136 101.311 100 0.477864 0.622821
401 16 10177 10161 101.307 100 0.609951 0.622777
402 16 10203 10187 101.314 104 0.910757 0.622761
403 16 10225 10209 101.281 88 0.429068 0.622846
404 16 10251 10235 101.288 104 0.631193 0.622948
405 16 10274 10258 101.265 92 0.874939 0.623077
406 16 10301 10285 101.281 108 0.902182 0.623067
407 16 10325 10309 101.268 96 1.37668 0.622906
408 16 10352 10336 101.284 108 0.456801 0.623025
409 16 10378 10362 101.291 104 0.131875 0.622916
410 15 10406 10391 101.326 116 0.957796 0.622832
411 16 10434 10418 101.343 108 0.512615 0.622766
412 16 10457 10441 101.32 92 0.998587 0.622743
413 16 10482 10466 101.317 100 0.910021 0.622885
414 16 10509 10493 101.333 108 0.179019 0.622681
415 16 10535 10519 101.339 104 0.148482 0.622501
416 16 10562 10546 101.355 108 0.230529 0.622384
417 16 10587 10571 101.352 100 0.456228 0.622491
418 16 10614 10598 101.367 108 0.255072 0.622472
419 16 10639 10623 101.364 100 0.482745 0.62235
2019-05-18 16:40:02.298111 min lat: 0.0162125 max lat: 2.24639 avg lat: 0.622485
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 10661 10645 101.332 88 0.207673 0.622485
421 16 10689 10673 101.357 112 0.358032 0.62242
422 16 10714 10698 101.354 100 0.32986 0.622377
423 16 10739 10723 101.351 100 0.361026 0.622366
424 15 10761 10746 101.329 92 0.628947 0.622615
425 16 10786 10770 101.316 96 0.315204 0.622629
426 16 10811 10795 101.313 100 1.88423 0.622736
427 16 10836 10820 101.31 100 1.3641 0.622664
428 16 10860 10844 101.297 96 0.18561 0.622787
429 16 10885 10869 101.294 100 1.53478 0.622776
430 15 10907 10892 101.273 92 0.463149 0.622786
431 16 10932 10916 101.26 96 0.348039 0.622989
432 16 10958 10942 101.267 104 1.2145 0.622954
433 16 10980 10964 101.236 88 1.74358 0.623082
434 16 11005 10989 101.233 100 1.37648 0.623134
435 16 11031 11015 101.239 104 0.318803 0.62318
436 16 11056 11040 101.236 100 1.38882 0.623289
437 16 11081 11065 101.234 100 1.40606 0.623378
438 16 11103 11087 101.203 88 1.43175 0.623454
439 16 11126 11110 101.182 92 1.34274 0.623678
2019-05-18 16:40:22.303904 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.623615
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 11150 11134 101.17 96 1.34174 0.623615
441 16 11178 11162 101.195 112 0.364975 0.623576
442 16 11203 11187 101.192 100 0.286892 0.623624
443 15 11229 11214 101.207 108 0.165792 0.62332
444 15 11253 11238 101.196 96 1.32024 0.62343
445 16 11274 11258 101.148 80 0.248892 0.623578
446 16 11301 11285 101.163 108 1.62232 0.623629
447 16 11328 11312 101.178 108 0.14199 0.623607
448 16 11352 11336 101.167 96 0.432177 0.623638
449 16 11376 11360 101.155 96 1.03429 0.623848
450 16 11405 11389 101.188 116 0.400924 0.623709
451 15 11428 11413 101.176 96 0.198013 0.623615
452 16 11455 11439 101.183 104 0.281004 0.623568
453 16 11479 11463 101.171 96 0.933552 0.623691
454 16 11506 11490 101.186 108 0.24476 0.62357
455 15 11530 11515 101.183 100 0.377038 0.623707
456 16 11553 11537 101.154 88 0.351084 0.623754
457 16 11579 11563 101.161 104 0.733259 0.623832
458 16 11603 11587 101.149 96 1.28903 0.62382
459 16 11627 11611 101.138 96 0.707616 0.623828
2019-05-18 16:40:42.310545 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.623834
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 11655 11639 101.162 112 1.0956 0.623834
461 15 11682 11667 101.185 112 0.138695 0.623625
462 16 11708 11692 101.182 100 0.423769 0.623504
463 16 11732 11716 101.171 96 0.135995 0.623511
464 16 11755 11739 101.151 92 0.991618 0.623869
465 16 11781 11765 101.157 104 0.21035 0.623825
466 15 11807 11792 101.172 108 0.732382 0.623807
467 16 11834 11818 101.178 104 0.154895 0.623704
468 16 11858 11842 101.167 96 0.593438 0.623723
469 15 11885 11870 101.19 112 0.846727 0.623747
470 16 11909 11893 101.17 92 0.645488 0.623661
471 15 11934 11919 101.176 104 0.0976751 0.623595
472 16 11959 11943 101.165 96 1.50445 0.623631
473 16 11983 11967 101.154 96 0.29084 0.623521
474 16 12006 11990 101.135 92 0.505835 0.623797
475 15 12036 12021 101.183 124 0.769563 0.623879
476 16 12062 12046 101.18 100 0.254357 0.623686
477 16 12086 12070 101.169 96 0.310546 0.623523
478 16 12112 12096 101.175 104 0.362366 0.623741
479 16 12139 12123 101.189 108 1.02392 0.623669
2019-05-18 16:41:02.317042 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.62351
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 12163 12147 101.178 96 0.26893 0.62351
481 16 12191 12175 101.201 112 1.66726 0.62341
482 15 12215 12200 101.198 100 0.263703 0.623547
483 16 12242 12226 101.204 104 0.0997891 0.623308
484 16 12266 12250 101.193 96 0.561882 0.623572
485 16 12290 12274 101.183 96 0.252513 0.623642
486 16 12320 12304 101.221 120 1.31505 0.623549
487 16 12345 12329 101.219 100 0.488425 0.623551
488 16 12367 12351 101.191 88 0.258854 0.623593
489 16 12392 12376 101.189 100 0.35106 0.623608
490 16 12416 12400 101.178 96 0.276139 0.623636
491 16 12440 12424 101.168 96 0.398463 0.623796
492 16 12467 12451 101.182 108 0.627924 0.623832
493 16 12489 12473 101.155 88 1.15664 0.623973
494 15 12515 12500 101.169 108 0.232104 0.623907
495 16 12540 12524 101.158 96 0.221388 0.623876
496 16 12567 12551 101.172 108 0.556117 0.623892
497 16 12591 12575 101.161 96 0.296713 0.623887
498 16 12616 12600 101.159 100 0.891934 0.623997
499 16 12643 12627 101.173 108 0.755513 0.623871
2019-05-18 16:41:22.322391 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.62367
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 12671 12655 101.194 112 1.15459 0.62367
501 16 12696 12680 101.192 100 1.66049 0.623678
502 15 12719 12704 101.181 96 0.141729 0.623795
503 16 12745 12729 101.179 100 0.208191 0.623707
504 16 12771 12755 101.184 104 0.288907 0.623617
505 16 12792 12776 101.15 84 1.13545 0.623843
506 16 12816 12800 101.14 96 1.56839 0.623935
507 16 12844 12828 101.162 112 0.427444 0.623991
508 16 12871 12855 101.175 108 1.3028 0.623872
509 16 12894 12878 101.157 92 0.487957 0.623944
510 16 12922 12906 101.178 112 0.303017 0.623721
511 15 12943 12928 101.152 88 0.613304 0.623758
512 15 12969 12954 101.158 104 0.379331 0.623884
513 16 12998 12982 101.179 112 0.268611 0.623842
514 16 13023 13007 101.176 100 1.24489 0.623836
515 15 13044 13029 101.151 88 0.748533 0.623915
516 16 13067 13051 101.125 88 0.249582 0.623969
517 16 13092 13076 101.123 100 0.178046 0.624084
518 16 13117 13101 101.121 100 0.243927 0.6241
519 15 13143 13128 101.134 108 0.216457 0.623941
2019-05-18 16:41:42.328125 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.62396
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 15 13168 13153 101.132 100 0.268564 0.62396
521 16 13194 13178 101.13 100 0.65215 0.624151
522 16 13223 13207 101.158 116 0.16556 0.623938
523 16 13248 13232 101.156 100 1.15989 0.623904
524 16 13274 13258 101.161 104 1.54665 0.623833
525 16 13301 13285 101.174 108 0.84592 0.623819
526 15 13322 13307 101.149 88 0.521004 0.623804
527 16 13344 13328 101.116 84 0.238329 0.623946
528 15 13369 13354 101.122 104 0.423115 0.623914
529 16 13394 13378 101.112 96 0.264633 0.624021
530 16 13420 13404 101.118 104 0.484972 0.624191
531 16 13446 13430 101.123 104 0.840807 0.624324
532 16 13471 13455 101.121 100 0.336517 0.62414
533 15 13495 13480 101.119 100 0.296705 0.624111
534 15 13521 13506 101.124 104 0.398844 0.624245
535 15 13549 13534 101.144 112 0.531099 0.624137
536 16 13577 13561 101.157 108 0.537673 0.623909
537 16 13605 13589 101.177 112 0.270723 0.623817
538 16 13628 13612 101.16 92 1.38737 0.623876
539 16 13651 13635 101.143 92 0.66444 0.623879
2019-05-18 16:42:02.333256 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.623895
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 13677 13661 101.148 104 0.485122 0.623895
541 16 13705 13689 101.168 112 0.325711 0.623817
542 15 13729 13714 101.166 100 0.23929 0.623739
543 16 13754 13738 101.156 96 0.225668 0.623774
544 16 13778 13762 101.147 96 0.314298 0.624024
545 16 13804 13788 101.152 104 0.299513 0.623983
546 16 13830 13814 101.157 104 0.423109 0.623898
547 15 13851 13836 101.133 88 0.349053 0.624062
548 16 13878 13862 101.138 104 0.815443 0.624154
549 16 13904 13888 101.144 104 0.829252 0.624041
550 16 13930 13914 101.149 104 0.227483 0.624092
551 16 13954 13938 101.139 96 0.802361 0.62404
552 15 13979 13964 101.144 104 0.619761 0.62405
553 16 14004 13988 101.135 96 0.578863 0.624097
554 16 14032 14016 101.155 112 0.423879 0.623955
555 16 14057 14041 101.152 100 0.255772 0.623944
556 15 14084 14069 101.172 112 0.423165 0.623851
557 16 14106 14090 101.141 84 1.00822 0.624029
558 16 14133 14117 101.153 108 0.28102 0.624015
559 16 14158 14142 101.151 100 0.891352 0.624117
2019-05-18 16:42:22.338394 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.62392
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 14187 14171 101.178 116 1.08444 0.62392
561 16 14210 14194 101.161 92 0.361377 0.623915
562 16 14233 14217 101.145 92 0.641609 0.624037
563 16 14260 14244 101.157 108 0.656376 0.624076
564 16 14281 14265 101.127 84 0.327605 0.624055
565 15 14306 14291 101.132 104 1.09122 0.624213
566 16 14333 14317 101.137 104 1.12775 0.624032
567 16 14361 14345 101.156 112 0.233945 0.623897
568 16 14386 14370 101.154 100 0.4656 0.623927
569 16 14409 14393 101.138 92 1.49445 0.624031
570 16 14434 14418 101.135 100 1.17812 0.624084
571 16 14461 14445 101.147 108 0.245416 0.623968
572 16 14484 14468 101.131 92 0.211553 0.624014
573 16 14510 14494 101.136 104 0.316826 0.623956
574 15 14538 14523 101.162 116 1.30802 0.62402
575 15 14562 14547 101.153 96 0.259787 0.623988
576 16 14587 14571 101.144 96 0.370403 0.624095
577 16 14611 14595 101.135 96 0.301201 0.624094
578 16 14637 14621 101.14 104 0.35234 0.624165
579 16 14663 14647 101.145 104 1.00704 0.624065
2019-05-18 16:42:42.343942 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.624077
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 14688 14672 101.143 100 0.681862 0.624077
581 16 14713 14697 101.141 100 0.264823 0.624135
582 16 14734 14718 101.111 84 1.4241 0.624202
583 16 14758 14742 101.103 96 0.348541 0.624348
584 15 14783 14768 101.108 104 0.316723 0.62437
585 16 14811 14795 101.119 108 0.309481 0.624302
586 16 14835 14819 101.11 96 0.301595 0.62437
587 15 14859 14844 101.108 100 0.335854 0.624303
588 16 14886 14870 101.113 104 1.32654 0.624342
589 16 14914 14898 101.132 112 0.560732 0.62427
590 16 14937 14921 101.116 92 0.393511 0.624277
591 16 14963 14947 101.121 104 0.119612 0.624149
592 15 14989 14974 101.133 108 0.214154 0.62413
593 15 15014 14999 101.131 100 0.578366 0.624081
594 16 15039 15023 101.122 96 0.607088 0.624198
595 15 15063 15048 101.12 100 1.17425 0.624298
596 16 15088 15072 101.111 96 0.971621 0.624359
597 16 15113 15097 101.11 100 0.337654 0.624336
598 15 15138 15123 101.114 104 0.886411 0.624304
599 16 15165 15149 101.119 104 0.193092 0.624228
2019-05-18 16:43:02.350410 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.624284
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 16 15183 15167 101.071 72 0.382814 0.624284
Total time run: 600.630222
Total reads made: 15183
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 101.114
Average IOPS: 25
Stddev IOPS: 1
Max IOPS: 31
Min IOPS: 18
Average Latency(s): 0.624714
Max latency(s): 2.24639
Min latency(s): 0.0160655