ODROID-HC2 Nodes After 10 Weeks In My Ceph Cluster

Table of Contents

It’s been about 2½ months since I started adding ODROID-HC2 nodes to my Ceph Luminous cluster.

This is a quick update on how it’s doing so far.

The nodes are still on Fedora 29 armhfp, I’ll have to deal with an upgrade fop F30 or later within the next 6 months.

What This Post is About

Until recently I would adjust my playbooks, roles and fiddle with Ceph, followed by updating previous posts.

While that’s necessary (the updating of posts) since I pull in the latest versions of the Ansible bits, I feel I should record my findings so far.

Current Cluster State

[root@odroid-hc2-00 ~]# ceph -s
  cluster:
    id:     b1a134d0-927c-4eb9-9ae5-2f6902dce55c
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum odroid-hc2-00,odroid-hc2-01,odroid-hc2-02
    mgr: odroid-hc2-00(active), standbys: odroid-hc2-01, odroid-hc2-02
    mds: cephfs-1/1/1 up  {0=odroid-hc2-03=up:active}
    osd: 7 osds: 7 up, 7 in
    rgw: 1 daemon active
 
  data:
    pools:   9 pools, 208 pgs
    objects: 17.50k objects, 66.8GiB
    usage:   207GiB used, 9.80TiB / 10.0TiB avail
    pgs:     208 active+clean

The Bad

  • the nodes without serial console make more work than those with when something goes wrong with the low-cost hardware
  • the reboot issue plus having all nodes fed off one PSU is not such a good combination
  • while returns of broken hardware are painless in Europe, so far I had these failures
    • 3 µSD cards
    • 3 ODROID-U3 USB-UART MODULE
    • 1 ODROID-HC2

Room for Improvement

  • my current performance settings mean that the cluster needs a bit to spool up to speed (hence the test runs over 10 minutes (600 seconds) further down below)
    • Since this is a test cluster, I’d rather leave it that way, no need to nail down everything to max perf all the time, I’d rather save power.

The Good

  • the 2 nodes with constantly attached serial console (logging to a conserver) are quite painless
  • Ceph’s resilience means that even though the hardware side is too flaky for my taste (in my day job I nearly exclusively deal with enterprise grade x86_64 hardware), I have not lost any data stored on this Ceph cluster.
  • bandwidth is now within the levels of what can be expected of this low-cost hardware

Performance Measurements

Once again I gathered performance data, see 7. Benchmarking Performance of the RHCS 3 Administration Guide for details.

Write Test

[root@odroid-hc2-00 ~]# rados bench -p testbench 600 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_odroid-hc2-00.internal.pcfe.n_24309
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        20         4   16.0296        16    0.930154    0.855249
    2      16        37        21    42.031        68    0.632538    0.953409
    3      16        41        25   33.3464        16     0.33715    0.861344
    4      16        41        25   25.0047         0           -    0.861344
    5      16        41        25   20.0019         0           -    0.861344
    6      16        41        25   16.6672         0           -    0.861344
    7      16        41        25   14.2854         0           -    0.861344
    8      16        41        25   12.4992         0           -    0.861344
    9      16        41        25   11.1094         0           -    0.861344
   10      16        41        25   9.99799         0           -    0.861344
   11      16        41        25   9.08854         0           -    0.861344
   12      16        41        25   8.33066         0           -    0.861344
   13      16        41        25    7.6897         0           -    0.861344
   14      16        41        25   7.14035         0           -    0.861344
   15      16        41        25   6.66421         0           -    0.861344
   16      16        41        25   6.24756         0           -    0.861344
   17      16        50        34   7.99686   2.57143    0.333293     4.21942
   18      16        61        45   9.99615        44     1.01069     5.44457
   19      16        69        53   11.1537        32     2.69975     5.42909
2019-05-18 16:09:13.371129 min lat: 0.296097 max lat: 16.5859 avg lat: 4.66752
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16        82        66   13.1951        52    0.485576     4.66752
   21      16        97        81   15.4145        60     1.32828     4.01412
   22      16       110        94   17.0758        52     1.49541     3.62898
   23      16       128       112   19.4617        72     1.49855     3.23275
   24      16       137       121     20.15        36     1.73638     3.06026
   25      16       153       137   21.9023        64    0.717952     2.83943
   26      16       166       150   23.0588        52     1.74417     2.71984
   27      16       186       170   25.1659        80    0.714535     2.49741
   28      16       198       182   25.9806        48     0.61696     2.39038
   29      16       206       190   26.1877        32    0.328705     2.33554
   30      16       219       203   27.0471        52    0.548666     2.30743
   31      16       233       217   27.9664        56    0.544651     2.24442
   32      16       246       230   28.7163        52     0.79844     2.17362
   33      16       264       248   30.0262        72     1.31281     2.09175
   34      16       283       267   31.3767        76     1.37788     2.00764
   35      16       301       285   32.5349        72    0.608292     1.93916
   36      16       309       293   32.5196        32     0.35753     1.91056
   37      16       325       309   33.3691        64    0.499244     1.88549
   38      16       343       327   34.3842        72     1.20053     1.83698
   39      16       353       337   34.5273        40    0.558362     1.80665
2019-05-18 16:09:33.406019 min lat: 0.296097 max lat: 16.5859 avg lat: 1.78314
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16       369       353   35.2628        64    0.360414     1.78314
   41      16       384       368   35.8647        60     1.73654       1.753
   42      15       407       392   37.2946        96    0.481849     1.70109
   43      16       418       402   37.3555        40     0.50218     1.67719
   44      16       432       416   37.7782        56    0.526682      1.6628
   45      16       451       435   38.6257        76     1.07195     1.64098
   46      16       462       446   38.7406        44     1.23201     1.62098
   47      16       477       461   39.1918        60    0.504165     1.61046
   48      16       493       477   39.7079        64    0.602835     1.59419
   49      16       514       498   40.6107        84    0.645411     1.55728
   50      16       532       516   41.2376        72     1.00432     1.54031
   51      16       550       534     41.84        72     1.56522     1.51871
   52      16       560       544   41.8044        40    0.964614     1.50549
   53      16       576       560   42.2226        64    0.332654     1.49786
   54      16       587       571   42.2507        44      1.4148     1.49321
   55      16       598       582   42.2822        44     1.26732      1.4859
   56      16       611       595   42.4552        52    0.633864     1.48008
   57      16       627       611   42.8326        64    0.562681     1.47483
   58      16       633       617   42.5079        24     1.51784     1.47024
   59      16       646       630   42.6684        52    0.665011     1.47437
2019-05-18 16:09:53.424128 min lat: 0.260421 max lat: 16.5859 avg lat: 1.47003
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16       660       644   42.8902        56    0.409845     1.47003
   61      16       672       656   42.9735        48     1.44398     1.46796
   62      16       692       676   43.5699        80    0.456449     1.45481
   63      16       701       685   43.4497        36     1.73998     1.45229
   64      16       712       696   43.4529        44    0.493589     1.45176
   65      16       724       708   43.5227        48    0.312931     1.44536
   66      16       740       724   43.8324        64    0.451251     1.44561
   67      16       756       740    44.133        64    0.573374     1.43748
   68      16       769       753   44.2484        52    0.365358     1.42875
   69      16       791       775   44.8817        88     1.36392      1.4197
   70      16       802       786   44.8689        44     0.39191     1.41026
   71      16       821       805   45.3068        76     1.64609     1.40038
   72      16       837       821    45.566        64    0.622298      1.3894
   73      16       854       838   45.8728        68    0.539862     1.38664
   74      16       869       853   46.0634        60     1.07408     1.37739
   75      16       887       871    46.402        72     1.38983     1.36975
   76      16       902       886   46.5806        60    0.533069     1.36304
   77      16       914       898   46.5989        48    0.496655     1.35985
   78      16       931       915   46.8728        68    0.544218     1.35366
   79      16       941       925   46.7858        40    0.708357     1.35069
2019-05-18 16:10:13.447849 min lat: 0.260421 max lat: 16.5859 avg lat: 1.34329
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16       964       948   47.3502        92    0.745056     1.34329
   81      16       977       961   47.3968        52    0.371952     1.33965
   82      16       991       975   47.5001        56    0.678969     1.33616
   83      16      1009       993   47.7947        72    0.514251     1.32903
   84      16      1024      1008   47.9395        60     1.24164     1.32275
   85      16      1041      1025   48.1751        68    0.730028     1.31787
   86      16      1057      1041   48.3588        64     0.51351     1.31322
   87      16      1067      1051   48.2627        40     1.32062     1.31128
   88      16      1078      1062   48.2142        44    0.316048     1.30865
   89      16      1091      1075   48.2562        52     2.10825     1.31268
   90      16      1105      1089   48.3419        56     1.66736     1.31318
   91      16      1122      1106   48.5575        68     1.74501     1.30937
   92      16      1136      1120   48.6381        56     0.60998     1.30401
   93      16      1148      1132     48.63        48     1.74713     1.30776
   94      16      1155      1139   48.4106        28     1.79613     1.30918
   95      16      1172      1156   48.6165        68    0.560134     1.30849
   96      16      1183      1167   48.5683        44     1.38118     1.30658
   97      16      1197      1181   48.6446        56    0.591372     1.30637
   98      16      1208      1192   48.5969        44     1.91099      1.3036
   99      16      1222      1206   48.6713        56    0.550189     1.30388
2019-05-18 16:10:33.478342 min lat: 0.260421 max lat: 16.5859 avg lat: 1.30363
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      1235      1219   48.7042        52     2.11017     1.30363
  101      16      1249      1233   48.7761        56    0.552273     1.30166
  102      16      1264      1248   48.8857        60    0.303107     1.29806
  103      16      1277      1261   48.9157        52     1.78821     1.29731
  104      16      1286      1270   48.7914        36     1.74565     1.29844
  105      16      1303      1287    48.974        68    0.631927     1.29911
  106      16      1324      1308   49.3038        84    0.885227     1.29361
  107      16      1337      1321   49.3286        52    0.475104      1.2892
  108      16      1354      1338   49.5011        68    0.626061      1.2853
  109      15      1374      1359   49.8171        84    0.799113     1.27858
  110      16      1387      1371   49.8002        48      0.8391     1.27591
  111      16      1400      1384    49.819        52     1.76957     1.27751
  112      16      1413      1397   49.8382        52    0.414058     1.27581
  113      16      1425      1409   49.8216        48     1.58055     1.27407
  114      16      1434      1418      49.7        36    0.625207      1.2749
  115      16      1442      1426    49.546        32    0.573376     1.27731
  116      16      1461      1445   49.7734        76     1.54251     1.27927
  117      16      1479      1463   49.9629        72    0.615321     1.27566
  118      16      1486      1470   49.7769        28     1.40165     1.27336
  119      16      1502      1486   49.8961        64    0.686663     1.27543
2019-05-18 16:10:53.491667 min lat: 0.260421 max lat: 16.5859 avg lat: 1.27614
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      1513      1497   49.8469        44     1.68219     1.27614
  121      16      1527      1511   49.8971        56    0.935795     1.27465
  122      16      1544      1528   50.0451        68    0.666474     1.27167
  123      16      1559      1543   50.1254        60     1.46194     1.26863
  124      16      1570      1554   50.0758        44     0.32392     1.26717
  125      16      1590      1574   50.3148        80    0.875384     1.26823
  126      16      1599      1583    50.201        36     1.53562     1.26728
  127      16      1614      1598   50.2778        60     1.19224     1.26553
  128      16      1629      1613   50.3536        60    0.709749     1.26244
  129      16      1647      1631   50.5207        72     1.80161     1.26207
  130      16      1660      1644   50.5317        52     1.84457     1.26187
  131      16      1672      1656   50.5121        48    0.444286     1.26024
  132      16      1689      1673   50.6441        68     0.47536     1.25649
  133      16      1710      1694   50.8945        84    0.624648     1.25324
  134      16      1722      1706   50.8702        48     1.42011     1.25104
  135      16      1735      1719   50.8772        52     1.96487     1.25107
  136      16      1748      1732   50.8815        52      1.7548     1.25084
  137      16      1762      1746   50.9187        56    0.526288     1.25028
  138      16      1778      1762   51.0134        64    0.589687     1.24794
  139      16      1795      1779   51.1353        68    0.430793     1.24651
2019-05-18 16:11:13.524136 min lat: 0.260421 max lat: 16.5859 avg lat: 1.24183
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      1819      1803   51.4553        96    0.633373     1.24183
  141      16      1831      1815   51.4307        48    0.816879     1.23845
  142      16      1841      1825   51.3502        40     1.54713     1.23841
  143      16      1858      1842   51.4665        68     1.49424        1.24
  144      16      1869      1853   51.4146        44    0.509308     1.23792
  145      16      1883      1867   51.4461        56    0.429097     1.23692
  146      16      1898      1882   51.5045        60    0.459762     1.23583
  147      16      1913      1897   51.5621        60    0.905544     1.23579
  148      16      1923      1907   51.4839        40     0.65524     1.23417
  149      16      1934      1918   51.4319        44     2.16588     1.23903
  150      16      1947      1931   51.4356        52     0.32784     1.23821
  151      16      1960      1944   51.4392        52     0.80663     1.23628
  152      16      1981      1965   51.6531        84    0.743415     1.23557
  153      16      1990      1974   51.5508        36     0.41452      1.2332
  154      16      2008      1992   51.6834        72     1.37006     1.23235
  155      16      2021      2005   51.6853        52     1.58191     1.23132
  156      16      2038      2022   51.7897        68    0.302508       1.231
  157      16      2057      2041   51.9437        76     1.13743      1.2295
  158      16      2069      2053   51.9187        48     1.10888     1.22656
  159      16      2082      2066   51.9191        52    0.460752     1.22603
2019-05-18 16:11:33.534628 min lat: 0.260421 max lat: 16.5859 avg lat: 1.22595
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      2098      2082   51.9945        64    0.311714     1.22595
  161      16      2112      2096   52.0192        56    0.636634     1.22468
  162      16      2126      2110   52.0437        56    0.451941     1.22366
  163      16      2142      2126   52.1169        64      1.5064     1.22373
  164      16      2156      2140   52.1405        56     1.52295     1.22367
  165      16      2170      2154   52.1638        56     1.28338     1.22162
  166      16      2185      2169   52.2109        60     1.47192     1.22065
  167      16      2197      2181   52.1856        48    0.647264     1.21922
  168      16      2211      2195   52.2081        56    0.645364     1.22003
  169      16      2229      2213    52.325        72    0.676861     1.21939
  170      16      2248      2232   52.4641        76     1.54151     1.21728
  171      16      2255      2239   52.3211        28    0.347513     1.21549
  172      16      2269      2253   52.3424        56    0.654888     1.21679
  173      16      2284      2268   52.3865        60      1.2619     1.21645
  174      16      2300      2284   52.4531        64     1.29193     1.21674
  175      16      2314      2298   52.4733        56     1.44989     1.21519
  176      16      2326      2310   52.4478        48     1.72853      1.2156
  177      16      2337      2321   52.4001        44      1.3404     1.21573
  178      16      2350      2334   52.3978        52    0.567191     1.21587
  179      16      2360      2344   52.3285        40     1.44666     1.21578
2019-05-18 16:11:53.539663 min lat: 0.260421 max lat: 16.5859 avg lat: 1.21767
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      2373      2357   52.3266        52    0.393848     1.21767
  181      16      2386      2370   52.3247        52    0.278133     1.21819
  182      16      2397      2381   52.2788        44    0.590739     1.21835
  183      16      2409      2393   52.2554        48     1.60516     1.21978
  184      16      2424      2408   52.2974        60    0.455627     1.22111
  185      16      2435      2419   52.2525        44     1.31594     1.22062
  186      16      2444      2428   52.1651        36    0.568799     1.21909
  187      16      2466      2450   52.3564        88    0.892549     1.21922
  188      16      2478      2462   52.3332        48    0.299418       1.217
  189      16      2496      2480   52.4371        72    0.695307     1.21672
  190      16      2514      2498   52.5398        72    0.480004     1.21425
  191      16      2523      2507   52.4532        36     1.72035     1.21515
  192      16      2535      2519     52.43        48    0.605721     1.21559
  193      16      2550      2534   52.4691        60     1.38608     1.21557
  194      16      2565      2549   52.5078        60    0.268934      1.2137
  195      16      2578      2562   52.5051        52    0.476035     1.21393
  196      16      2593      2577   52.5432        60    0.459752     1.21394
  197      16      2609      2593   52.6013        64    0.515444     1.21261
  198      16      2625      2609   52.6587        64    0.431176     1.21234
  199      16      2643      2627   52.7531        72     1.17626      1.2102
2019-05-18 16:12:13.556963 min lat: 0.245755 max lat: 16.5859 avg lat: 1.21002
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      2649      2633   52.6092        24    0.381636     1.21002
  201      16      2665      2649   52.6657        64     1.97941     1.21109
  202      16      2679      2663   52.6821        56    0.433315     1.20983
  203      16      2696      2680   52.7573        68     1.30231      1.2103
  204      16      2701      2685   52.5968        20     1.44582     1.20977
  205      16      2716      2700   52.6328        60     1.39051     1.21112
  206      16      2726      2710   52.5715        40    0.531209     1.21193
  207      16      2739      2723   52.5686        52    0.513902     1.21093
  208      16      2753      2737    52.585        56     1.70676     1.21246
  209      16      2775      2759   52.7543        88     1.26044     1.21163
  210      16      2788      2772   52.7506        52     1.29441     1.20984
  211      16      2802      2786   52.7659        56    0.596399     1.20857
  212      16      2818      2802   52.8187        64    0.333844     1.20671
  213      16      2836      2820   52.9087        72     1.08406     1.20647
  214      16      2850      2834    52.923        56     1.33532     1.20574
  215      16      2864      2848   52.9372        56    0.599857     1.20524
  216      16      2874      2858   52.8773        40     1.99396     1.20439
  217      16      2888      2872   52.8916        56    0.614629     1.20525
  218      16      2902      2886   52.9058        56    0.345928     1.20577
  219      16      2919      2903   52.9746        68     1.22508      1.2058
2019-05-18 16:12:33.563410 min lat: 0.204739 max lat: 16.5859 avg lat: 1.20445
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      2934      2918   53.0064        60    0.321194     1.20445
  221      16      2944      2928   52.9476        40     1.16769     1.20475
  222      16      2956      2940   52.9176        48     2.03323      1.2043
  223      16      2970      2954   52.9313        56    0.361867     1.20369
  224      16      2984      2968   52.9449        56    0.698546     1.20355
  225      16      3002      2986   53.0293        72    0.580753      1.2037
  226      16      3017      3001   53.0601        60     1.21367     1.20232
  227      16      3032      3016   53.0905        60    0.741994     1.20071
  228      16      3050      3034   53.1731        72     1.32353     1.20131
  229      16      3065      3049   53.2027        60     1.08493     1.20039
  230      16      3085      3069    53.319        80     0.63936     1.19817
  231      16      3099      3083   53.3305        56     1.45461     1.19674
  232      16      3114      3098   53.3591        60     1.07505     1.19661
  233      16      3131      3115   53.4218        68     1.21683     1.19473
  234      16      3147      3131   53.4669        64    0.563806     1.19456
  235      16      3162      3146   53.4945        60    0.489637     1.19357
  236      16      3178      3162   53.5389        64     1.05553     1.19238
  237      16      3191      3175   53.5323        52    0.544769     1.19196
  238      16      3211      3195   53.6433        80    0.705249      1.1902
  239      16      3225      3209   53.6531        56     1.09667     1.18922
2019-05-18 16:12:53.604936 min lat: 0.204739 max lat: 16.5859 avg lat: 1.18838
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      3240      3224   53.6794        60    0.312957     1.18838
  241      16      3256      3240   53.7221        64    0.482895     1.18821
  242      16      3271      3255   53.7479        60    0.640144     1.18715
  243      16      3283      3267   53.7242        48    0.507039      1.1874
  244      16      3300      3284   53.7826        68     1.41019     1.18698
  245      16      3316      3300   53.8241        64    0.321793       1.186
  246      16      3334      3318   53.8978        72    0.957751     1.18526
  247      16      3348      3332   53.9062        56    0.733105     1.18416
  248      16      3365      3349   53.9628        68     0.45977     1.18333
  249      16      3373      3357   53.8745        32     1.35831     1.18334
  250      16      3392      3376   53.9628        76    0.980545      1.1833
  251      16      3410      3394   54.0345        72    0.557226     1.18218
  252      16      3417      3401   53.9311        28    0.452422     1.18133
  253      16      3441      3425   54.0972        96    0.929698     1.18145
  254      15      3452      3437   54.0731        48     1.38442     1.18009
  255      16      3467      3451   54.0805        56    0.436901     1.17982
  256      16      3488      3472   54.1971        84     1.93081      1.1791
  257      16      3502      3486    54.204        56    0.488884     1.17699
  258      16      3518      3502   54.2418        64    0.494519     1.17616
  259      16      3530      3514   54.2176        48     1.28206     1.17737
2019-05-18 16:13:13.615649 min lat: 0.204739 max lat: 16.5859 avg lat: 1.17626
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      3549      3533   54.3012        76     0.68118     1.17626
  261      16      3561      3545    54.277        48     1.36922     1.17597
  262      16      3578      3562   54.3293        68     1.13122     1.17618
  263      16      3589      3573   54.2899        44     1.15971     1.17496
  264      16      3601      3585   54.2661        48    0.333611     1.17498
  265      16      3616      3600   54.2876        60    0.711266     1.17525
  266      16      3625      3609   54.2188        36    0.252959     1.17601
  267      16      3644      3628   54.3001        76     0.80956     1.17609
  268      16      3660      3644   54.3362        64    0.485574     1.17565
  269      16      3668      3652   54.2531        32    0.358967     1.17428
  270      16      3687      3671   54.3335        76     1.86974     1.17627
  271      16      3691      3675   54.1921        16    0.433627     1.17573
  272      16      3704      3688    54.184        52     0.65242     1.17744
  273      16      3717      3701   54.1759        52    0.687427     1.17768
  274      16      3726      3710   54.1096        36     1.87406     1.17902
  275      16      3737      3721   54.0727        44    0.700663     1.17899
  276      16      3751      3735   54.0795        56    0.418646     1.17986
  277      16      3765      3749   54.0863        56     1.41936     1.18035
  278      16      3782      3766   54.1362        68    0.538819      1.1801
  279      16      3798      3782   54.1714        64    0.496792     1.17917
2019-05-18 16:13:33.625596 min lat: 0.204739 max lat: 16.5859 avg lat: 1.17953
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      3809      3793   54.1351        44     1.54345     1.17953
  281      16      3819      3803   54.0847        40      1.6929     1.18013
  282      16      3833      3817   54.0914        56     0.56953     1.17999
  283      16      3854      3838   54.1967        84    0.323926     1.17841
  284      16      3871      3855   54.2452        68     1.00298     1.17832
  285      16      3885      3869   54.2513        56     1.37687     1.17782
  286      16      3900      3884   54.2712        60      1.0587     1.17702
  287      16      3912      3896   54.2493        48     1.20156     1.17693
  288      16      3931      3915   54.3247        76     1.62476     1.17693
  289      16      3949      3933   54.3858        72    0.430894     1.17501
  290      16      3962      3946   54.3775        52     1.01582     1.17428
  291      16      3976      3960   54.3829        56     1.31065     1.17449
  292      16      3992      3976   54.4157        64    0.528849     1.17405
  293      16      4004      3988   54.3938        48    0.361629     1.17338
  294      16      4023      4007   54.4671        76    0.633131     1.17284
  295      16      4037      4021   54.4721        56     1.12174      1.1722
  296      16      4054      4038   54.5177        68     1.34113     1.17193
  297      16      4066      4050   54.4956        48    0.439699     1.17153
  298      16      4083      4067   54.5408        68    0.374661     1.17085
  299      16      4097      4081   54.5455        56       1.826     1.17071
2019-05-18 16:13:53.637452 min lat: 0.204739 max lat: 16.5859 avg lat: 1.17008
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      4110      4094   54.5369        52    0.772369     1.17008
  301      16      4124      4108   54.5415        56    0.335137     1.16989
  302      16      4139      4123   54.5593        60    0.287464     1.17074
  303      16      4151      4135   54.5376        48    0.491748     1.17015
  304      16      4166      4150   54.5554        60    0.450227     1.17106
  305      16      4174      4158   54.4815        32     1.52729     1.17149
  306      16      4182      4166    54.408        32    0.558202     1.17201
  307      16      4197      4181   54.4261        60    0.573123     1.17342
  308      16      4215      4199   54.4831        72    0.455909     1.17276
  309      16      4225      4209   54.4362        40    0.407753     1.17227
  310      16      4239      4223    54.441        56      1.9277     1.17261
  311      16      4252      4236    54.433        52    0.275288     1.17265
  312      16      4266      4250    54.438        56     1.66835     1.17265
  313      16      4286      4270   54.5195        80    0.742528     1.17161
  314      16      4303      4287   54.5622        68     1.20689     1.17167
  315      16      4315      4299   54.5394        48    0.430751     1.17066
  316      16      4331      4315   54.5692        64    0.575494     1.17055
  317      16      4341      4325   54.5232        40     1.93008     1.17083
  318      16      4356      4340   54.5403        60    0.642964     1.17099
  319      16      4370      4354   54.5448        56     1.47414     1.17033
2019-05-18 16:14:13.661444 min lat: 0.204739 max lat: 16.5859 avg lat: 1.17029
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16      4381      4365   54.5118        44    0.312343     1.17029
  321      16      4397      4381   54.5412        64    0.511051     1.17074
  322      16      4409      4393   54.5208        48      1.6233     1.17184
  323      16      4423      4407   54.5253        56     1.30348     1.17186
  324      16      4435      4419    54.505        48    0.567373      1.1717
  325      16      4446      4430   54.4726        44     1.37418      1.1719
  326      16      4463      4447   54.5139        68     0.51342     1.17228
  327      16      4468      4452   54.4083        20      2.1888     1.17229
  328      16      4481      4465   54.4009        52    0.399453      1.1736
  329      16      4497      4481     54.43        64     2.99338     1.17409
  330      16      4514      4498   54.4709        68      1.0119       1.173
  331      16      4532      4516   54.5236        72    0.609291     1.17212
  332      16      4547      4531     54.54        60     1.48135     1.17084
  333      16      4564      4548   54.5803        68    0.574407     1.17022
  334      16      4578      4562   54.5845        56     1.51042     1.17021
  335      16      4590      4574   54.5648        48     1.53674     1.17045
  336      16      4604      4588    54.569        56    0.507811     1.17048
  337      16      4621      4605   54.6027        68    0.529507     1.16953
  338      16      4633      4617    54.583        48    0.332042     1.16911
  339      16      4650      4634   54.6224        68    0.567079     1.16926
2019-05-18 16:14:33.712040 min lat: 0.204739 max lat: 16.5859 avg lat: 1.16964
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16      4659      4643   54.5676        36    0.314137     1.16964
  341      16      4675      4659   54.5951        64     1.90094     1.17062
  342      16      4690      4674   54.6108        60    0.699515     1.16977
  343      16      4705      4689   54.6264        60    0.376131     1.16905
  344      16      4722      4706   54.6652        68    0.410512     1.16831
  345      16      4744      4728   54.7616        88    0.735994     1.16676
  346      16      4763      4747   54.8229        76     1.18011      1.1661
  347      16      4776      4760   54.8147        52     1.03632     1.16539
  348      16      4796      4780   54.8868        80     1.11234     1.16441
  349      16      4812      4796   54.9127        64    0.423259     1.16305
  350      16      4823      4807    54.881        44     0.49099     1.16325
  351      16      4844      4828   54.9638        84    0.716123     1.16269
  352      16      4858      4842   54.9667        56    0.858849     1.16156
  353      16      4878      4862   55.0374        80    0.993311     1.16133
  354      16      4886      4870   54.9724        32     1.89012     1.16108
  355      16      4895      4879   54.9189        36     2.15176     1.16175
  356      16      4914      4898    54.978        76     1.37308     1.16274
  357      16      4929      4913    54.992        60    0.292571     1.16173
  358      16      4938      4922   54.9389        36    0.360288     1.16201
  359      16      4957      4941   54.9975        76     1.28495     1.16154
2019-05-18 16:14:53.726181 min lat: 0.204739 max lat: 16.5859 avg lat: 1.16102
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16      4969      4953    54.978        48    0.585598     1.16102
  361      16      4987      4971    55.025        72     0.63191      1.1617
  362      16      5001      4985   55.0277        56    0.636301      1.1613
  363      16      5018      5002   55.0633        68     1.20723     1.16064
  364      16      5035      5019   55.0987        68     0.86308     1.15978
  365      16      5050      5034   55.1121        60     1.25283     1.15972
  366      16      5067      5051   55.1472        68     1.04086     1.15898
  367      16      5076      5060    55.095        36     1.86551     1.15882
  368      16      5090      5074   55.0974        56     1.92613     1.15843
  369      16      5103      5087   55.0889        52    0.733858      1.1588
  370      16      5124      5108   55.1669        84    0.901831     1.15829
  371      16      5134      5118    55.126        40     1.63097     1.15765
  372      16      5142      5126   55.0638        32     2.49766     1.15865
  373      16      5157      5141    55.077        60     0.86992     1.15961
  374      16      5171      5155   55.0794        56     0.78585     1.15943
  375      16      5192      5176   55.1563        84    0.430681     1.15891
  376      16      5204      5188   55.1373        48     0.99319     1.15886
  377      16      5221      5205   55.1713        68    0.416587     1.15826
  378      16      5240      5224   55.2262        76    0.756114     1.15727
  379      16      5259      5243   55.2809        76     1.17882     1.15674
2019-05-18 16:15:13.735791 min lat: 0.204739 max lat: 16.5859 avg lat: 1.15574
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16      5273      5257   55.2827        56     1.30316     1.15574
  381      16      5280      5264    55.211        28    0.421515      1.1554
  382      16      5294      5278    55.213        56    0.495295      1.1564
  383      16      5309      5293   55.2254        60    0.310589     1.15657
  384      16      5329      5313   55.2898        80    0.843848     1.15622
  385      16      5345      5329   55.3123        64     1.11149     1.15561
  386      16      5362      5346    55.345        68    0.402285     1.15421
  387      16      5379      5363   55.3776        68    0.847615      1.1544
  388      16      5394      5378   55.3895        60    0.470617     1.15407
  389      16      5408      5392    55.391        56    0.421586     1.15373
  390      16      5420      5404    55.372        48    0.331792     1.15319
  391      16      5437      5421   55.4042        68    0.877509     1.15325
  392      16      5453      5437   55.4261        64     1.80872     1.15301
  393      16      5467      5451   55.4275        56    0.542076      1.1524
  394      16      5482      5466    55.439        60    0.732007     1.15192
  395      16      5499      5483   55.4707        68     1.41413     1.15188
  396      16      5509      5493   55.4316        40     1.51009     1.15191
  397      16      5529      5513   55.4934        80     1.44471     1.15151
  398      16      5537      5521   55.4344        32     1.56754     1.15178
  399      16      5556      5540   55.4858        76    0.743028     1.15181
2019-05-18 16:15:33.746068 min lat: 0.204739 max lat: 16.5859 avg lat: 1.15114
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16      5569      5553    55.477        52    0.468189     1.15114
  401      16      5587      5571    55.518        72    0.533789     1.15089
  402      16      5603      5587   55.5391        64     1.18352      1.1509
  403      16      5613      5597   55.5005        40     1.49849     1.15079
  404      16      5632      5616   55.5511        76     0.80678     1.15109
  405      16      5646      5630   55.5521        56    0.353304     1.15025
  406      16      5666      5650   55.6122        80    0.617229     1.14954
  407      16      5676      5660   55.5738        40     1.30539     1.14893
  408      16      5682      5666   55.4965        24     2.12495     1.15003
  409      16      5695      5679   55.4878        52    0.593519     1.15105
  410      16      5705      5689   55.4499        40    0.602261     1.15109
  411      16      5720      5704   55.4609        60     1.94311     1.15183
  412      16      5734      5718   55.4621        56    0.490515      1.1515
  413      16      5751      5735   55.4924        68    0.645214      1.1513
  414      16      5765      5749   55.4936        56    0.354147      1.1508
  415      16      5780      5764   55.5044        60     0.64927     1.15095
  416      16      5792      5776   55.4863        48    0.502865     1.15079
  417      16      5808      5792   55.5067        64    0.575211     1.15128
  418      16      5822      5806   55.5078        56    0.615406     1.15115
  419      16      5835      5819   55.4994        52    0.508867      1.1508
2019-05-18 16:15:53.756481 min lat: 0.204739 max lat: 16.5859 avg lat: 1.15129
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16      5854      5838   55.5481        76     1.44501     1.15129
  421      16      5863      5847   55.5016        36     1.15826     1.15113
  422      16      5879      5863   55.5217        64    0.314032     1.15043
  423      16      5894      5878   55.5321        60    0.703674     1.15077
  424      16      5908      5892   55.5331        56     1.53015     1.15052
  425      16      5923      5907   55.5435        60    0.280521     1.15031
  426      16      5946      5930    55.629        92    0.429985     1.14956
  427      16      5955      5939    55.583        36    0.384777      1.1492
  428      16      5968      5952   55.5746        52    0.721271     1.14907
  429      16      5985      5969   55.6035        68     1.74732     1.14977
  430      16      5998      5982    55.595        52     1.05115      1.1497
  431      16      6010      5994   55.5774        48     2.07382     1.14997
  432      16      6022      6006   55.5598        48     1.37511     1.14971
  433      16      6034      6018   55.5423        48     1.65097     1.14982
  434      16      6050      6034   55.5616        64     1.62414     1.15012
  435      16      6066      6050   55.5809        64     0.43617     1.15008
  436      16      6069      6053   55.4809        12     0.32026     1.14998
  437      16      6084      6068   55.4912        60     1.99363     1.15125
  438      16      6094      6078   55.4557        40    0.691585     1.15172
  439      16      6109      6093    55.466        60     1.14974     1.15185
2019-05-18 16:16:13.768496 min lat: 0.204739 max lat: 16.5859 avg lat: 1.15192
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16      6124      6108   55.4762        60    0.676361     1.15192
  441      16      6139      6123   55.4864        60    0.378641     1.15134
  442      16      6156      6140   55.5146        68    0.621909     1.15149
  443      16      6168      6152   55.4976        48    0.344402     1.15114
  444      16      6186      6170   55.5346        72    0.570477     1.15102
  445      16      6196      6180   55.4997        40    0.388845     1.15086
  446      16      6218      6202   55.5724        88     1.40662     1.15087
  447      16      6229      6213   55.5465        44    0.350942     1.15013
  448      16      6241      6225   55.5296        48     1.87033      1.1504
  449      16      6256      6240   55.5349        60    0.743087     1.15051
  450      16      6270      6254   55.5359        56     1.68079     1.15047
  451      16      6286      6270   55.5546        64    0.290122     1.14984
  452      16      6296      6280   55.5202        40     1.75126     1.15047
  453      16      6310      6294   55.5212        56    0.501221     1.15033
  454      16      6328      6312   55.5574        72    0.500216     1.15075
  455      16      6342      6326   55.5584        56    0.283727     1.15027
  456      16      6354      6338   55.5418        48     1.40202     1.15017
  457      16      6368      6352   55.5428        56    0.481239     1.15094
  458      16      6381      6365    55.535        52    0.355391     1.15059
  459      16      6396      6380   55.5447        60     1.24346     1.15015
2019-05-18 16:16:33.813952 min lat: 0.204739 max lat: 16.5859 avg lat: 1.15009
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16      6411      6395   55.5543        60    0.391753     1.15009
  461      16      6424      6408   55.5466        52     1.98274     1.15033
  462      16      6440      6424   55.5648        64    0.590823     1.15035
  463      16      6455      6439   55.5744        60    0.237137     1.15004
  464      16      6467      6451    55.558        48     0.65687     1.15009
  465      16      6487      6471   55.6105        80     0.75469     1.14977
  466      16      6501      6485   55.6113        56    0.995754     1.14932
  467      16      6515      6499   55.6121        56     1.33373     1.14926
  468      16      6523      6507   55.5617        32     1.97651     1.14912
  469      16      6537      6521   55.5626        56     1.98675     1.15049
  470      16      6553      6537   55.5805        64    0.293269     1.15005
  471      16      6568      6552   55.5899        60    0.599057     1.14996
  472      16      6581      6565   55.5822        52    0.398278     1.14939
  473      16      6603      6587   55.6507        88    0.450805     1.14941
  474      16      6620      6604   55.6767        68    0.460914     1.14847
  475      16      6629      6613   55.6353        36    0.485028     1.14832
  476      16      6638      6622    55.594        36     1.43775       1.149
  477      16      6652      6636   55.5948        56    0.553024     1.14981
  478      16      6660      6644   55.5453        32    0.319692     1.14934
  479      16      6679      6663   55.5874        76    0.531922     1.14985
2019-05-18 16:16:53.825232 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14924
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16      6691      6675   55.5716        48    0.512707     1.14924
  481      16      6708      6692   55.5973        68     1.51198     1.14926
  482      16      6716      6700   55.5483        32    0.526603     1.14942
  483      16      6731      6715   55.5573        60    0.843943     1.14994
  484      16      6753      6737   55.6242        88    0.793789     1.14939
  485      16      6768      6752   55.6331        60    0.312519     1.14866
  486      16      6787      6771   55.6749        76    0.963633     1.14875
  487      16      6800      6784   55.6673        52     1.03712     1.14818
  488      16      6812      6796   55.6516        48    0.380842      1.1478
  489      16      6825      6809   55.6441        52     1.41718     1.14846
  490      16      6838      6822   55.6365        52    0.614704      1.1482
  491      15      6855      6840   55.6697        72    0.306905     1.14796
  492      16      6870      6854   55.6704        56    0.671838      1.1481
  493      16      6884      6868    55.671        56    0.527099     1.14827
  494      16      6892      6876   55.6231        32    0.453514      1.1484
  495      16      6902      6886   55.5914        40     2.57987     1.14888
  496      16      6924      6908   55.6566        88    0.969073     1.14913
  497      16      6936      6920   55.6412        48     1.26388     1.14867
  498      16      6952      6936   55.6579        64     1.46033     1.14879
  499      16      6970      6954   55.6905        72    0.413916     1.14791
2019-05-18 16:17:13.838553 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14763
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16      6984      6968   55.6911        56    0.468279     1.14763
  501      16      6999      6983   55.6994        60      1.5795     1.14763
  502      16      7013      6997   55.6999        56     1.72081     1.14741
  503      16      7035      7019   55.7641        88    0.802843     1.14663
  504      16      7040      7024   55.6931        20      1.3077     1.14642
  505      16      7057      7041   55.7174        68    0.538387     1.14694
  506      16      7067      7051   55.6863        40    0.293349     1.14711
  507      16      7081      7065   55.6868        56    0.336351     1.14757
  508      16      7097      7081   55.7031        64    0.536487     1.14737
  509      16      7111      7095   55.7036        56    0.290292     1.14712
  510      16      7125      7109   55.7042        56     0.50036     1.14748
  511      16      7133      7117   55.6578        32      1.5987     1.14797
  512      16      7148      7132   55.6662        60    0.301433     1.14831
  513      16      7160      7144   55.6512        48    0.399621     1.14804
  514      16      7175      7159   55.6595        60     1.67203     1.14842
  515      16      7192      7176   55.6834        68    0.465412      1.1479
  516      16      7208      7192   55.6995        64    0.429258     1.14742
  517      16      7227      7211   55.7386        76    0.798461     1.14702
  518      16      7240      7224   55.7314        52     1.53195     1.14677
  519      16      7250      7234   55.7011        40     1.56924     1.14699
2019-05-18 16:17:33.851722 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14686
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16      7267      7251   55.7246        68    0.479032     1.14686
  521      16      7282      7266   55.7327        60    0.912604     1.14674
  522      16      7297      7281   55.7408        60    0.545733     1.14597
  523      16      7309      7293   55.7259        48     1.76437     1.14667
  524      16      7321      7305   55.7111        48     1.77764     1.14684
  525      16      7337      7321   55.7268        64     0.69031     1.14708
  526      16      7356      7340   55.7645        76    0.308001     1.14691
  527      16      7369      7353   55.7573        52       1.229     1.14676
  528      16      7381      7365   55.7426        48     1.11876     1.14662
  529      16      7396      7380   55.7506        60    0.611191      1.1464
  530      16      7414      7398   55.7811        72    0.250219     1.14617
  531      16      7431      7415    55.804        68     1.07885     1.14592
  532      16      7440      7424   55.7667        36    0.669192     1.14532
  533      16      7446      7430    55.707        24    0.426968     1.14489
  534      16      7461      7445    55.715        60    0.558104     1.14709
  535      16      7479      7463   55.7454        72     1.37258     1.14753
  536      16      7488      7472   55.7085        36     1.05389     1.14747
  537      16      7495      7479   55.6569        28    0.391677     1.14723
  538      16      7513      7497   55.6872        72    0.486923     1.14804
  539      16      7534      7518   55.7396        84     1.88252     1.14771
2019-05-18 16:17:53.872656 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14745
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16      7547      7531   55.7327        52    0.567006     1.14745
  541      16      7564      7548   55.7553        68     1.05759     1.14701
  542      16      7578      7562   55.7557        56       1.448      1.1469
  543      16      7588      7572   55.7266        40      1.3252     1.14678
  544      16      7595      7579   55.6756        28    0.361036     1.14646
  545      16      7616      7600   55.7275        84    0.712029     1.14786
  546      16      7624      7608    55.684        32     1.17169     1.14749
  547      16      7640      7624   55.6992        64    0.609843     1.14756
  548      16      7661      7645   55.7507        84    0.442766     1.14715
  549      16      7678      7662    55.773        68    0.990156     1.14643
  550      16      7693      7677   55.7806        60      1.3231     1.14613
  551      16      7708      7692   55.7882        60    0.676965     1.14567
  552      16      7730      7714   55.8464        88     1.43524     1.14535
  553      16      7747      7731   55.8683        68    0.548032     1.14468
  554      16      7765      7749   55.8974        72     1.29826     1.14421
  555      16      7769      7753   55.8255        16    0.650907     1.14414
  556      16      7785      7769   55.8402        64     1.69654     1.14462
  557      16      7794      7778   55.8045        36    0.353513     1.14455
  558      16      7815      7799   55.8549        84    0.525465     1.14492
  559      16      7828      7812   55.8479        52    0.884194      1.1444
2019-05-18 16:18:13.883823 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14433
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16      7847      7831   55.8838        76     1.28146     1.14433
  561      16      7854      7838   55.8341        28    0.306502     1.14395
  562      16      7867      7851   55.8273        52    0.414001     1.14449
  563      16      7883      7867   55.8417        64     1.38221     1.14474
  564      16      7897      7881   55.8419        56     1.41273     1.14456
  565      16      7918      7902   55.8917        84     1.40659     1.14443
  566      16      7932      7916   55.8918        56    0.461647     1.14401
  567      16      7948      7932   55.9061        64    0.578833     1.14369
  568      16      7961      7945   55.8991        52    0.835055     1.14356
  569      16      7972      7956   55.8782        44      1.6421     1.14382
  570      16      7982      7966   55.8503        40     1.71238      1.1443
  571      16      7994      7978   55.8365        48    0.544987     1.14462
  572      16      8007      7991   55.8298        52    0.371788     1.14482
  573      16      8020      8004    55.823        52    0.486054     1.14488
  574      16      8034      8018   55.8233        56      1.4323     1.14495
  575      16      8046      8030   55.8097        48     1.78167     1.14499
  576      16      8061      8045   55.8169        60    0.566864     1.14516
  577      16      8081      8065   55.8587        80     0.79225     1.14484
  578      16      8098      8082   55.8796        68    0.518239     1.14419
  579      16      8110      8094   55.8659        48    0.259853     1.14349
2019-05-18 16:18:33.894234 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14333
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16      8130      8114   55.9075        80    0.873793     1.14333
  581      16      8144      8128   55.9076        56     0.45542     1.14289
  582      16      8156      8140   55.8939        48     1.56962     1.14336
  583      16      8167      8151   55.8735        44     0.32456     1.14353
  584      16      8182      8166   55.8804        60         1.7     1.14398
  585      16      8200      8184   55.9078        72     1.14439     1.14363
  586      16      8212      8196   55.8942        48    0.505604     1.14297
  587      16      8231      8215   55.9284        76       1.103     1.14329
  588      16      8243      8227   55.9149        48    0.412295     1.14305
  589      16      8252      8236    55.881        36    0.444145     1.14288
  590      16      8264      8248   55.8676        48     2.05193     1.14416
  591      16      8272      8256   55.8272        32     1.83562     1.14378
  592      16      8284      8268    55.814        48     2.29641     1.14555
  593      16      8288      8272   55.7468        16     2.64731     1.14552
  594      16      8310      8294    55.801        88    0.948032     1.14599
  595      16      8324      8308   55.8013        56    0.974495     1.14556
  596      16      8347      8331    55.862        92     1.60445     1.14496
  597      16      8357      8341   55.8354        40    0.696737     1.14438
  598      16      8368      8352   55.8155        44     1.53799     1.14499
  599      16      8379      8363   55.7958        44     0.72372     1.14497
2019-05-18 16:18:53.907850 min lat: 0.204739 max lat: 16.5859 avg lat: 1.14518
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      14      8388      8374   55.7761        44     1.57904     1.14518
Total time run:         600.691810
Total writes made:      8388
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     55.8556
Stddev Bandwidth:       17.0047
Max bandwidth (MB/sec): 96
Min bandwidth (MB/sec): 0
Average IOPS:           13
Stddev IOPS:            4
Max IOPS:               24
Min IOPS:               0
Average Latency(s):     1.14563
Stddev Latency(s):      0.840153
Max latency(s):         16.5859
Min latency(s):         0.204739

Sequential Read Test

[root@odroid-hc2-00 ~]# rados bench -p testbench 600 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        34        18   71.9509        72    0.649461    0.465242
    2      16        61        45   89.9379       108     0.22313    0.547868
    3      16        81        65   86.6131        80    0.211665    0.446539
    4      16        82        66   65.9582         4    0.135089     0.44182
    5      16        82        66   52.7651         0           -     0.44182
    6      16        82        66   43.9723         0           -     0.44182
    7      16        82        66   37.6915         0           -     0.44182
    8      16        82        66   32.9806         0           -     0.44182
    9      16        82        66   29.3167         0           -     0.44182
   10      16        82        66   26.3851         0           -     0.44182
   11      16        82        66   23.9863         0           -     0.44182
   12      16        82        66   21.9879         0           -     0.44182
   13      16        82        66   20.2968         0           -     0.44182
   14      16        82        66   18.8469         0           -     0.44182
   15      16        82        66   17.5905         0           -     0.44182
   16      15        86        71   17.7352   1.66667     14.7545     1.24742
   17      16        96        80   18.8085        36    0.136803     1.49274
   18      16        97        81    17.986         4    0.147446     1.47613
   19      16        97        81   17.0392         0           -     1.47613
2019-05-18 16:26:46.043620 min lat: 0.0977557 max lat: 14.9532 avg lat: 1.47613
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16        97        81   16.1871         0           -     1.47613
   21      16        97        81   15.4162         0           -     1.47613
   22      16       106        90   16.3509         9    0.248036     2.12854
   23      16       130       114   19.8111        96     20.0951     2.83186
   24      16       157       141   23.4825       108    0.478491     2.65431
   25      16       182       166   26.5408       100     0.54867     2.34945
   26      15       208       193   29.6676       108     1.08236     2.11712
   27      15       235       220   32.5602       108    0.244662     1.92329
   28      16       263       247   35.2491       108     1.03459     1.77847
   29      16       288       272   37.4779       100    0.324194     1.66429
   30      16       313       297   39.5596       100    0.256307     1.58032
   31      16       338       322    41.507       100    0.409362     1.50381
   32      15       362       347   43.3273       100    0.223785     1.44598
   33      15       386       371   44.9175        96    0.128119     1.39374
   34      16       414       398   46.7704       108    0.140685     1.33956
   35      15       438       423   48.2876       100    0.293822     1.30218
   36      16       465       449    49.833       104    0.929303     1.26329
   37      16       492       476   51.4031       108     1.01234     1.22373
   38      16       519       503   52.8906       108     1.16127     1.18722
   39      16       547       531   54.4042       112      0.9601     1.15267
2019-05-18 16:27:06.069087 min lat: 0.089375 max lat: 20.4885 avg lat: 1.13349
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16       569       553   55.2428        88     1.31021     1.13349
   41      16       593       577   56.2355        96    0.271013     1.11265
   42      16       621       605   57.5616       112     1.35264     1.09212
   43      16       648       632   58.7332       108    0.146986      1.0714
   44      16       672       656    59.579        96     1.19001     1.05477
   45      16       699       683   60.6536       108    0.548563     1.03666
   46      16       724       708   61.5079       100    0.909568     1.02225
   47      16       751       735   62.4959       108    0.363091     1.00547
   48      15       777       762   63.4405       108    0.507468    0.990405
   49      16       802       786    64.104        96    0.407017     0.98194
   50      16       827       811    64.821       100     1.23883    0.969793
   51      16       856       840   65.8233       116    0.572234    0.957716
   52      16       882       866   66.5565       104    0.370699     0.94461
   53      16       907       891   67.1865       100    0.241297    0.936921
   54      16       932       916   67.7934       100    0.879143    0.928525
   55      16       959       943   68.5234       108    0.614999    0.919552
   56      16       984       968   69.0848       100    0.229312    0.911014
   57      16      1009       993   69.6263       100    0.241872    0.902711
   58      16      1036      1020   70.2871       108    0.436083    0.895007
   59      15      1062      1047    70.923       108    0.389651    0.888343
2019-05-18 16:27:26.077915 min lat: 0.089375 max lat: 20.4885 avg lat: 0.883441
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      1085      1069   71.2071        88    0.247771    0.883441
   61      16      1107      1091   71.4818        88    0.319624    0.880142
   62      16      1133      1117   72.0056       104    0.329838    0.873547
   63      15      1158      1143   72.5121       104     1.19158    0.866999
   64      16      1183      1167   72.8786        96      1.6969    0.862038
   65      16      1209      1193   73.3567       104    0.520301    0.857765
   66      15      1231      1216   73.6229        92    0.254234    0.854008
   67      16      1257      1241    74.016       100    0.747341    0.850056
   68      16      1283      1267   74.4562       104    0.651322    0.845709
   69      16      1307      1291   74.7679        96    0.528358    0.840703
   70      16      1333      1317   75.1847       104      1.0838    0.838281
   71      16      1357      1341   75.4743        96    0.277656    0.833992
   72      16      1382      1366   75.8143       100    0.554572    0.830625
   73      16      1404      1388   75.9808        88      1.0496    0.829183
   74      16      1429      1413   76.3048       100    0.905241    0.825251
   75      16      1454      1438   76.6202       100    0.728943    0.822016
   76      16      1481      1465   77.0325       108     1.16819    0.818149
   77      16      1504      1488   77.2262        92     1.41273    0.815016
   78      16      1526      1510   77.3639        88     1.45944     0.81292
   79      16      1554      1538   77.8017       112     1.53274    0.808983
2019-05-18 16:27:46.118494 min lat: 0.0883547 max lat: 20.4885 avg lat: 0.804803
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      15      1578      1563   78.0614       100   0.0931126    0.804803
   81      15      1606      1591   78.4773       112    0.232075    0.802335
   82      16      1632      1616   78.7392       100     1.22366    0.799794
   83      16      1659      1643    79.091       108    0.184041    0.796709
   84      16      1684      1668   79.3394       100    0.314756     0.79322
   85      16      1709      1693    79.582       100    0.246481    0.791329
   86      16      1733      1717   79.7724        96    0.384007    0.788049
   87      15      1757      1742   79.9874       100    0.232566     0.78663
   88      16      1785      1769   80.3051       108     1.30851    0.783935
   89      16      1810      1794   80.5201       100    0.259362    0.781293
   90      15      1837      1822   80.8676       112    0.199282     0.77932
   91      16      1862      1846   81.0334        96    0.341418    0.777467
   92      16      1887      1871   81.2391       100     1.31535    0.775779
   93      16      1912      1896   81.4402       100    0.208707    0.773902
   94      16      1937      1921   81.6372       100    0.590208     0.77177
   95      16      1964      1948   81.9141       108      0.2059    0.769184
   96      15      1987      1972   82.0584        96    0.245602    0.766938
   97      16      2011      1995   82.1605        92    0.119004    0.765537
   98      15      2037      2022   82.4177       108     1.06405     0.76481
   99      15      2063      2048   82.6207       104     1.14823    0.763587
2019-05-18 16:28:06.183096 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.760714
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      15      2088      2073   82.7913       100     1.38304    0.760714
  101      15      2115      2100   83.0389       108     1.16035    0.759352
  102      15      2140      2125   83.1963       100    0.230518     0.75629
  103      16      2169      2153   83.4753       112     1.71676    0.753751
  104      16      2190      2174     83.48        84    0.377225      0.7546
  105      16      2214      2198   83.5988        96     1.04079    0.753256
  106      16      2240      2224   83.7906       104    0.511616    0.752575
  107      16      2268      2252   84.0536       112     1.27808     0.75003
  108      16      2293      2277   84.2008       100    0.591175    0.748349
  109      16      2316      2300   84.2717        92    0.397413    0.747774
  110      16      2343      2327   84.4867       108    0.184993    0.744705
  111      16      2367      2351     84.59        96    0.304083    0.743798
  112      16      2396      2380   84.8679       116    0.303333     0.74207
  113      16      2423      2407   85.0721       108    0.248038    0.740364
  114      16      2447      2431   85.1677        96    0.525181    0.739478
  115      16      2473      2457    85.331       104    0.197812    0.737703
  116      16      2501      2485   85.5603       112    0.178938    0.736079
  117      15      2527      2512   85.7496       108    0.159411    0.734786
  118      15      2550      2535   85.7997        92    0.200366    0.733558
  119      16      2573      2557   85.8178        88    0.229232    0.733471
2019-05-18 16:28:26.210712 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.733638
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      2596      2580    85.869        92    0.636454    0.733638
  121      16      2621      2605   85.9854       100     1.28439    0.733321
  122      16      2649      2633   86.1981       112   0.0995623    0.730976
  123      15      2672      2657   86.2593        96    0.120085    0.729898
  124      16      2697      2681   86.3375        96     0.95693    0.730524
  125      16      2722      2706   86.4464       100     1.06586     0.72968
  126      16      2746      2730    86.522        96    0.297313    0.728528
  127      16      2773      2757   86.6907       108    0.189163    0.728046
  128      15      2797      2782   86.7925       100    0.222701    0.726587
  129      16      2822      2806   86.8635        96     1.05778    0.726072
  130      16      2845      2829   86.9027        92    0.237288    0.725759
  131      16      2871      2855   87.0328       104      0.2232    0.724304
  132      16      2894      2878   87.0703        92    0.276495    0.723182
  133      16      2919      2903   87.1671       100     1.70854    0.722995
  134      15      2946      2931   87.3509       112    0.240077    0.721212
  135      16      2974      2958   87.5034       108    0.959884    0.720882
  136      16      2998      2982   87.5657        96    0.702406    0.720009
  137      16      3025      3009   87.7144       108     1.10943    0.719172
  138      16      3047      3031   87.7164        88     1.21191    0.718943
  139      15      3070      3055   87.7754        96    0.470351    0.718459
2019-05-18 16:28:46.246905 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.717831
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      3097      3081    87.891       104    0.229593    0.717831
  141      15      3124      3109   88.0561       112     1.09959    0.716613
  142      16      3152      3136   88.1961       108    0.233637    0.714827
  143      15      3174      3159   88.2226        92    0.599847    0.713792
  144      15      3200      3185   88.3292       104    0.215048    0.713827
  145      15      3227      3212   88.4596       108    0.542062    0.712929
  146      16      3257      3241   88.6478       116    0.883503    0.711706
  147      16      3283      3267   88.7519       104    0.910553    0.710833
  148      16      3309      3293   88.8546       104    0.258162    0.710026
  149      16      3336      3320   88.9828       108     1.21196    0.708694
  150      16      3358      3342   88.9761        88     1.71717    0.708478
  151      16      3383      3367   89.0488       100    0.097676    0.708006
  152      16      3409      3393   89.1469       104    0.197949    0.707226
  153      16      3433      3417   89.1914        96    0.174553    0.706958
  154      16      3458      3442   89.2614       100    0.429104    0.706735
  155      16      3485      3469   89.3819       108    0.254237    0.705395
  156      16      3509      3493   89.4242        96    0.446979    0.704827
  157      16      3533      3517   89.4658        96     1.74394    0.704498
  158      16      3560      3544   89.5828       108    0.254862    0.703977
  159      16      3585      3569   89.6481       100    0.536047    0.703279
2019-05-18 16:29:06.272821 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.702655
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      3610      3594   89.7126       100    0.499299    0.702655
  161      16      3635      3619   89.7761       100    0.282486    0.702283
  162      15      3661      3646   89.8807       108      1.1474    0.701903
  163      15      3688      3673   89.9794       108    0.388947    0.700798
  164      16      3714      3698   90.0403       100    0.347593    0.700595
  165      16      3744      3728   90.2215       120    0.457697    0.699318
  166      16      3769      3753   90.2801       100     1.15885    0.698339
  167      16      3793      3777   90.3141        96    0.275888     0.69741
  168      16      3818      3802   90.3715       100     0.22709    0.697647
  169      16      3843      3827   90.4282       100    0.226985    0.696795
  170      15      3870      3855   90.5524       112    0.329313    0.696379
  171      16      3893      3877   90.5373        88     1.58285    0.696692
  172      15      3919      3904   90.6368       108    0.704328    0.695854
  173      16      3948      3932     90.76       112     1.11327    0.695163
  174      16      3974      3958   90.8358       104    0.475799    0.694865
  175      16      3997      3981   90.8423        92    0.305772    0.694271
  176      15      4021      4006   90.8885       100      1.3461    0.694088
  177      16      4049      4033   90.9849       108     0.30406     0.69337
  178      15      4075      4060   91.0773       108    0.332814    0.692678
  179      16      4101      4085   91.1269       100    0.424896    0.691768
2019-05-18 16:29:26.338383 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.691053
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      4125      4109   91.1538        96     0.29066    0.691053
  181      16      4150      4134   91.2024       100    0.306401    0.691114
  182      16      4176      4160   91.2724       104    0.417583    0.690919
  183      16      4199      4183   91.2763        92    0.360782    0.690085
  184      16      4223      4207   91.3018        96    0.295387    0.689989
  185      16      4249      4233   91.3702       104    0.318681     0.69005
  186      16      4274      4258   91.4164       100    0.740703     0.68992
  187      16      4296      4280    91.398        88    0.492648    0.689863
  188      16      4323      4307   91.4861       108    0.649533    0.689777
  189      16      4350      4334   91.5732       108     1.10171    0.688964
  190      16      4375      4359   91.6173       100      0.2567    0.688335
  191      16      4398      4382   91.6192        92     1.35683    0.688417
  192      16      4425      4409   91.7043       108    0.263076     0.68808
  193      15      4448      4433   91.7228        96    0.956792    0.687832
  194      16      4475      4459   91.7857       104    0.302509    0.687035
  195      15      4499      4484   91.8275       100    0.352269    0.686929
  196      16      4526      4510   91.8894       104    0.456594      0.6868
  197      16      4554      4538   91.9912       112     1.31374    0.685685
  198      16      4579      4563   92.0314       100    0.737832     0.68535
  199      16      4602      4586   92.0311        92     1.18962    0.684826
2019-05-18 16:29:46.351818 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.684069
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      4628      4612   92.0908       104     1.85666    0.684069
  201      16      4653      4637   92.1298       100    0.349701    0.683663
  202      16      4676      4660   92.1291        92    0.116027    0.684061
  203      16      4702      4686   92.1859       104    0.436322    0.683872
  204      16      4730      4714   92.2826       112     1.68873    0.683235
  205      16      4755      4739     92.32       100     0.40087     0.68341
  206      16      4780      4764    92.357       100     0.43221     0.68287
  207      16      4808      4792   92.4516       112    0.774458    0.682526
  208      16      4834      4818   92.5068       104    0.341888    0.681914
  209      16      4863      4847   92.6189       116    0.249431    0.681263
  210      16      4887      4871   92.6347        96    0.482918    0.680565
  211      15      4913      4898   92.6997       108    0.255771    0.680287
  212      15      4941      4926   92.7892       112    0.990813    0.680322
  213      16      4968      4952   92.8416       104    0.220118    0.679066
  214      16      4992      4976   92.8561        96     1.94265    0.678654
  215      16      5015      4999    92.852        92     1.84279      0.6788
  216      16      5041      5025   92.9033       104    0.366245    0.679139
  217      16      5069      5053    92.991       112    0.776837    0.678807
  218      16      5095      5079    93.041       104    0.223128    0.678052
  219      16      5117      5101   93.0179        88    0.379307    0.678041
2019-05-18 16:30:06.383558 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.6779
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      5146      5130   93.1221       116     1.07424      0.6779
  221      16      5170      5154   93.1349        96    0.347358    0.677567
  222      16      5198      5182   93.2197       112     1.37466    0.676895
  223      16      5221      5205   93.2141        92    0.435221    0.676695
  224      15      5249      5234   93.3135       116    0.420222     0.67648
  225      15      5276      5261   93.3693       108    0.408896    0.675628
  226      16      5303      5287   93.4162       104     0.80939    0.675319
  227      15      5327      5312   93.4428       100     1.66228     0.67529
  228      16      5354      5338    93.489       104    0.588061    0.675052
  229      16      5380      5364   93.5347       104    0.413989    0.674534
  230      16      5405      5389   93.5626       100    0.784262    0.674463
  231      16      5429      5413    93.573        96    0.446694    0.674204
  232      16      5455      5439   93.6177       104     1.06563    0.673836
  233      16      5479      5463   93.6279        96    0.151696    0.673185
  234      16      5506      5490    93.689       108    0.460371    0.673363
  235      16      5533      5517   93.7497       108    0.621171    0.673046
  236      15      5557      5542   93.7744       100    0.854384    0.672891
  237      15      5584      5569   93.8337       108     1.19965    0.672703
  238      16      5611      5595   93.8762       104    0.910699    0.672185
  239      16      5636      5620   93.9015       100    0.296045    0.671709
2019-05-18 16:30:26.430199 min lat: 0.0762972 max lat: 20.4885 avg lat: 0.671832
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      15      5658      5643   93.8925        92     1.76473    0.671832
  241      15      5682      5667    93.901        96    0.630003    0.672041
  242      16      5712      5696   93.9909       116    0.275767    0.671112
  243      16      5736      5720   93.9987        96     1.29537    0.670659
  244      15      5757      5742   93.9705        88    0.367462    0.670431
  245      16      5784      5768   94.0112       104    0.199453    0.670598
  246      16      5808      5792    94.019        96    0.380391    0.670767
  247      16      5834      5818   94.0591       104    0.274492    0.670727
  248      16      5859      5843   94.0827       100    0.288092    0.670483
  249      15      5886      5871   94.1537       112    0.902737      0.6704
  250      16      5915      5899   94.2247       112      0.1058    0.669707
  251      16      5942      5926   94.2794       108      1.2498    0.668788
  252      16      5967      5951   94.3019       100   0.0702014      0.6683
  253      16      5993      5977   94.3367       104      0.2925     0.66828
  254      15      6019      6004   94.3899       108    0.154874    0.668287
  255      16      6048      6032   94.4587       112    0.389957    0.668076
  256      16      6073      6057   94.4801       100    0.613371    0.667623
  257      16      6098      6082   94.5015       100    0.283191    0.667479
  258      16      6124      6108   94.5381       104    0.205475    0.666867
  259      16      6149      6133    94.559       100    0.439629    0.666714
2019-05-18 16:30:46.464212 min lat: 0.0702014 max lat: 20.4885 avg lat: 0.666356
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      6176      6160   94.6104       108    0.147709    0.666356
  261      15      6201      6186   94.6378       104    0.444546    0.666173
  262      16      6229      6213   94.6885       108    0.474866    0.665829
  263      16      6255      6239   94.7237       104     1.07845    0.665858
  264      16      6282      6266   94.7737       108    0.153152    0.665209
  265      16      6308      6292   94.8083       104    0.246811    0.664923
  266      16      6330      6314   94.7827        88    0.301539    0.664682
  267      16      6357      6341   94.8319       108    0.168142    0.664857
  268      16      6380      6364   94.8212        92       1.232    0.665078
  269      16      6404      6388   94.8253        96    0.331193    0.664991
  270      16      6429      6413   94.8444       100    0.791554    0.664882
  271      16      6458      6442   94.9222       116    0.421307    0.664248
  272      16      6486      6470   94.9847       112    0.158938    0.663664
  273      16      6508      6492   94.9591        88    0.705513    0.663375
  274      16      6530      6514   94.9335        88    0.567068    0.664039
  275      16      6557      6541   94.9807       108     1.49898    0.664032
  276      16      6583      6567   95.0132       104     0.34375    0.663265
  277      16      6607      6591   95.0166        96    0.370137    0.663109
  278      16      6635      6619   95.0775       112     1.62973    0.663323
  279      16      6661      6645   95.1093       104     1.09981    0.663127
2019-05-18 16:31:06.500490 min lat: 0.0702014 max lat: 20.4885 avg lat: 0.662423
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      15      6687      6672   95.1536       108    0.181715    0.662423
  281      16      6713      6697   95.1707       100    0.241655    0.662289
  282      16      6740      6724   95.2159       108    0.502332    0.662137
  283      16      6767      6751   95.2609       108    0.523246    0.662163
  284      16      6793      6777   95.2915       104    0.222285    0.661771
  285      16      6817      6801   95.2939        96     0.97777    0.661658
  286      16      6842      6826   95.3101       100     0.46303    0.661258
  287      16      6867      6851   95.3251       100     2.06722     0.66074
  288      16      6893      6877   95.3536       104    0.357226    0.661123
  289      16      6920      6904   95.3972       108    0.231899    0.660786
  290      16      6947      6931   95.4405       108    0.375587    0.660537
  291      16      6973      6957   95.4693       104    0.295895    0.660247
  292      15      6996      6981   95.4656        96    0.562043    0.660555
  293      16      7024      7008   95.5082       108     0.71041    0.660282
  294      16      7055      7039   95.6028       124    0.276101     0.65944
  295      15      7080      7065   95.6306       104    0.185409     0.65949
  296      16      7105      7089   95.6317        96    0.146311     0.65949
  297      16      7132      7116   95.6732       108    0.795265    0.659381
  298      16      7158      7142    95.701       104    0.117077    0.658996
  299      16      7182      7166   95.7018        96    0.231621     0.65881
2019-05-18 16:31:26.542031 min lat: 0.0702014 max lat: 20.4885 avg lat: 0.659082
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      7204      7188    95.676        88    0.734178    0.659082
  301      16      7229      7213   95.6902       100    0.655116    0.658974
  302      15      7255      7240   95.7303       108    0.848148    0.658957
  303      16      7282      7266   95.7575       104    0.160946    0.658426
  304      15      7310      7295   95.8202       116    0.282947    0.658226
  305      15      7335      7320   95.8329       100     1.26983     0.65798
  306      16      7363      7347   95.8725       108    0.186013    0.657633
  307      16      7389      7373   95.8989       104    0.358382    0.657644
  308      16      7417      7401   95.9509       112    0.734936    0.657326
  309      16      7439      7423   95.9248        88     1.41027    0.657385
  310      15      7462      7447   95.9236        96    0.256498    0.657517
  311      16      7487      7471   95.9237        96    0.223205    0.657493
  312      16      7516      7500   95.9877       116    0.848565     0.65737
  313      16      7543      7527   96.0257       108    0.263861    0.656776
  314      16      7565      7549   96.0001        88    0.700857     0.65683
  315      16      7587      7571   95.9746        88    0.282148     0.65701
  316      15      7615      7600   96.0375       116    0.225031    0.656858
  317      16      7641      7625   96.0498       100   0.0870665    0.656489
  318      16      7665      7649   96.0495        96    0.350301     0.65656
  319      16      7692      7676   96.0867       108    0.946106    0.656574
2019-05-18 16:31:46.573000 min lat: 0.0702014 max lat: 20.4885 avg lat: 0.656309
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16      7717      7701   96.0987       100    0.189171    0.656309
  321      16      7743      7727   96.1228       104     0.20635    0.656048
  322      16      7766      7750   96.1095        92    0.107305    0.656156
  323      16      7790      7774   96.1091        96    0.841181    0.656212
  324      16      7814      7798   96.1086        96     1.61225    0.655964
  325      15      7839      7824   96.1312       104    0.524573    0.655988
  326      16      7867      7851   96.1673       108    0.195431    0.655979
  327      16      7893      7877   96.1911       104    0.140344    0.655711
  328      16      7916      7900   96.1782        92    0.866095    0.655767
  329      16      7942      7926   96.2018       104     0.55399    0.655742
  330      15      7966      7951    96.213       100     1.10584    0.655765
  331      16      7993      7977   96.2364       104    0.271027     0.65548
  332      16      8018      8002   96.2477       100    0.185103    0.655168
  333      16      8044      8028   96.2707       104    0.326841    0.654999
  334      16      8068      8052   96.2697        96    0.229361    0.655127
  335      16      8091      8075   96.2568        92    0.276868    0.654857
  336      15      8116      8101   96.2778       104    0.546563    0.654517
  337      15      8140      8125   96.2701        96     0.60756    0.654944
  338      16      8164      8148   96.2574        92     0.79943    0.655273
  339      16      8189      8173   96.2683       100    0.213498    0.655065
2019-05-18 16:32:06.620439 min lat: 0.0702014 max lat: 20.4885 avg lat: 0.655098
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16      8213      8197   96.2675        96    0.486064    0.655098
  341      15      8237      8222   96.2777       100    0.368972    0.654855
  342      16      8262      8246   96.2768        96     1.26142    0.654944
  343      16      8287      8271   96.2875       100     1.16228    0.655047
  344      15      8310      8295   96.2849        96    0.276478    0.654849
  345      16      8338      8322   96.3187       108    0.337555    0.654757
  346      16      8358      8342   96.2715        80    0.124775    0.654621
  347      16      8383      8367   96.2817       100    0.387493    0.655212
Total time run:       348.446419
Total reads made:     8388
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   96.2903
Average IOPS:         24
Stddev IOPS:          5
Max IOPS:             31
Min IOPS:             0
Average Latency(s):   0.655719
Max latency(s):       20.4885
Min latency(s):       0.0702014

Random Read Test

[root@odroid-hc2-00 ~]# rados bench -p testbench 600 rand
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      15        38        23   91.9614        92    0.460823    0.459978
    2      16        62        46   91.9588        92    0.102533    0.530199
    3      16        89        73   97.2974       108    0.563032    0.559597
    4      16       114        98   97.9662       100    0.540406    0.571641
    5      16       138       122   97.5686        96    0.143201    0.582627
    6      15       163       148   98.6343       104     1.15127    0.594922
    7      16       189       173    98.826       100     1.11067    0.604032
    8      16       213       197     98.47        96    0.367997    0.606495
    9      15       237       222   98.6355       100    0.304864    0.607933
   10      16       261       245   97.9697        92     1.28205    0.613282
   11      15       289       274   99.6063       116    0.245911    0.603371
   12      16       317       301   100.302       108     1.09828    0.609613
   13      15       341       326   100.276       100    0.336652     0.60194
   14      16       367       351   100.255       100     0.33114    0.605137
   15      15       395       380   101.302       116    0.028177    0.596902
   16      16       421       405   101.219       100    0.725493    0.603266
   17      16       448       432   101.616       108     1.26503    0.605315
   18      16       474       458   101.748       104   0.0796884    0.602191
   19      16       498       482   101.444        96    0.324958    0.601594
2019-05-18 16:33:22.102095 min lat: 0.0178536 max lat: 1.7715 avg lat: 0.604092
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       525       509    101.77       108     1.62598    0.604092
   21      16       551       535   101.875       104     1.12436    0.608823
   22      16       577       561    101.97       104    0.332545    0.607396
   23      16       603       587   102.058       104     1.07456    0.607911
   24      16       626       610   101.637        92    0.523947    0.606173
   25      16       650       634   101.411        96     1.35517    0.611286
   26      16       675       659   101.355       100     1.25767    0.611671
   27      16       700       684   101.304       100     1.21896    0.610098
   28      16       725       709   101.257       100    0.278581    0.609667
   29      15       753       738   101.762       116    0.152244     0.61278
   30      15       780       765   101.969       108     1.20678    0.610946
   31      16       805       789   101.776        96    0.238761    0.611217
   32      16       832       816    101.97       108     1.43774    0.608713
   33      16       855       839   101.667        92    0.275256    0.609295
   34      15       879       864   101.617       100    0.506553    0.610091
   35      16       903       887   101.341        92     0.42799    0.614838
   36      15       929       914   101.525       108    0.246686    0.614385
   37      15       954       939   101.483       100    0.660536    0.614812
   38      16       978       962   101.233        92     1.19454    0.615563
   39      15      1004       989   101.406       108    0.913785    0.616659
2019-05-18 16:33:42.108159 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.615418
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1034      1018    101.77       116    0.642701    0.615418
   41      16      1059      1043   101.726       100    0.907593    0.613914
   42      16      1084      1068   101.684       100    0.374597    0.614907
   43      16      1110      1094   101.737       104     0.96958    0.615492
   44      16      1136      1120   101.788       104    0.840982    0.615351
   45      16      1161      1145   101.748       100    0.735584    0.615418
   46      16      1189      1173    101.97       112    0.720255    0.614002
   47      16      1213      1197   101.842        96     1.27825    0.614561
   48      16      1242      1226   102.137       116     0.61307    0.614118
   49      16      1267      1251   102.093       100    0.225578    0.611559
   50      16      1293      1277    102.13       104    0.947885    0.613134
   51      16      1317      1301   102.009        96     0.22036    0.612555
   52      15      1344      1329   102.201       112    0.233945    0.613195
   53      15      1369      1354   102.159       100    0.292725    0.613236
   54      16      1396      1380   102.193       104    0.244423    0.612626
   55      15      1420      1405   102.152       100     1.50651     0.61284
   56      16      1443      1427   101.899        88    0.360348    0.614463
   57      16      1469      1453   101.936       104    0.261534    0.615286
   58      16      1496      1480    102.04       108    0.364362    0.613428
   59      16      1522      1506   102.073       104    0.493277    0.613733
2019-05-18 16:34:02.113333 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.614096
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      1546      1530   101.971        96    0.330609    0.614096
   61      16      1572      1556   102.004       104     1.52149    0.615369
   62      16      1595      1579   101.842        92    0.318493    0.615996
   63      15      1621      1606   101.939       108    0.977447    0.615806
   64      16      1648      1632   101.971       104    0.086239    0.616227
   65      16      1674      1658   102.002       104    0.663038    0.615419
   66      16      1699      1683   101.971       100    0.365909    0.614713
   67      16      1727      1711    102.12       112    0.308491    0.614882
   68      16      1752      1736   102.075       100     1.05294    0.615594
   69      16      1779      1763   102.161       108    0.217732    0.614623
   70      16      1806      1790   102.244       108     1.60054    0.614634
   71      16      1833      1817   102.325       108    0.232145    0.614549
   72      16      1859      1843   102.346       104    0.226481    0.613269
   73      16      1887      1871   102.477       112    0.309953    0.612531
   74      16      1915      1899   102.605       112    0.294306    0.612399
   75      16      1941      1925   102.624       104    0.239048    0.611767
   76      15      1964      1949   102.536        96    0.361543    0.612859
   77      15      1987      1972   102.398        92    0.600103    0.613286
   78      16      2015      1999    102.47       108    0.105284    0.613295
   79      15      2043      2028    102.64       116     1.30296    0.612416
2019-05-18 16:34:22.129791 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.612544
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      15      2066      2051   102.507        92    0.378624    0.612544
   81      16      2090      2074   102.376        92    0.304344    0.613135
   82      15      2114      2099   102.347       100     1.35539     0.61378
   83      15      2141      2126   102.415       108    0.727947    0.612929
   84      16      2168      2152   102.433       104     1.33487    0.612479
   85      16      2193      2177   102.404       100     1.68349    0.612177
   86      16      2219      2203   102.423       104    0.614819     0.61321
   87      16      2245      2229   102.441       104     1.02461    0.614046
   88      16      2270      2254   102.412       100    0.728825    0.614221
   89      16      2297      2281   102.475       108    0.500017    0.613279
   90      16      2321      2305   102.402        96    0.542202    0.613601
   91      16      2344      2328   102.287        92    0.428398    0.614248
   92      16      2368      2352   102.219        96    0.261816    0.614744
   93      16      2393      2377   102.195       100     1.60576    0.615061
   94      16      2417      2401   102.129        96    0.504581    0.615406
   95      16      2442      2426   102.106       100    0.238524      0.6155
   96      15      2468      2453   102.167       108    0.345518    0.615092
   97      16      2494      2478   102.144       100    0.639688    0.614828
   98      16      2515      2499   101.959        84    0.339075    0.616186
   99      16      2539      2523   101.898        96     1.35541     0.61736
2019-05-18 16:34:42.136600 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.617872
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      2564      2548   101.879       100     1.18894    0.617872
  101      16      2588      2572   101.818        96     1.41383    0.617181
  102      15      2609      2594   101.682        88     1.41548    0.618115
  103      16      2637      2621   101.743       108     1.59457    0.618172
  104      15      2660      2645   101.687        96     1.29024    0.618203
  105      15      2683      2668   101.595        92    0.271359    0.618088
  106      16      2709      2693   101.579       100    0.271561    0.619286
  107      16      2736      2720   101.639       108     1.21346    0.619218
  108      16      2761      2745   101.623       100     0.23701    0.619419
  109      15      2782      2767   101.498        88     1.11943    0.620276
  110      16      2803      2787   101.302        80    0.127282    0.620966
  111      16      2828      2812    101.29       100     1.19177    0.621593
  112      15      2854      2839   101.349       108    0.500804    0.621117
  113      15      2883      2868   101.479       116    0.979941    0.621037
  114      15      2908      2893   101.465       100    0.595129    0.620529
  115      16      2934      2918   101.452       100    0.631809    0.620587
  116      16      2958      2942   101.405        96     1.30586    0.620826
  117      16      2983      2967   101.392       100     1.43655      0.6207
  118      16      3012      2996   101.516       116     0.28437    0.620198
  119      15      3037      3022   101.536       104    0.396062    0.620285
2019-05-18 16:35:02.147676 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.620297
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      15      3059      3044   101.423        88     1.20749    0.620297
  121      16      3085      3069   101.411       100    0.289698    0.619866
  122      16      3109      3093   101.366        96     1.47013     0.62052
  123      15      3137      3122   101.484       116    0.539145    0.620032
  124      16      3163      3147   101.472       100    0.300984    0.620212
  125      16      3187      3171   101.428        96    0.245069    0.620262
  126      16      3212      3196   101.416       100    0.382789    0.619751
  127      15      3237      3222   101.436       104    0.624405    0.620601
  128      15      3265      3250   101.518       112    0.150987    0.620461
  129      16      3288      3272   101.413        88    0.328893    0.620595
  130      15      3313      3298   101.433       104    0.524091    0.621403
  131      16      3339      3323   101.421       100    0.289123    0.620895
  132      16      3368      3352   101.531       116    0.395289     0.62023
  133      16      3392      3376   101.489        96     1.36399     0.61999
  134      16      3419      3403   101.538       108    0.107805    0.619212
  135      16      3444      3428   101.526       100    0.380014    0.619493
  136      15      3471      3456   101.603       112    0.553851    0.619538
  137      16      3497      3481   101.591       100    0.413806    0.619751
  138      16      3519      3503   101.492        88     1.57274    0.619977
  139      16      3545      3529    101.51       104    0.812276    0.621023
2019-05-18 16:35:22.157837 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.620722
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      3571      3555   101.527       104    0.701561    0.620722
  141      16      3595      3579   101.487        96    0.641256    0.620624
  142      16      3619      3603   101.448        96    0.406191     0.62071
  143      16      3644      3628   101.438       100    0.847267     0.62114
  144      15      3668      3653   101.427       100     1.02263     0.62112
  145      16      3695      3679   101.445       104    0.690649    0.621159
  146      16      3718      3702    101.38        92     1.16805    0.620918
  147      16      3744      3728   101.397       104    0.869129     0.62144
  148      16      3769      3753   101.388       100    0.553545    0.621529
  149      16      3792      3776   101.324        92    0.461667     0.62138
  150      16      3818      3802   101.342       104    0.291619    0.621279
  151      16      3844      3828   101.359       104    0.973913    0.621405
  152      16      3869      3853    101.35       100    0.369249    0.621225
  153      16      3895      3879   101.367       104     1.42668    0.621382
  154      16      3919      3903   101.332        96    0.238997    0.621625
  155      16      3946      3930   101.375       108     1.34762    0.621129
  156      16      3969      3953   101.315        92     1.17845    0.621669
  157      16      3995      3979   101.332       104    0.473975    0.621859
  158      15      4020      4005   101.348       104    0.584256    0.621786
  159      15      4043      4028   101.289        92    0.267278    0.621693
2019-05-18 16:35:42.166110 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.621476
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      4071      4055   101.331       108    0.194401    0.621476
  161      16      4096      4080   101.322       100    0.315621     0.62117
  162      16      4123      4107   101.363       108    0.322259    0.621174
  163      16      4148      4132   101.354       100     0.48933    0.621475
  164      15      4168      4153   101.248        84    0.285016    0.621883
  165      16      4194      4178   101.241       100     1.00267    0.622445
  166      16      4222      4206   101.305       112    0.302076    0.622458
  167      16      4247      4231   101.297       100    0.607249    0.622223
  168      16      4271      4255   101.265        96     1.10633    0.622223
  169      15      4295      4280   101.257       100     1.40437    0.622956
  170      16      4320      4304   101.226        96     1.29874    0.622788
  171      16      4348      4332   101.289       112    0.891543    0.622758
  172      16      4373      4357   101.281       100     1.01804    0.622526
  173      16      4397      4381    101.25        96    0.314154    0.622338
  174      16      4425      4409   101.312       112    0.280333    0.622013
  175      16      4453      4437   101.372       112    0.507101    0.621668
  176      16      4478      4462   101.364       100    0.369535    0.621579
  177      16      4500      4484   101.288        88    0.286021    0.621995
  178      16      4524      4508   101.258        96   0.0986942     0.62182
  179      16      4550      4534   101.274       104    0.291024    0.622242
2019-05-18 16:36:02.175971 min lat: 0.0178536 max lat: 2.24639 avg lat: 0.622272
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      15      4572      4557   101.222        92    0.134211    0.622272
  181      15      4599      4584   101.259       108     1.43393     0.62254
  182      16      4625      4609   101.252       100    0.283741    0.622358
  183      16      4650      4634   101.245       100     1.54544    0.622435
  184      15      4675      4660   101.259       104    0.855797    0.622905
  185      15      4699      4684    101.23        96    0.996938    0.623006
  186      16      4723      4707    101.18        92    0.508726    0.622852
  187      16      4748      4732   101.174       100     1.15762    0.623139
  188      16      4773      4757   101.167       100    0.300548    0.622578
  189      16      4800      4784   101.203       108      1.6523    0.622247
  190      16      4827      4811   101.238       108    0.335725    0.622353
  191      16      4853      4837   101.252       104    0.297378     0.62186
  192      16      4879      4863   101.266       104    0.378816    0.622332
  193      16      4905      4889    101.28       104    0.630036     0.62254
  194      16      4929      4913   101.252        96     1.31716     0.62255
  195      16      4953      4937   101.225        96     1.44665    0.622514
  196      16      4979      4963   101.239       104     1.29166     0.62281
  197      16      5002      4986   101.192        92    0.649523    0.622641
  198      16      5025      5009   101.145        92     1.40315    0.622802
  199      16      5051      5035   101.159       104     1.85447    0.623106
2019-05-18 16:36:22.189333 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.623351
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      5077      5061   101.173       104    0.228415    0.623351
  201      16      5102      5086   101.167       100      1.1258    0.623219
  202      16      5128      5112   101.181       104    0.144679    0.623165
  203      16      5152      5136   101.155        96     0.45859    0.623267
  204      15      5176      5161   101.149       100    0.246263    0.623226
  205      16      5200      5184   101.104        92     0.15298    0.622968
  206      16      5226      5210   101.118       104    0.314363    0.623264
  207      16      5255      5239    101.19       116    0.296073    0.622894
  208      16      5277      5261   101.126        88    0.123729    0.623258
  209      16      5303      5287    101.14       104    0.243795    0.623575
  210      16      5326      5310   101.096        92    0.683485    0.623494
  211      16      5350      5334   101.071        96    0.250412    0.623411
  212      16      5377      5361   101.104       108     1.42768    0.623551
  213      16      5401      5385    101.08        96    0.471156    0.623626
  214      16      5429      5413    101.13       112     1.12058    0.623706
  215      16      5457      5441   101.181       112    0.149699    0.623356
  216      15      5482      5467   101.194       104    0.382906    0.623127
  217      16      5506      5490   101.151        92     1.43476      0.6236
  218      16      5531      5515   101.145       100    0.831024    0.623643
  219      15      5557      5542   101.173       108     1.26516     0.62353
2019-05-18 16:36:42.207277 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.623377
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      15      5581      5566   101.149        96     1.41688    0.623377
  221      16      5607      5591   101.143       100     1.22669    0.623508
  222      16      5635      5619   101.192       112    0.702633    0.623437
  223      15      5660      5645   101.204       104    0.658568    0.623323
  224      16      5685      5669   101.181        96    0.450759    0.623488
  225      16      5712      5696   101.211       108    0.695003    0.623314
  226      16      5742      5726   101.294       120    0.289723      0.6227
  227      16      5766      5750    101.27        96    0.739123    0.622798
  228      15      5790      5775   101.264       100    0.321139    0.622755
  229      15      5816      5801   101.276       104    0.998477    0.622826
  230      16      5844      5828   101.305       108    0.353511    0.622642
  231      16      5868      5852   101.282        96    0.795351     0.62282
  232      16      5893      5877   101.276       100    0.849987    0.622851
  233      16      5919      5903   101.288       104    0.208041    0.622614
  234      15      5944      5929   101.299       104    0.222121    0.622392
  235      15      5971      5956   101.327       108     1.20031     0.62237
  236      15      5998      5983   101.356       108    0.129616    0.621988
  237      16      6024      6008    101.35       100    0.360037    0.622042
  238      16      6049      6033   101.344       100    0.266008    0.622015
  239      15      6076      6061   101.388       112    0.616195    0.622186
2019-05-18 16:37:02.217008 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.621925
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      6102      6086   101.382       100    0.292008    0.621925
  241      16      6127      6111   101.376       100    0.477008    0.621951
  242      16      6153      6137   101.387       104     1.52713    0.621981
  243      16      6176      6160   101.348        92     1.05464    0.622165
  244      15      6199      6184   101.326        96    0.910721    0.622189
  245      15      6225      6210   101.337       104     1.07306    0.622479
  246      15      6253      6238    101.38       112    0.828068      0.6224
  247      16      6277      6261   101.342        92    0.420895    0.622246
  248      16      6303      6287   101.352       104    0.214216    0.622144
  249      16      6331      6315   101.395       112    0.638923     0.62214
  250      16      6356      6340   101.389       100    0.238125    0.621844
  251      16      6380      6364   101.367        96     1.45499    0.622043
  252      16      6406      6390   101.377       104    0.417803    0.622315
  253      15      6431      6416   101.388       104    0.104336    0.622099
  254      16      6456      6440   101.366        96    0.278432    0.622136
  255      16      6480      6464   101.345        96    0.243936    0.622064
  256      15      6506      6491   101.371       108    0.214287    0.621874
  257      16      6530      6514   101.334        92    0.267618    0.621983
  258      16      6555      6539   101.329       100    0.454341    0.622434
  259      15      6579      6564   101.323       100    0.313691    0.622321
2019-05-18 16:37:22.227463 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.621957
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      15      6606      6591   101.349       108   0.0984729    0.621957
  261      15      6630      6615   101.328        96    0.186963    0.622095
  262      16      6657      6641   101.338       104    0.309167    0.622338
  263      16      6684      6668   101.363       108    0.470031    0.622299
  264      16      6708      6692   101.343        96    0.357501     0.62221
  265      16      6728      6712   101.262        80    0.874509    0.622712
  266      15      6751      6736   101.242        96    0.636873    0.623195
  267      15      6776      6761   101.237       100    0.811325    0.623353
  268      16      6799      6783   101.188        88    0.299508    0.623158
  269      15      6825      6810   101.213       108    0.423659    0.623193
  270      16      6850      6834   101.193        96    0.320671    0.623018
  271      16      6877      6861   101.218       108    0.461819    0.622948
  272      15      6905      6890   101.272       116      1.1124    0.622878
  273      15      6931      6916   101.282       104    0.653229    0.622811
  274      15      6957      6942   101.292       104     0.33856    0.622757
  275      16      6983      6967   101.287       100    0.276897    0.622739
  276      16      7009      6993   101.297       104    0.917901    0.622597
  277      16      7034      7018   101.292       100     1.29641    0.622527
  278      16      7060      7044   101.302       104    0.675619    0.622507
  279      16      7083      7067   101.268        92     0.48588    0.622615
2019-05-18 16:37:42.236221 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.622716
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      7108      7092   101.264       100    0.250136    0.622716
  281      16      7133      7117   101.259       100    0.837257    0.623053
  282      16      7158      7142   101.254       100     0.62718    0.622826
  283      15      7185      7170   101.292       112    0.141372    0.622738
  284      16      7208      7192   101.245        88    0.217206    0.622943
  285      16      7233      7217   101.241       100    0.207664    0.622944
  286      15      7257      7242   101.236       100     0.33304    0.622862
  287      16      7282      7266   101.217        96    0.629921    0.622786
  288      15      7305      7290   101.199        96    0.345602    0.622856
  289      15      7334      7319    101.25       116    0.473422    0.622672
  290      16      7360      7344   101.246       100    0.953368    0.622936
  291      16      7381      7365   101.186        84    0.246506    0.622984
  292      16      7408      7392    101.21       108    0.325467    0.623197
  293      16      7432      7416   101.192        96   0.0971356     0.62308
  294      15      7456      7441   101.187       100   0.0976546    0.623063
  295      15      7482      7467   101.197       104    0.978687    0.623418
  296      16      7506      7490   101.166        92     1.04283    0.623518
  297      16      7532      7516   101.175       104    0.452704    0.623388
  298      15      7556      7541   101.171       100    0.239914    0.623335
  299      16      7580      7564    101.14        92    0.268488    0.623272
2019-05-18 16:38:02.245811 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.62314
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      7604      7588   101.123        96    0.261678     0.62314
  301      16      7629      7613   101.119       100    0.345197    0.623453
  302      16      7655      7639   101.128       104     1.27369    0.623607
  303      16      7681      7665   101.138       104    0.424676    0.623547
  304      16      7704      7688   101.108        92    0.489889    0.623969
  305      16      7727      7711   101.077        92    0.409788    0.624143
  306      16      7756      7740   101.126       116    0.781044    0.623874
  307      16      7784      7768   101.161       112    0.308747    0.623663
  308      16      7809      7793   101.157       100    0.385183    0.623397
  309      16      7836      7820   101.179       108     1.08285    0.623437
  310      16      7861      7845   101.176       100    0.219953    0.623419
  311      16      7886      7870   101.172       100    0.473606    0.623255
  312      16      7913      7897   101.193       108    0.373179    0.623166
  313      16      7940      7924   101.215       108    0.299289    0.623072
  314      16      7961      7945    101.16        84    0.460902    0.623042
  315      16      7989      7973   101.194       112      1.6262    0.623145
  316      16      8013      7997   101.177        96     1.03781    0.623491
  317      16      8038      8022   101.173       100    0.415992    0.623553
  318      15      8067      8052   101.232       120    0.326914    0.623371
  319      16      8090      8074   101.191        88    0.715763    0.623521
2019-05-18 16:38:22.256352 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.623574
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16      8113      8097   101.162        92    0.642659    0.623574
  321      16      8138      8122   101.158       100    0.290379    0.623648
  322      16      8164      8148   101.167       104    0.294173    0.623614
  323      16      8190      8174   101.175       104    0.283546    0.623708
  324      16      8213      8197   101.147        92    0.900547    0.623813
  325      15      8237      8222   101.143       100    0.190628    0.623697
  326      16      8263      8247    101.14       100    0.259647    0.623555
  327      15      8289      8274    101.16       108     0.67046     0.62356
  328      16      8316      8300   101.169       104    0.612266    0.623628
  329      15      8345      8330   101.226       120     0.10275      0.6233
  330      15      8371      8356   101.234       104    0.306141    0.623092
  331      15      8398      8383   101.255       108    0.499758    0.622897
  332      16      8424      8408   101.251       100    0.334223    0.622795
  333      16      8451      8435   101.271       108    0.241176    0.622599
  334      16      8478      8462   101.291       108    0.897724    0.622809
  335      16      8502      8486   101.275        96    0.610627    0.622765
  336      15      8528      8513   101.295       108     0.61575     0.62254
  337      16      8555      8539   101.303       104     1.26721    0.622779
  338      15      8579      8564   101.299       100    0.453215    0.622726
  339      15      8607      8592    101.33       112     0.17955     0.62261
2019-05-18 16:38:42.264516 min lat: 0.0164646 max lat: 2.24639 avg lat: 0.62247
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16      8635      8619    101.35       108   0.0334024     0.62247
  341      16      8661      8645   101.357       104    0.988156    0.622354
  342      16      8689      8673   101.388       112     1.26421     0.62202
  343      15      8715      8700   101.407       108    0.312452    0.621784
  344      16      8739      8723    101.38        92     1.32538    0.622013
  345      16      8763      8747   101.364        96    0.315033    0.622122
  346      16      8789      8773   101.372       104     1.32989    0.622242
  347      16      8813      8797   101.356        96    0.302235    0.622284
  348      16      8837      8821   101.341        96    0.288684     0.62229
  349      16      8862      8846   101.337       100     1.31734    0.622471
  350      16      8890      8874   101.367       112    0.924812    0.622399
  351      16      8916      8900   101.374       104    0.164308    0.622384
  352      15      8940      8925    101.37       100    0.333364    0.622447
  353      16      8967      8951   101.378       104    0.533656    0.622316
  354      15      8994      8979   101.408       112    0.899128    0.622287
  355      15      9018      9003   101.392        96    0.108629    0.622123
  356      16      9045      9029   101.399       104     1.38919    0.621981
  357      16      9069      9053   101.384        96    0.321945    0.622057
  358      16      9094      9078    101.38       100     1.39323    0.622078
  359      16      9119      9103   101.376       100    0.333002    0.622029
2019-05-18 16:39:02.273571 min lat: 0.0162125 max lat: 2.24639 avg lat: 0.62199
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16      9145      9129   101.383       104    0.313679     0.62199
  361      16      9173      9157   101.413       112    0.359067    0.621675
  362      16      9197      9181   101.398        96     1.97909    0.621929
  363      16      9219      9203    101.36        88    0.390756     0.62227
  364      16      9245      9229   101.368       104    0.267295    0.622265
  365      16      9271      9255   101.375       104    0.398881    0.622302
  366      16      9297      9281   101.382       104    0.165758    0.622169
  367      16      9320      9304   101.356        92    0.409822    0.622241
  368      16      9347      9331   101.374       108    0.319764    0.622099
  369      16      9374      9358   101.392       108    0.708846    0.622193
  370      16      9398      9382   101.377        96    0.870945    0.622281
  371      16      9423      9407   101.373       100    0.016725    0.622284
  372      16      9451      9435   101.402       112    0.398917    0.622177
  373      16      9477      9461   101.409       104    0.473994    0.622106
  374      16      9498      9482   101.362        84    0.709182    0.622427
  375      16      9523      9507   101.358       100     0.82152    0.622516
  376      16      9547      9531   101.344        96   0.0842975    0.622507
  377      16      9569      9553   101.308        88    0.352341     0.62252
  378      16      9591      9575   101.273        88    0.222197    0.622671
  379      16      9616      9600    101.27       100      1.9463    0.622653
2019-05-18 16:39:22.281595 min lat: 0.0162125 max lat: 2.24639 avg lat: 0.622871
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      15      9641      9626   101.277       104    0.556325    0.622871
  381      16      9668      9652   101.284       104    0.271111    0.622929
  382      16      9693      9677   101.281       100    0.724258    0.623197
  383      15      9719      9704   101.298       108    0.304844    0.622945
  384      16      9742      9726   101.263        88    0.181045    0.622843
  385      16      9766      9750    101.25        96    0.250694    0.623063
  386      16      9791      9775   101.246       100    0.736536    0.623055
  387      16      9815      9799   101.233        96    0.352645    0.623219
  388      16      9843      9827    101.26       112    0.554207      0.6232
  389      16      9870      9854   101.278       108    0.197903    0.623133
  390      16      9895      9879   101.274       100    0.143791    0.622853
  391      16      9923      9907   101.301       112     1.55711    0.622798
  392      16      9946      9930   101.278        92     1.73373    0.622746
  393      16      9973      9957   101.295       108      1.0784    0.622856
  394      16      9997      9981   101.281        96    0.382286     0.62273
  395      16     10024     10008   101.298       108    0.360277    0.622619
  396      16     10047     10031   101.274        92    0.435118    0.622975
  397      16     10077     10061   101.321       120     0.39789    0.622904
  398      15     10101     10086   101.318       100    0.607862    0.622839
  399      16     10127     10111   101.314       100    0.325718    0.622647
2019-05-18 16:39:42.289432 min lat: 0.0162125 max lat: 2.24639 avg lat: 0.622821
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     10152     10136   101.311       100    0.477864    0.622821
  401      16     10177     10161   101.307       100    0.609951    0.622777
  402      16     10203     10187   101.314       104    0.910757    0.622761
  403      16     10225     10209   101.281        88    0.429068    0.622846
  404      16     10251     10235   101.288       104    0.631193    0.622948
  405      16     10274     10258   101.265        92    0.874939    0.623077
  406      16     10301     10285   101.281       108    0.902182    0.623067
  407      16     10325     10309   101.268        96     1.37668    0.622906
  408      16     10352     10336   101.284       108    0.456801    0.623025
  409      16     10378     10362   101.291       104    0.131875    0.622916
  410      15     10406     10391   101.326       116    0.957796    0.622832
  411      16     10434     10418   101.343       108    0.512615    0.622766
  412      16     10457     10441    101.32        92    0.998587    0.622743
  413      16     10482     10466   101.317       100    0.910021    0.622885
  414      16     10509     10493   101.333       108    0.179019    0.622681
  415      16     10535     10519   101.339       104    0.148482    0.622501
  416      16     10562     10546   101.355       108    0.230529    0.622384
  417      16     10587     10571   101.352       100    0.456228    0.622491
  418      16     10614     10598   101.367       108    0.255072    0.622472
  419      16     10639     10623   101.364       100    0.482745     0.62235
2019-05-18 16:40:02.298111 min lat: 0.0162125 max lat: 2.24639 avg lat: 0.622485
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     10661     10645   101.332        88    0.207673    0.622485
  421      16     10689     10673   101.357       112    0.358032     0.62242
  422      16     10714     10698   101.354       100     0.32986    0.622377
  423      16     10739     10723   101.351       100    0.361026    0.622366
  424      15     10761     10746   101.329        92    0.628947    0.622615
  425      16     10786     10770   101.316        96    0.315204    0.622629
  426      16     10811     10795   101.313       100     1.88423    0.622736
  427      16     10836     10820    101.31       100      1.3641    0.622664
  428      16     10860     10844   101.297        96     0.18561    0.622787
  429      16     10885     10869   101.294       100     1.53478    0.622776
  430      15     10907     10892   101.273        92    0.463149    0.622786
  431      16     10932     10916    101.26        96    0.348039    0.622989
  432      16     10958     10942   101.267       104      1.2145    0.622954
  433      16     10980     10964   101.236        88     1.74358    0.623082
  434      16     11005     10989   101.233       100     1.37648    0.623134
  435      16     11031     11015   101.239       104    0.318803     0.62318
  436      16     11056     11040   101.236       100     1.38882    0.623289
  437      16     11081     11065   101.234       100     1.40606    0.623378
  438      16     11103     11087   101.203        88     1.43175    0.623454
  439      16     11126     11110   101.182        92     1.34274    0.623678
2019-05-18 16:40:22.303904 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.623615
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     11150     11134    101.17        96     1.34174    0.623615
  441      16     11178     11162   101.195       112    0.364975    0.623576
  442      16     11203     11187   101.192       100    0.286892    0.623624
  443      15     11229     11214   101.207       108    0.165792     0.62332
  444      15     11253     11238   101.196        96     1.32024     0.62343
  445      16     11274     11258   101.148        80    0.248892    0.623578
  446      16     11301     11285   101.163       108     1.62232    0.623629
  447      16     11328     11312   101.178       108     0.14199    0.623607
  448      16     11352     11336   101.167        96    0.432177    0.623638
  449      16     11376     11360   101.155        96     1.03429    0.623848
  450      16     11405     11389   101.188       116    0.400924    0.623709
  451      15     11428     11413   101.176        96    0.198013    0.623615
  452      16     11455     11439   101.183       104    0.281004    0.623568
  453      16     11479     11463   101.171        96    0.933552    0.623691
  454      16     11506     11490   101.186       108     0.24476     0.62357
  455      15     11530     11515   101.183       100    0.377038    0.623707
  456      16     11553     11537   101.154        88    0.351084    0.623754
  457      16     11579     11563   101.161       104    0.733259    0.623832
  458      16     11603     11587   101.149        96     1.28903     0.62382
  459      16     11627     11611   101.138        96    0.707616    0.623828
2019-05-18 16:40:42.310545 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.623834
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     11655     11639   101.162       112      1.0956    0.623834
  461      15     11682     11667   101.185       112    0.138695    0.623625
  462      16     11708     11692   101.182       100    0.423769    0.623504
  463      16     11732     11716   101.171        96    0.135995    0.623511
  464      16     11755     11739   101.151        92    0.991618    0.623869
  465      16     11781     11765   101.157       104     0.21035    0.623825
  466      15     11807     11792   101.172       108    0.732382    0.623807
  467      16     11834     11818   101.178       104    0.154895    0.623704
  468      16     11858     11842   101.167        96    0.593438    0.623723
  469      15     11885     11870    101.19       112    0.846727    0.623747
  470      16     11909     11893    101.17        92    0.645488    0.623661
  471      15     11934     11919   101.176       104   0.0976751    0.623595
  472      16     11959     11943   101.165        96     1.50445    0.623631
  473      16     11983     11967   101.154        96     0.29084    0.623521
  474      16     12006     11990   101.135        92    0.505835    0.623797
  475      15     12036     12021   101.183       124    0.769563    0.623879
  476      16     12062     12046    101.18       100    0.254357    0.623686
  477      16     12086     12070   101.169        96    0.310546    0.623523
  478      16     12112     12096   101.175       104    0.362366    0.623741
  479      16     12139     12123   101.189       108     1.02392    0.623669
2019-05-18 16:41:02.317042 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.62351
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     12163     12147   101.178        96     0.26893     0.62351
  481      16     12191     12175   101.201       112     1.66726     0.62341
  482      15     12215     12200   101.198       100    0.263703    0.623547
  483      16     12242     12226   101.204       104   0.0997891    0.623308
  484      16     12266     12250   101.193        96    0.561882    0.623572
  485      16     12290     12274   101.183        96    0.252513    0.623642
  486      16     12320     12304   101.221       120     1.31505    0.623549
  487      16     12345     12329   101.219       100    0.488425    0.623551
  488      16     12367     12351   101.191        88    0.258854    0.623593
  489      16     12392     12376   101.189       100     0.35106    0.623608
  490      16     12416     12400   101.178        96    0.276139    0.623636
  491      16     12440     12424   101.168        96    0.398463    0.623796
  492      16     12467     12451   101.182       108    0.627924    0.623832
  493      16     12489     12473   101.155        88     1.15664    0.623973
  494      15     12515     12500   101.169       108    0.232104    0.623907
  495      16     12540     12524   101.158        96    0.221388    0.623876
  496      16     12567     12551   101.172       108    0.556117    0.623892
  497      16     12591     12575   101.161        96    0.296713    0.623887
  498      16     12616     12600   101.159       100    0.891934    0.623997
  499      16     12643     12627   101.173       108    0.755513    0.623871
2019-05-18 16:41:22.322391 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.62367
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     12671     12655   101.194       112     1.15459     0.62367
  501      16     12696     12680   101.192       100     1.66049    0.623678
  502      15     12719     12704   101.181        96    0.141729    0.623795
  503      16     12745     12729   101.179       100    0.208191    0.623707
  504      16     12771     12755   101.184       104    0.288907    0.623617
  505      16     12792     12776    101.15        84     1.13545    0.623843
  506      16     12816     12800    101.14        96     1.56839    0.623935
  507      16     12844     12828   101.162       112    0.427444    0.623991
  508      16     12871     12855   101.175       108      1.3028    0.623872
  509      16     12894     12878   101.157        92    0.487957    0.623944
  510      16     12922     12906   101.178       112    0.303017    0.623721
  511      15     12943     12928   101.152        88    0.613304    0.623758
  512      15     12969     12954   101.158       104    0.379331    0.623884
  513      16     12998     12982   101.179       112    0.268611    0.623842
  514      16     13023     13007   101.176       100     1.24489    0.623836
  515      15     13044     13029   101.151        88    0.748533    0.623915
  516      16     13067     13051   101.125        88    0.249582    0.623969
  517      16     13092     13076   101.123       100    0.178046    0.624084
  518      16     13117     13101   101.121       100    0.243927      0.6241
  519      15     13143     13128   101.134       108    0.216457    0.623941
2019-05-18 16:41:42.328125 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.62396
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      15     13168     13153   101.132       100    0.268564     0.62396
  521      16     13194     13178    101.13       100     0.65215    0.624151
  522      16     13223     13207   101.158       116     0.16556    0.623938
  523      16     13248     13232   101.156       100     1.15989    0.623904
  524      16     13274     13258   101.161       104     1.54665    0.623833
  525      16     13301     13285   101.174       108     0.84592    0.623819
  526      15     13322     13307   101.149        88    0.521004    0.623804
  527      16     13344     13328   101.116        84    0.238329    0.623946
  528      15     13369     13354   101.122       104    0.423115    0.623914
  529      16     13394     13378   101.112        96    0.264633    0.624021
  530      16     13420     13404   101.118       104    0.484972    0.624191
  531      16     13446     13430   101.123       104    0.840807    0.624324
  532      16     13471     13455   101.121       100    0.336517     0.62414
  533      15     13495     13480   101.119       100    0.296705    0.624111
  534      15     13521     13506   101.124       104    0.398844    0.624245
  535      15     13549     13534   101.144       112    0.531099    0.624137
  536      16     13577     13561   101.157       108    0.537673    0.623909
  537      16     13605     13589   101.177       112    0.270723    0.623817
  538      16     13628     13612    101.16        92     1.38737    0.623876
  539      16     13651     13635   101.143        92     0.66444    0.623879
2019-05-18 16:42:02.333256 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.623895
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     13677     13661   101.148       104    0.485122    0.623895
  541      16     13705     13689   101.168       112    0.325711    0.623817
  542      15     13729     13714   101.166       100     0.23929    0.623739
  543      16     13754     13738   101.156        96    0.225668    0.623774
  544      16     13778     13762   101.147        96    0.314298    0.624024
  545      16     13804     13788   101.152       104    0.299513    0.623983
  546      16     13830     13814   101.157       104    0.423109    0.623898
  547      15     13851     13836   101.133        88    0.349053    0.624062
  548      16     13878     13862   101.138       104    0.815443    0.624154
  549      16     13904     13888   101.144       104    0.829252    0.624041
  550      16     13930     13914   101.149       104    0.227483    0.624092
  551      16     13954     13938   101.139        96    0.802361     0.62404
  552      15     13979     13964   101.144       104    0.619761     0.62405
  553      16     14004     13988   101.135        96    0.578863    0.624097
  554      16     14032     14016   101.155       112    0.423879    0.623955
  555      16     14057     14041   101.152       100    0.255772    0.623944
  556      15     14084     14069   101.172       112    0.423165    0.623851
  557      16     14106     14090   101.141        84     1.00822    0.624029
  558      16     14133     14117   101.153       108     0.28102    0.624015
  559      16     14158     14142   101.151       100    0.891352    0.624117
2019-05-18 16:42:22.338394 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.62392
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     14187     14171   101.178       116     1.08444     0.62392
  561      16     14210     14194   101.161        92    0.361377    0.623915
  562      16     14233     14217   101.145        92    0.641609    0.624037
  563      16     14260     14244   101.157       108    0.656376    0.624076
  564      16     14281     14265   101.127        84    0.327605    0.624055
  565      15     14306     14291   101.132       104     1.09122    0.624213
  566      16     14333     14317   101.137       104     1.12775    0.624032
  567      16     14361     14345   101.156       112    0.233945    0.623897
  568      16     14386     14370   101.154       100      0.4656    0.623927
  569      16     14409     14393   101.138        92     1.49445    0.624031
  570      16     14434     14418   101.135       100     1.17812    0.624084
  571      16     14461     14445   101.147       108    0.245416    0.623968
  572      16     14484     14468   101.131        92    0.211553    0.624014
  573      16     14510     14494   101.136       104    0.316826    0.623956
  574      15     14538     14523   101.162       116     1.30802     0.62402
  575      15     14562     14547   101.153        96    0.259787    0.623988
  576      16     14587     14571   101.144        96    0.370403    0.624095
  577      16     14611     14595   101.135        96    0.301201    0.624094
  578      16     14637     14621    101.14       104     0.35234    0.624165
  579      16     14663     14647   101.145       104     1.00704    0.624065
2019-05-18 16:42:42.343942 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.624077
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     14688     14672   101.143       100    0.681862    0.624077
  581      16     14713     14697   101.141       100    0.264823    0.624135
  582      16     14734     14718   101.111        84      1.4241    0.624202
  583      16     14758     14742   101.103        96    0.348541    0.624348
  584      15     14783     14768   101.108       104    0.316723     0.62437
  585      16     14811     14795   101.119       108    0.309481    0.624302
  586      16     14835     14819    101.11        96    0.301595     0.62437
  587      15     14859     14844   101.108       100    0.335854    0.624303
  588      16     14886     14870   101.113       104     1.32654    0.624342
  589      16     14914     14898   101.132       112    0.560732     0.62427
  590      16     14937     14921   101.116        92    0.393511    0.624277
  591      16     14963     14947   101.121       104    0.119612    0.624149
  592      15     14989     14974   101.133       108    0.214154     0.62413
  593      15     15014     14999   101.131       100    0.578366    0.624081
  594      16     15039     15023   101.122        96    0.607088    0.624198
  595      15     15063     15048    101.12       100     1.17425    0.624298
  596      16     15088     15072   101.111        96    0.971621    0.624359
  597      16     15113     15097    101.11       100    0.337654    0.624336
  598      15     15138     15123   101.114       104    0.886411    0.624304
  599      16     15165     15149   101.119       104    0.193092    0.624228
2019-05-18 16:43:02.350410 min lat: 0.0160655 max lat: 2.24639 avg lat: 0.624284
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      16     15183     15167   101.071        72    0.382814    0.624284
Total time run:       600.630222
Total reads made:     15183
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   101.114
Average IOPS:         25
Stddev IOPS:          1
Max IOPS:             31
Min IOPS:             18
Average Latency(s):   0.624714
Max latency(s):       2.24639
Min latency(s):       0.0160655
Patrick C. F. Ernzer avatar
Patrick C. F. Ernzer