TerraMaster F5-422 with Fedora 29

Table of Contents

I installed Fedora 29 on my TerraMaster F5-422 for a quick Ceph test.

Obviously if you plan to use Fedora on this hardware for anything more than a quick test, use the latest version! Or try RHEL7 or CentOS Stream on it.

This is my braindump.

Install Fedora

  1. create USB boot medium as per docs
  2. connect screen, keyboard, mouse and USB boot medium
  3. enter firmware setup
  4. change OS type from Windows to Intel Linux
  5. set boot to UEFI only
  6. save and exit
  7. boot from USB
  8. I did a minimal install with the Ansible node add-on selected

I started with F29, even though today F31 is the current version, as my existing Ceph cluster runs that and before I order at least 3 more of these F5-422 I wanted to see the F5-422 do Ceph for a week or three. So I quickly added it to my known working Ceph setup.

The plan is to eventually remove the 32 bit hardware (there’s nothing wrong with ARM, aarch64 is great). But I want to rid myself of the ODROID-HC2 boards, the 32 bit arch is old and the hardware turned out to have too many issues for my liking.

Install ssh Public Key and Test

pcfe@workstation ~ $ ssh-copy-id -f -i ~/.ssh/id_USBkey.pub root@f5-422-01
pcfe@workstation ansible (master) $ ansible f5-422-01 -m ping -e ansible_user=root
f5-422-01 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Configure

since this is only for a quick test, I did not bother to properly rename my existing arm playbooks, I just added the host.

pcfe@workstation pcfe.net (master) $ ansible-playbook -l f5-422-01 arm-fedora-initial-setup.yml
pcfe@workstation pcfe.net (master) $ ansible-playbook -l f5-422-01 ceph-prepare-arm.yml
pcfe@workstation pcfe.net (master) $ ansible-playbook -l f5-422-01 terramaster-general-setup.yml

Setup for Ceph

The F5-422 was added to my known working Ceph setup simply by adding the host to my [osds] section and in my hostvars file for it

devices:
  - /dev/sdc

(the latter because until now all my ODROID OSDs used /dev/sda).

and then I ran ceph-ansible again.

[ansible@ceph-ansible ceph-ansible]$ ansible-playbook -vv site.yml 

Check

OSD shows up as expected

[root@ceph-mon-01 ~]# ceph osd tree
ID  CLASS WEIGHT   TYPE NAME              STATUS REWEIGHT PRI-AFF 
 -1       11.82544 root default                                   
-17        1.81929     host f5-422-01                             
  7   hdd  1.81929         osd.7              up  1.00000 1.00000 
-13        0.90959     host odroid-hc2-00                         
  6   hdd  0.90959         osd.6              up  1.00000 1.00000 
-11        0.90970     host odroid-hc2-01                         
  0   hdd  0.90970         osd.0              up  1.00000 1.00000 
 -9        0.90970     host odroid-hc2-02                         
  4   hdd  0.90970         osd.4              up  1.00000 1.00000 
-15        1.81929     host odroid-hc2-03                         
  5   hdd  1.81929         osd.5              up  1.00000 1.00000 
 -3        1.81929     host odroid-hc2-04                         
  1   hdd  1.81929         osd.1              up  1.00000 1.00000 
 -5        1.81929     host odroid-hc2-05                         
  2   hdd  1.81929         osd.2              up  1.00000 1.00000 
 -7        1.81929     host odroid-hc2-06                         
  3   hdd  1.81929         osd.3              up  1.00000 1.00000 

Testing Expectations

What follows is just a quick test to see if things work as expected.

I started with only a single HDD with BlueStore in the F5-422.

The plan is to eventually use 1 SATA SSD and up to 4 SATA HDD in this 5 slot chassis, as well as the 10G NIC for cluster_network, once I get rid of the ODROID-HC2 boards and purchased a 10Gig switch.

Note that since the ODROID-HC2 nodes (⅞ of the storage cluster) have only one network interface, the below quick test was run with only the PCIe 1Gig connection of the F5-422 (⅛ of cluster).

Since ⅞ of my OSDs are small ARM 32 bit boards, this is not about how much performance I get, this is just to give the F5-422 a smoke test. My expectation is to see roughly the numbers I get from ODROID-HC nodes with possibly a small amount of performance increase. If only because since the last test I moved everything besides the OSD daemons away from the ODROID-HC2 nodes onto separate VMs on my hypervisor.

root@epyc ~ # virsh list | grep -i ceph
 22    ceph-rgw                       running
 23    ceph-dashboard                 running
 24    ceph-ansible.sattest.pcfe.net  running
 25    ceph-mds-01                    running
 26    ceph-mds-02                    running
 27    ceph-mon-mgr-01                running
 28    ceph-mon-mgr-02                running
 29    ceph-mon-mgr-03                running

Performance Measurements with rados bench

All rados bench tests were run on ceph-mon-01.

Sequential Write Test

Total time run:         600.902758
Total writes made:      9075
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     60.4091
Stddev Bandwidth:       16.1433
Max bandwidth (MB/sec): 104
Min bandwidth (MB/sec): 0
Average IOPS:           15
Stddev IOPS:            4
Max IOPS:               26
Min IOPS:               0
Average Latency(s):     1.05936
Stddev Latency(s):      0.528235
Max latency(s):         5.63799
Min latency(s):         0.205941

rados bench -p testbench 600 write --no-cleanup | tee --append $(date +%F)-rados-test-write

hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_ceph-mon-01.internal.pcfe.net_9766
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0      16        16         0         0         0           -           0
    1      16        25         9   35.9437        36    0.999667    0.719038
    2      16        36        20   39.9654        44    0.965181    0.820069
    3      16        53        37    49.303        68    0.514562    0.984278
    4      16        70        54   53.9737        68     1.62787    0.970278
    5      16        87        71   56.7762        68     1.85171    0.988755
    6      16       101        85   56.6459        56     1.78279     1.02105
    7      16       119       103   58.8366        72     1.83816     1.02544
    8      16       132       116   57.9816        52     1.01994     1.02574
    9      16       151       135   59.9821        76     0.84474     1.00335
   10      16       168       152    60.783        68      1.2096    0.985232
   11      16       181       165   59.9841        52     1.47608    0.993869
   12      16       194       178   59.3184        52    0.355273     1.01453
   13      16       209       193   59.3703        60     1.44779     1.01178
   14      16       224       208   59.4148        60    0.421049     1.02496
   15      16       237       221     58.92        52     1.33695     1.03794
   16      16       258       242   60.4866        84    0.867177     1.02964
   17      16       270       254   59.7519        48     1.15287     1.02833
   18      16       292       276   61.3191        88    0.512198     1.01418
   19      16       303       287   60.4074        44     1.33367     1.01234
2020-02-09 13:03:27.986707 min lat: 0.257212 max lat: 2.39554 avg lat: 1.0285
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       321       305   60.9865        72    0.299995      1.0285
   21      16       337       321   61.1281        64    0.462599     1.01964
   22      16       351       335   60.8947        56      1.3906     1.02236
   23      16       371       355   61.7249        80    0.279012     1.01151
   24      16       384       368   61.3194        52    0.494773      1.0052
   25      16       405       389   62.2262        84    0.979229     1.00596
   26      16       423       407   62.6018        72    0.920855     1.00169
   27      16       441       425   62.9495        72    0.446715    0.997841
   28      16       454       438   62.5583        52     1.40589    0.995986
   29      16       470       454   62.6078        64    0.707754    0.997711
   30      16       483       467    62.254        52     1.31482     1.00094
   31      16       495       479    61.793        48    0.502646     1.00576
   32      16       513       497   62.1117        72     0.72096      1.0149
   33      16       526       510   61.8051        52    0.641428     1.01502
   34      16       534       518   60.9285        32     1.52362     1.02316
   35      16       548       532   60.7875        56    0.396146     1.02767
   36      16       572       556   61.7652        96     1.01682     1.02673
   37      16       586       570   61.6092        56      1.0081     1.02211
   38      16       595       579   60.9353        36    0.545813     1.02074
   39      16       615       599   61.4238        80    0.551727     1.03196
2020-02-09 13:03:47.990056 min lat: 0.220596 max lat: 2.39554 avg lat: 1.02556
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16       630       614   61.3881        60      1.0477     1.02556
   41      16       639       623   60.7688        36    0.367287     1.02865
   42      16       662       646   61.5121        92     1.22463     1.03095
   43      16       675       659   61.2908        52     1.12965     1.02829
   44      16       688       672   61.0795        52    0.346489     1.03076
   45      16       707       691   61.4108        76    0.923547     1.02874
   46      16       724       708   61.5538        68    0.459035     1.02586
   47      16       745       729   62.0312        84     1.41263     1.02193
   48      16       756       740   61.6554        44    0.315988     1.02113
   49      16       765       749   61.1318        36     1.92003     1.02667
   50      16       785       769   61.5089        80    0.678639     1.02788
   51      16       805       789    61.871        80    0.438226     1.02257
   52      16       825       809   62.2194        80    0.703293     1.01647
   53      16       844       828   62.4793        76    0.822998     1.01489
   54      16       868       852   63.0998        96    0.518895     1.00806
   55      16       879       863   62.7525        44    0.297265     1.00619
   56      16       892       876   62.5604        52     2.09594     1.00896
   57      16       910       894   62.7258        72     1.16125     1.01106
   58      16       919       903    62.265        36     1.81771     1.01342
   59      16       935       919   62.2943        64    0.564432     1.01436
2020-02-09 13:04:07.992664 min lat: 0.220596 max lat: 2.39554 avg lat: 1.01463
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16       955       939   62.5892        80     0.37489     1.01463
   61      16       973       957   62.7433        72    0.499918     1.01136
   62      16       986       970   62.5699        52     1.69739     1.00877
   63      16      1005       989   62.7829        76     0.40583     1.01176
   64      16      1015       999   62.4269        40    0.334687      1.0126
   65      16      1025      1009   62.0818        40    0.371407      1.0149
   66      16      1039      1023   61.9896        56    0.793355     1.02191
   67      16      1061      1045   62.3776        88    0.494712     1.02043
   68      16      1075      1059   62.2837        56    0.392389     1.01806
   69      16      1092      1076   62.3664        68    0.558789     1.01538
   70      16      1101      1085   61.9897        36     1.67686     1.01626
   71      16      1113      1097   61.7926        48      1.3903     1.02393
   72      16      1125      1109    61.601        48     1.29205       1.027
   73      16      1145      1129   61.8529        80    0.759281     1.02669
   74      16      1162      1146   61.9358        68     1.09196      1.0274
   75      16      1181      1165   62.1232        76    0.558122     1.02336
   76      16      1195      1179   62.0426        56     0.36944     1.02193
   77      16      1212      1196   62.1199        68     1.50207     1.02285
   78      16      1220      1204   61.7337        32    0.881767     1.02454
   79      16      1234      1218    61.661        56      1.5472     1.02887
2020-02-09 13:04:27.995120 min lat: 0.220596 max lat: 2.39554 avg lat: 1.02964
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      1245      1229   61.4401        44     0.46947     1.02964
   81      16      1266      1250   61.7185        84    0.869824     1.03075
   82      16      1286      1270   61.9413        80    0.940127     1.02798
   83      16      1297      1281   61.7251        44    0.543775     1.02864
   84      16      1320      1304   62.0854        92      1.1444     1.02551
   85      16      1335      1319   62.0607        60     0.91686     1.02384
   86      16      1349      1333   61.9902        56     1.34621      1.0261
   87      16      1361      1345   61.8293        48    0.436642     1.02657
   88      16      1380      1364   61.9903        76    0.982497     1.02576
   89      16      1399      1383   62.1476        76    0.886141     1.02289
   90      16      1413      1397   62.0792        56    0.236951     1.02206
   91      16      1431      1415   62.1881        72    0.794521     1.02333
   92      16      1446      1430   62.1643        60    0.750718     1.02147
   93      16      1461      1445   62.1409        60     1.67608     1.02152
   94      16      1471      1455   61.9053        40     1.20769     1.02554
   95      16      1483      1467   61.7589        48    0.359532      1.0249
   96      16      1496      1480   61.6572        52     2.05729     1.02768
   97      16      1518      1502   61.9286        88     1.32309     1.02994
   98      16      1529      1513   61.7456        44      1.5995     1.03004
   99      16      1539      1523   61.5259        40    0.360335     1.03056
2020-02-09 13:04:47.997533 min lat: 0.220596 max lat: 2.39554 avg lat: 1.03137
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      1552      1536   61.4306        52    0.695813     1.03137
  101      16      1566      1550   61.3768        56    0.506788     1.03692
  102      16      1582      1566   61.4024        64    0.707194      1.0361
  103      16      1595      1579   61.3111        52     1.37598     1.03694
  104      16      1612      1596   61.3753        68    0.910059     1.03735
  105      16      1626      1610   61.3241        56     1.39157     1.04042
  106      16      1636      1620   61.1229        40    0.320451     1.03944
  107      16      1641      1625   60.7385        20    0.517629      1.0398
  108      16      1654      1638   60.6576        52     1.74088     1.04754
  109      16      1674      1658   60.8349        80     0.82953     1.04772
  110      16      1694      1678    61.009        80    0.924831     1.04541
  111      16      1708      1692   60.9639        56     1.07404     1.04466
  112      16      1726      1710   61.0623        72    0.767294     1.04258
  113      16      1743      1727   61.1236        68     1.18545     1.04204
  114      16      1761      1745    61.219        72     0.70038      1.0401
  115      16      1779      1763   61.3126        72      1.2081     1.03977
  116      16      1795      1779   61.3357        64    0.493872     1.03732
  117      16      1814      1798   61.4609        76     1.53608     1.03645
  118      16      1827      1811   61.3807        52     0.86784     1.03661
  119      16      1845      1829   61.4699        72     1.41413     1.03794
2020-02-09 13:05:08.000090 min lat: 0.220596 max lat: 2.49436 avg lat: 1.03769
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      1854      1838   61.2576        36       1.569     1.03769
  121      16      1869      1853   61.2471        60     0.31099     1.03764
  122      16      1886      1870   61.3024        68     1.62688     1.04082
  123      16      1903      1887   61.3568        68    0.990556     1.04129
  124      16      1921      1905   61.4425        72     1.00014      1.0381
  125      16      1938      1922   61.4949        68     1.18729     1.03735
  126      16      1954      1938   61.5148        64    0.362773      1.0356
  127      16      1968      1952   61.4713        56     1.49735     1.03482
  128      16      1984      1968    61.491        64    0.670749     1.03615
  129      16      2000      1984   61.5104        64    0.837719     1.03516
  130      16      2016      2000   61.5295        64    0.465029     1.03617
  131      16      2037      2021   61.7009        84    0.513157      1.0352
  132      16      2051      2035   61.6577        56    0.875675     1.03358
  133      16      2065      2049   61.6151        56    0.358026     1.03268
  134      16      2082      2066   61.6627        68    0.730844     1.03323
  135      16      2100      2084   61.7392        72     0.93341      1.0318
  136      16      2122      2106    61.932        88    0.542591     1.03002
  137      16      2140      2124   62.0054        72     1.29368     1.02922
  138      16      2151      2135   61.8749        44    0.255559     1.02919
  139      16      2168      2152   61.9189        68    0.884169     1.02933
2020-02-09 13:05:28.003044 min lat: 0.220596 max lat: 2.49436 avg lat: 1.02885
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      2178      2162   61.7623        40    0.750233     1.02885
  141      16      2192      2176   61.7214        56     1.64731      1.0324
  142      16      2212      2196     61.85        80    0.374879     1.03084
  143      16      2228      2212    61.865        64    0.869236     1.02976
  144      16      2253      2237   62.1297       100    0.776265     1.02849
  145      16      2266      2250   62.0598        52    0.822938     1.02738
  146      16      2273      2257   61.8265        28     1.36192     1.02872
  147      16      2284      2268   61.7052        44    0.403214     1.03043
  148      16      2298      2282   61.6666        56     1.69363     1.03147
  149      16      2323      2307   61.9238       100    0.855189     1.03117
  150      16      2341      2325   61.9909        72    0.608582     1.02907
  151      16      2362      2346   62.1366        84     1.35466     1.02789
  152      16      2373      2357   62.0173        44    0.300306     1.02737
  153      16      2388      2372    62.004        60    0.651139     1.02708
  154      16      2403      2387    61.991        60    0.563983     1.02757
  155      16      2425      2409   62.1587        88     1.13456     1.02645
  156      16      2437      2421   62.0679        48    0.343948     1.02596
  157      16      2458      2442   62.2075        84    0.773788      1.0263
  158      16      2471      2455   62.1429        52    0.512806     1.02592
  159      16      2485      2469   62.1042        56    0.517473     1.02558
2020-02-09 13:05:48.005461 min lat: 0.220596 max lat: 2.49436 avg lat: 1.02636
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      2499      2483    62.066        56    0.375696     1.02636
  161      16      2517      2501   62.1276        72     1.54075     1.02669
  162      16      2528      2512   62.0157        44     1.56129     1.02713
  163      16      2542      2526   61.9788        56     1.02257     1.02927
  164      16      2555      2539   61.9179        52    0.438684     1.02941
  165      16      2570      2554   61.9062        60    0.601587     1.02947
  166      16      2584      2568   61.8706        56    0.540074     1.03022
  167      16      2601      2585   61.9073        68     1.05229     1.03053
  168      16      2619      2603   61.9673        72     0.74145     1.02944
  169      16      2634      2618   61.9556        60    0.883269     1.02854
  170      16      2654      2638   62.0617        80    0.254259     1.02775
  171      16      2672      2656   62.1198        72    0.728238      1.0275
  172      16      2686      2670   62.0841        56    0.515241     1.02622
  173      16      2697      2681   61.9796        44    0.365901     1.02593
  174      16      2713      2697   61.9911        64    0.770919       1.027
  175      16      2736      2720   62.1626        92    0.887043     1.02693
  176      16      2748      2732   62.0821        48    0.505998     1.02611
  177      16      2767      2751   62.1606        76    0.520819      1.0279
  178      16      2779      2763   62.0811        48    0.813506     1.02719
  179      16      2797      2781   62.1364        72    0.838757     1.02715
2020-02-09 13:06:08.007867 min lat: 0.220596 max lat: 2.49436 avg lat: 1.02639
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      2815      2799   62.1912        72    0.458732     1.02639
  181      16      2828      2812   62.1348        52     1.21387     1.02615
  182      16      2851      2835   62.2989        92    0.968025     1.02555
  183      16      2867      2851   62.3081        64    0.601778     1.02494
  184      16      2882      2866   62.2955        60     1.03278     1.02478
  185      16      2895      2879   62.2399        52     1.03038       1.025
  186      16      2905      2889   62.1203        40     1.01385     1.02566
  187      16      2921      2905   62.1303        64    0.684071     1.02599
  188      16      2936      2920   62.1189        60    0.378872     1.02502
  189      16      2951      2935   62.1077        60     1.00953     1.02695
  190      16      2967      2951   62.1176        64    0.337264     1.02701
  191      16      2980      2964   62.0646        52     1.38757     1.02714
  192      16      3002      2986   62.1996        88    0.459886     1.02686
  193      16      3016      3000   62.1674        56    0.524664      1.0253
  194      16      3027      3011   62.0738        44      1.8324     1.02701
  195      16      3039      3023   62.0016        48    0.405856     1.02924
  196      16      3057      3041   62.0525        72    0.963887      1.0288
  197      16      3077      3061   62.1436        80     0.71254     1.02875
  198      16      3090      3074   62.0923        52    0.975912     1.02771
  199      16      3103      3087   62.0416        52     1.04967     1.02893
2020-02-09 13:06:28.010229 min lat: 0.220596 max lat: 2.50054 avg lat: 1.02913
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      3117      3101   62.0113        56    0.585423     1.02913
  201      16      3137      3121   62.1008        80     1.21548      1.0293
  202      16      3149      3133   62.0309        48    0.287415     1.02835
  203      16      3159      3143   61.9224        40    0.837013     1.02883
  204      16      3179      3163   62.0109        80    0.303604     1.02961
  205      16      3197      3181   62.0596        72     1.26161      1.0295
  206      16      3211      3195   62.0302        56    0.529859     1.02928
  207      16      3227      3211   62.0397        64     1.00323     1.02997
  208      16      3244      3228   62.0683        68     0.54914     1.02875
  209      16      3261      3245   62.0966        68    0.879331     1.02803
  210      16      3282      3266   62.2009        84    0.443783     1.02641
  211      16      3302      3286   62.2852        80    0.759304     1.02482
  212      16      3321      3305   62.3493        76     1.22509     1.02394
  213      16      3338      3322   62.3758        68    0.344837     1.02359
  214      16      3357      3341   62.4394        76    0.813468     1.02318
  215      16      3370      3354   62.3908        52     1.29412     1.02281
  216      16      3389      3373   62.4538        76    0.676758      1.0224
  217      16      3406      3390   62.4793        68    0.682132     1.02198
  218      16      3419      3403   62.4312        52    0.630897     1.02085
  219      16      3441      3425   62.5479        88    0.605661      1.0207
2020-02-09 13:06:48.014569 min lat: 0.205941 max lat: 2.50054 avg lat: 1.01987
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      3460      3444    62.609        76    0.629193     1.01987
  221      16      3472      3456   62.5429        48    0.311749       1.019
  222      16      3489      3473   62.5674        68    0.346805     1.01954
  223      16      3501      3485   62.5021        48     1.54776      1.0207
  224      16      3517      3501   62.5087        64     0.48248     1.02122
  225      16      3526      3510   62.3909        36    0.383721     1.02057
  226      16      3544      3528   62.4333        72     0.45851      1.0231
  227      16      3558      3542    62.405        56     1.00434     1.02321
  228      16      3577      3561   62.4646        76     1.13025     1.02347
  229      16      3593      3577   62.4712        64     1.20973     1.02262
  230      16      3610      3594   62.4952        68    0.841369     1.02209
  231      16      3622      3606   62.4325        48     1.14124     1.02199
  232      16      3630      3614   62.3013        32     2.05326     1.02212
  233      16      3644      3628   62.2742        56     2.20645     1.02501
  234      16      3660      3644   62.2816        64    0.442021     1.02496
  235      16      3676      3660   62.2888        64     1.01779      1.0256
  236      16      3689      3673   62.2452        52    0.429697     1.02504
  237      16      3711      3695   62.3538        88    0.884283       1.025
  238      16      3719      3703   62.2263        32    0.350776     1.02422
  239      16      3737      3721   62.2671        72     1.92895     1.02503
2020-02-09 13:07:08.016976 min lat: 0.205941 max lat: 2.76841 avg lat: 1.02506
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      3749      3733   62.2077        48     1.68159     1.02506
  241      16      3763      3747   62.1819        56     2.23375     1.02586
  242      16      3777      3761   62.1563        56    0.651304     1.02582
  243      16      3794      3778   62.1803        68     1.20135     1.02726
  244      16      3812      3796   62.2205        72    0.304737     1.02673
  245      16      3829      3813   62.2441        68    0.827886     1.02579
  246      16      3848      3832      62.3        76     1.30815     1.02562
  247      16      3866      3850    62.339        72     1.24866     1.02474
  248      16      3885      3869   62.3941        76    0.344778     1.02343
  249      16      3903      3887   62.4326        72     1.20339     1.02308
  250      16      3915      3899   62.3749        48    0.388509     1.02282
  251      16      3936      3920    62.461        84    0.612729     1.02368
  252      16      3949      3933   62.4194        52    0.800179     1.02307
  253      16      3964      3948   62.4098        60    0.569355     1.02277
  254      16      3984      3968   62.4791        80    0.538567     1.02193
  255      16      4006      3990   62.5791        88    0.943727     1.02117
  256      16      4018      4002   62.5221        48    0.893481     1.02086
  257      16      4031      4015   62.4812        52    0.307218     1.02115
  258      16      4045      4029   62.4559        56     1.63501     1.02232
  259      16      4064      4048   62.5082        76      1.0753     1.02197
2020-02-09 13:07:28.020600 min lat: 0.205941 max lat: 3.24391 avg lat: 1.02176
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      4075      4059    62.437        44     1.25923     1.02176
  261      16      4088      4072   62.3969        52    0.575064     1.02285
  262      16      4107      4091   62.4488        76     1.15361      1.0226
  263      16      4125      4109   62.4851        72    0.734936     1.02225
  264      16      4147      4131   62.5817        88    0.881404     1.02143
  265      16      4158      4142   62.5116        44     1.11599     1.02103
  266      16      4173      4157   62.5021        60      1.4441     1.02109
  267      16      4187      4171   62.4777        56    0.777546     1.02133
  268      16      4194      4178   62.3491        28    0.345568     1.02147
  269      16      4204      4188    62.266        40    0.392465     1.02385
  270      16      4221      4205   62.2872        68      0.6742     1.02485
  271      16      4235      4219    62.264        56    0.457756     1.02509
  272      16      4252      4236    62.285        68     1.14141     1.02512
  273      16      4271      4255   62.3352        76    0.403897     1.02461
  274      16      4280      4264   62.2391        36     1.11847     1.02491
  275      16      4300      4284   62.3036        80    0.416841     1.02543
  276      16      4312      4296   62.2518        48    0.480459       1.025
  277      16      4324      4308   62.2003        48    0.880668      1.0258
  278      16      4344      4328   62.2643        80     0.50112     1.02584
  279      16      4360      4344   62.2705        64     1.51894     1.02538
2020-02-09 13:07:48.022958 min lat: 0.205941 max lat: 3.24391 avg lat: 1.02544
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      4376      4360   62.2767        64    0.298465     1.02544
  281      16      4387      4371   62.2116        44      1.2007     1.02566
  282      16      4406      4390   62.2605        76    0.852084     1.02632
  283      16      4421      4405   62.2525        60     1.22746     1.02621
  284      16      4437      4421   62.2586        64    0.537122     1.02592
  285      16      4452      4436   62.2506        60     1.12339     1.02612
  286      16      4473      4457   62.3265        84     1.25211      1.0259
  287      16      4485      4469   62.2766        48     1.56951     1.02577
  288      16      4499      4483   62.2548        56     1.07617     1.02578
  289      16      4520      4504     62.33        84    0.464628     1.02538
  290      16      4537      4521   62.3495        68     1.02101     1.02499
  291      16      4553      4537   62.3552        64    0.971103      1.0249
  292      16      4571      4555   62.3882        72    0.407792     1.02403
  293      16      4586      4570     62.38        60    0.960583     1.02378
  294      16      4599      4583   62.3447        52     1.39614     1.02458
  295      16      4621      4605   62.4316        88    0.921028     1.02364
  296      16      4629      4613   62.3288        32    0.499578     1.02363
  297      16      4647      4631   62.3613        72    0.767119     1.02416
  298      16      4666      4650   62.4069        76    0.520095     1.02394
  299      16      4681      4665   62.3988        60      1.3195     1.02341
2020-02-09 13:08:08.026566 min lat: 0.205941 max lat: 3.24391 avg lat: 1.02314
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      4700      4684   62.4441        76    0.912207     1.02314
  301      16      4713      4697   62.4094        52     1.26264     1.02314
  302      16      4730      4714   62.4279        68     1.30469      1.0233
  303      16      4747      4731   62.4462        68      1.1475     1.02362
  304      16      4763      4747   62.4513        64    0.515132     1.02338
  305      16      4777      4761   62.4302        56    0.776413     1.02327
  306      16      4782      4766   62.2915        20     1.72976      1.0234
  307      16      4798      4782    62.297        64     2.05171     1.02572
  308      16      4811      4795   62.2636        52     0.66068     1.02538
  309      16      4822      4806   62.2045        44       1.474     1.02616
  310      16      4826      4810   62.0554        16     1.61875     1.02632
  311      16      4848      4832   62.1388        88     1.60971     1.02848
  312      16      4862      4846   62.1191        56    0.648688     1.02866
  313      16      4874      4858    62.074        48      1.0823     1.02853
  314      16      4892      4876   62.1055        72     0.68574     1.02839
  315      16      4909      4893   62.1242        68     1.50787     1.02867
  316      16      4922      4906   62.0922        52      1.4479     1.02934
  317      16      4935      4919   62.0603        52    0.950579     1.02919
  318      16      4952      4936   62.0787        68     1.40094     1.02886
  319      16      4964      4948   62.0346        48     1.52399     1.02925
2020-02-09 13:08:28.030367 min lat: 0.205941 max lat: 3.24391 avg lat: 1.02983
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16      4979      4963   62.0282        60    0.773104     1.02983
  321      16      4996      4980   62.0468        68    0.380772     1.02956
  322      16      5014      4998   62.0776        72    0.968735     1.02929
  323      16      5029      5013   62.0712        60    0.921594     1.02895
  324      16      5044      5028   62.0648        60     1.42509     1.02931
  325      16      5057      5041   62.0338        52    0.494559     1.02956
  326      16      5073      5057   62.0398        64     1.42386     1.02956
  327      16      5084      5068   61.9846        44     1.67015     1.02991
  328      16      5103      5087   62.0273        76     1.70636     1.03066
  329      16      5113      5097   61.9603        40     1.56148     1.03027
  330      16      5126      5110   61.9301        52     1.44692     1.03113
  331      16      5136      5120   61.8639        40    0.520712     1.03115
  332      16      5152      5136   61.8703        64     1.78718     1.03266
  333      16      5168      5152   61.8766        64     1.34369     1.03248
  334      16      5188      5172   61.9309        80     0.82375     1.03196
  335      16      5202      5186   61.9131        56    0.468702     1.03185
  336      16      5213      5197   61.8598        44     1.36567     1.03194
  337      16      5226      5210   61.8305        52      1.7922     1.03301
  338      16      5243      5227   61.8488        68     1.41708     1.03352
  339      16      5256      5240   61.8197        52     1.06945     1.03398
2020-02-09 13:08:48.032895 min lat: 0.205941 max lat: 3.24391 avg lat: 1.03419
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16      5267      5251   61.7673        44     1.05264     1.03419
  341      16      5286      5270    61.809        76    0.781764     1.03431
  342      16      5298      5282   61.7686        48    0.400236     1.03419
  343      16      5315      5299   61.7867        68     0.30216     1.03396
  344      16      5333      5317   61.8164        72     1.18166     1.03362
  345      16      5350      5334   61.8343        68    0.517266     1.03348
  346      16      5366      5350   61.8405        64    0.446471     1.03369
  347      16      5376      5360   61.7776        40     1.23066      1.0335
  348      16      5392      5376   61.7839        64    0.267112     1.03374
  349      16      5407      5391   61.7788        60    0.263442      1.0342
  350      16      5420      5404   61.7508        52    0.432377     1.03446
  351      16      5434      5418   61.7344        56     1.53818     1.03475
  352      16      5447      5431   61.7068        52     1.09152     1.03539
  353      16      5460      5444   61.6793        52     1.44189     1.03574
  354      16      5480      5464   61.7309        80    0.733023     1.03564
  355      16      5493      5477   61.7034        52    0.698052     1.03498
  356      16      5506      5490   61.6762        52     1.34615     1.03611
  357      16      5522      5506   61.6827        64    0.524145     1.03558
  358      16      5538      5522   61.6891        64     1.37196     1.03607
  359      16      5554      5538   61.6955        64     1.24049     1.03582
2020-02-09 13:09:08.035982 min lat: 0.205941 max lat: 3.24391 avg lat: 1.03504
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16      5575      5559   61.7575        84    0.286644     1.03504
  361      16      5588      5572   61.7304        52     0.34341     1.03465
  362      16      5604      5588   61.7367        64    0.550665      1.0351
  363      16      5618      5602   61.7208        56    0.455966     1.03539
  364      16      5636      5620   61.7491        72    0.998482     1.03507
  365      16      5653      5637   61.7662        68      1.0843     1.03458
  366      16      5671      5655   61.7941        72    0.950376     1.03467
  367      16      5688      5672    61.811        68    0.593894     1.03392
  368      16      5704      5688   61.8169        64     1.31227     1.03378
  369      16      5718      5702   61.8011        56    0.293306     1.03356
  370      16      5734      5718   61.8071        64     0.63192      1.0345
  371      16      5750      5734   61.8129        64     1.27207     1.03411
  372      16      5761      5745   61.7651        44     1.99774     1.03409
  373      16      5776      5760   61.7603        60     2.19982     1.03442
  374      16      5794      5778   61.7877        72    0.535765     1.03488
  375      16      5808      5792   61.7722        56     1.44843     1.03473
  376      16      5828      5812   61.8207        80    0.336351       1.034
  377      16      5843      5827   61.8158        60    0.390151     1.03366
  378      16      5862      5846   61.8533        76     1.31242     1.03316
  379      16      5875      5859   61.8273        52     0.82727     1.03291
2020-02-09 13:09:28.038370 min lat: 0.205941 max lat: 3.24391 avg lat: 1.03359
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16      5890      5874   61.8225        60    0.632804     1.03359
  381      16      5902      5886   61.7862        48    0.483065     1.03344
  382      16      5912      5896   61.7291        40      2.1345     1.03466
  383      16      5930      5914   61.7559        72    0.800936     1.03515
  384      16      5947      5931   61.7721        68     1.24939     1.03471
  385      16      5966      5950   61.8091        76    0.334738     1.03429
  386      16      5978      5962   61.7733        48     1.24451     1.03434
  387      16      5991      5975    61.748        52     1.20561     1.03431
  388      16      6008      5992   61.7641        68    0.575021     1.03441
  389      16      6023      6007   61.7596        60     1.43968     1.03453
  390      16      6041      6025   61.7858        72     1.52231     1.03519
  391      16      6050      6034   61.7196        36    0.943453     1.03478
  392      16      6067      6051   61.7356        68      1.2327     1.03546
  393      16      6081      6065    61.721        56    0.251913     1.03533
  394      16      6098      6082    61.737        68    0.693522     1.03495
  395      16      6114      6098   61.7427        64    0.297509     1.03494
  396      16      6127      6111    61.718        52     0.35496       1.035
  397      16      6143      6127   61.7238        64    0.465007     1.03474
  398      16      6157      6141   61.7094        56    0.602841     1.03455
  399      16      6176      6160   61.7452        76     0.67921     1.03493
2020-02-09 13:09:48.041949 min lat: 0.205941 max lat: 3.24391 avg lat: 1.03464
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16      6194      6178   61.7708        72     0.55027     1.03464
  401      16      6211      6195   61.7863        68    0.628714     1.03428
  402      16      6230      6214   61.8216        76     0.96338      1.0342
  403      16      6249      6233   61.8568        76     1.15239     1.03366
  404      16      6266      6250    61.872        68     1.11788     1.03303
  405      16      6283      6267   61.8871        68      1.0447     1.03279
  406      16      6297      6281   61.8726        56     1.56996     1.03256
  407      16      6316      6300   61.9073        76     1.51083     1.03228
  408      16      6326      6310   61.8536        40    0.585202     1.03191
  409      16      6343      6327   61.8686        68    0.289246     1.03288
  410      16      6360      6344   61.8835        68     0.79128     1.03246
  411      16      6379      6363   61.9178        76    0.370006     1.03258
  412      16      6394      6378   61.9132        60     1.04894     1.03244
  413      16      6408      6392   61.8988        56     1.21645     1.03262
  414      16      6420      6404   61.8652        48     1.51116     1.03268
  415      16      6432      6416   61.8318        48      1.8108     1.03353
  416      16      6444      6428   61.7985        48     1.35205     1.03379
  417      16      6462      6446    61.823        72    0.477198     1.03392
  418      16      6481      6465   61.8569        76    0.281612     1.03334
  419      16      6496      6480   61.8524        60      0.6118     1.03339
2020-02-09 13:10:08.044431 min lat: 0.205941 max lat: 3.24391 avg lat: 1.03394
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16      6513      6497    61.867        68     1.10869     1.03394
  421      16      6521      6505   61.7961        32        0.59     1.03379
  422      16      6537      6521   61.8013        64    0.531809     1.03386
  423      16      6550      6534   61.7781        52     1.64083     1.03413
  424      16      6567      6551   61.7928        68    0.960975     1.03447
  425      16      6579      6563   61.7603        48    0.871078      1.0346
  426      16      6597      6581   61.7843        72    0.488322     1.03464
  427      16      6615      6599   61.8082        72     0.54849     1.03441
  428      16      6626      6610   61.7666        44    0.949446      1.0342
  429      16      6652      6636    61.865       104    0.375506     1.03365
  430      16      6667      6651   61.8607        60    0.944868     1.03333
  431      16      6689      6673   61.9213        88    0.790143     1.03269
  432      16      6705      6689   61.9261        64     1.20592     1.03216
  433      16      6718      6702   61.9031        52     1.29116     1.03226
  434      16      6726      6710   61.8342        32    0.625513     1.03219
  435      16      6743      6727   61.8484        68    0.967439     1.03313
  436      16      6764      6748   61.8992        84    0.868599     1.03273
  437      16      6777      6761   61.8765        52    0.967096     1.03258
  438      16      6794      6778   61.8905        68    0.365691     1.03241
  439      16      6811      6795   61.9044        68     1.16796     1.03228
2020-02-09 13:10:28.046795 min lat: 0.205941 max lat: 3.24391 avg lat: 1.03218
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16      6832      6816   61.9546        84    0.596438     1.03218
  441      16      6846      6830    61.941        56    0.497536     1.03174
  442      16      6861      6845   61.9366        60     1.31282     1.03149
  443      16      6872      6856   61.8961        44     1.73188     1.03211
  444      16      6885      6869   61.8738        52     1.89426     1.03242
  445      16      6897      6881   61.8426        48     1.24249     1.03348
  446      16      6913      6897   61.8475        64     1.13488     1.03326
  447      16      6921      6905   61.7807        32     1.74518     1.03368
  448      16      6934      6918   61.7588        52     1.59823     1.03522
  449      16      6951      6935   61.7727        68    0.545009     1.03464
  450      16      6971      6955   61.8132        80     1.20922     1.03434
  451      16      6989      6973   61.8358        72    0.573685     1.03393
  452      16      7005      6989   61.8406        64    0.873935     1.03362
  453      16      7012      6996   61.7658        28    0.417151     1.03337
  454      16      7012      6996   61.6298         0           -     1.03337
  455      16      7012      6996   61.4944         0           -     1.03337
  456      16      7012      6996   61.3595         0           -     1.03337
  457      16      7028      7012   61.3653        16       3.907     1.04027
  458      16      7043      7027   61.3623        60    0.793008     1.04172
  459      16      7055      7039   61.3331        48      1.4537     1.04231
2020-02-09 13:10:48.049082 min lat: 0.205941 max lat: 5.63799 avg lat: 1.0424
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16      7064      7048   61.2781        36     2.31373      1.0424
  461      16      7080      7064   61.2839        64    0.371133      1.0431
  462      16      7085      7069   61.1946        20     2.07235     1.04333
  463      16      7104      7088   61.2265        76    0.718602     1.04417
  464      16      7114      7098   61.1808        40    0.907796     1.04395
  465      16      7130      7114   61.1868        64      1.7112     1.04449
  466      16      7142      7126   61.1585        48     2.52743     1.04489
  467      16      7154      7138   61.1303        48    0.390537     1.04527
  468      16      7164      7148   61.0852        40     1.80031     1.04572
  469      16      7182      7166   61.1084        72     1.44167     1.04643
  470      16      7195      7179    61.089        52    0.828284     1.04628
  471      16      7205      7189   61.0442        40    0.867002     1.04642
  472      16      7209      7193   60.9488        16    0.867754     1.04633
  473      16      7211      7195   60.8369         8     3.20431     1.04694
  474      16      7224      7208   60.8182        52     2.64505     1.04986
  475      16      7239      7223   60.8165        60     1.38929     1.05108
  476      16      7256      7240   60.8315        68     1.11787     1.05092
  477      16      7271      7255   60.8298        60     1.21253     1.05097
  478      16      7287      7271   60.8364        64    0.576509     1.05062
  479      16      7303      7287    60.843        64    0.922556     1.05058
2020-02-09 13:11:08.051491 min lat: 0.205941 max lat: 5.63799 avg lat: 1.05043
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16      7321      7305   60.8662        72    0.953901     1.05043
  481      16      7331      7315   60.8228        40    0.238526     1.04998
  482      16      7348      7332   60.8377        68    0.715733     1.05086
  483      16      7363      7347    60.836        60    0.621127     1.05049
  484      16      7380      7364   60.8507        68     1.41647     1.05051
  485      16      7385      7369   60.7665        20     1.03776     1.05071
  486      16      7391      7375   60.6909        24    0.323356     1.05092
  487      16      7392      7376   60.5744         4     3.43949     1.05125
  488      16      7406      7390   60.5651        56     3.41944      1.0548
  489      16      7416      7400    60.523        40     1.59742      1.0557
  490      16      7427      7411   60.4893        44     1.65248     1.05627
  491      16      7442      7426   60.4882        60    0.544389      1.0571
  492      16      7455      7439    60.471        52    0.390467       1.057
  493      16      7468      7452   60.4538        52    0.339658     1.05667
  494      16      7487      7471   60.4852        76    0.309148     1.05714
  495      16      7505      7489   60.5085        72       1.154     1.05704
  496      16      7523      7507   60.5316        72     1.08829      1.0564
  497      16      7534      7518   60.4984        44    0.507418     1.05593
  498      16      7540      7524   60.4251        24    0.285469     1.05536
  499      16      7550      7534   60.3841        40    0.427295     1.05712
2020-02-09 13:11:28.054004 min lat: 0.205941 max lat: 5.63799 avg lat: 1.05812
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16      7570      7554   60.4233        80     1.57423     1.05812
  501      16      7579      7563   60.3746        36     1.37468     1.05814
  502      16      7599      7583   60.4136        80    0.388659     1.05825
  503      16      7620      7604   60.4605        84     1.21426     1.05765
  504      16      7633      7617   60.4437        52    0.737158     1.05731
  505      16      7650      7634   60.4587        68    0.467067     1.05726
  506      16      7667      7651   60.4735        68      1.2342     1.05714
  507      16      7677      7661   60.4331        40       1.277     1.05744
  508      16      7689      7673   60.4087        48     1.52622     1.05781
  509      16      7708      7692   60.4393        76    0.328656     1.05804
  510      16      7725      7709   60.4541        68    0.787158      1.0575
  511      16      7734      7718   60.4062        36    0.637417     1.05738
  512      16      7747      7731   60.3898        52    0.589714     1.05757
  513      16      7763      7747   60.3968        64     1.94518     1.05817
  514      16      7786      7770   60.4583        92    0.453422      1.0579
  515      16      7798      7782   60.4341        48    0.310308     1.05757
  516      16      7817      7801   60.4642        76    0.739695     1.05725
  517      16      7837      7821    60.502        80    0.880167     1.05674
  518      16      7855      7839   60.5242        72    0.367912      1.0567
  519      16      7872      7856   60.5386        68     1.26291      1.0563
2020-02-09 13:11:48.056526 min lat: 0.205941 max lat: 5.63799 avg lat: 1.05603
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16      7883      7867   60.5067        44    0.707214     1.05603
  521      16      7901      7885   60.5288        72     1.37964     1.05634
  522      16      7917      7901   60.5354        64    0.326238     1.05644
  523      16      7930      7914   60.5191        52    0.867512     1.05616
  524      16      7951      7935   60.5639        84     0.30196     1.05572
  525      16      7967      7951   60.5704        64     1.27139     1.05564
  526      16      7987      7971   60.6073        80     1.01489     1.05524
  527      16      8002      7986   60.6062        60    0.511636     1.05469
  528      16      8013      7997   60.5747        44    0.873698     1.05473
  529      16      8029      8013   60.5812        64     1.31828     1.05529
  530      16      8031      8015    60.482         8    0.695452      1.0553
  531      16      8056      8040   60.5564       100    0.938113     1.05621
  532      16      8066      8050   60.5177        40    0.842141     1.05611
  533      16      8081      8065   60.5167        60     1.50015     1.05665
  534      16      8093      8077   60.4933        48    0.371377     1.05673
  535      16      8111      8095   60.5148        72     1.20225     1.05663
  536      16      8123      8107   60.4914        48    0.377575     1.05652
  537      16      8141      8125   60.5128        72     0.27147     1.05627
  538      16      8156      8140   60.5119        60     1.15635     1.05645
  539      16      8174      8158   60.5332        72    0.572036     1.05609
2020-02-09 13:12:08.058892 min lat: 0.205941 max lat: 5.63799 avg lat: 1.05573
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16      8192      8176   60.5544        72     1.07973     1.05573
  541      16      8206      8190   60.5459        56     1.43701     1.05559
  542      16      8215      8199   60.5006        36      1.3601     1.05605
  543      16      8235      8219   60.5365        80    0.602381     1.05592
  544      16      8255      8239   60.5723        80    0.934667     1.05592
  545      16      8269      8253   60.5639        56    0.493145     1.05555
  546      16      8285      8269   60.5702        64    0.593794     1.05541
  547      16      8299      8283   60.5618        56    0.736539     1.05568
  548      16      8314      8298   60.5608        60     0.49003      1.0555
  549      16      8332      8316   60.5816        72     1.57046     1.05556
  550      16      8337      8321   60.5078        20    0.915039      1.0555
  551      16      8353      8337   60.5141        64     1.79984     1.05619
  552      16      8363      8347    60.477        40     1.85301     1.05661
  553      16      8373      8357   60.4399        40    0.383462     1.05652
  554      16      8389      8373   60.4463        64    0.475581     1.05718
  555      16      8402      8386   60.4311        52     1.40516     1.05802
  556      16      8417      8401   60.4303        60    0.892883     1.05786
  557      16      8437      8421   60.4654        80    0.789933     1.05754
  558      16      8448      8432   60.4359        44     1.91718     1.05759
  559      16      8462      8446    60.428        56    0.837368     1.05799
2020-02-09 13:12:28.061174 min lat: 0.205941 max lat: 5.63799 avg lat: 1.0572
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16      8484      8468   60.4772        88    0.273061      1.0572
  561      16      8497      8481   60.4621        52     1.58961     1.05717
  562      16      8509      8493   60.4399        48     1.29513     1.05744
  563      16      8526      8510   60.4533        68     1.16795     1.05767
  564      16      8542      8526   60.4595        64    0.499569     1.05746
  565      16      8559      8543   60.4729        68     1.53187     1.05725
  566      16      8575      8559   60.4791        64     1.36968     1.05724
  567      16      8592      8576   60.4923        68    0.385573     1.05697
  568      16      8614      8598   60.5407        88    0.658797     1.05656
  569      16      8626      8610   60.5187        48    0.461638     1.05637
  570      16      8641      8625   60.5177        60     1.32336     1.05649
  571      16      8659      8643   60.5378        72     0.60318     1.05626
  572      16      8678      8662   60.5648        76     1.04481     1.05568
  573      16      8697      8681   60.5917        76     1.22821     1.05536
  574      16      8713      8697   60.5976        64     1.27147     1.05506
  575      16      8725      8709   60.5757        48     1.68361     1.05526
  576      16      8743      8727   60.5955        72    0.549064     1.05523
  577      16      8763      8747   60.6289        80    0.837017     1.05506
  578      16      8775      8759    60.607        48    0.276577     1.05475
  579      16      8782      8766   60.5507        28     1.85951     1.05469
2020-02-09 13:12:48.067492 min lat: 0.205941 max lat: 5.63799 avg lat: 1.05548
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16      8796      8780   60.5428        56     2.12867     1.05548
  581      16      8808      8792   60.5212        48     1.83074     1.05589
  582      16      8824      8808   60.5272        64    0.703301     1.05628
  583      16      8841      8825     60.54        68     1.34371     1.05618
  584      16      8856      8840   60.5391        60    0.358973     1.05592
  585      16      8871      8855   60.5381        60    0.496655     1.05595
  586      16      8884      8868   60.5235        52     1.37576     1.05582
  587      16      8893      8877   60.4818        36     1.74324     1.05644
  588      16      8901      8885   60.4333        32     2.31317     1.05685
  589      16      8917      8901   60.4394        64     2.58398      1.0575
  590      16      8931      8915   60.4318        56    0.314227     1.05761
  591      16      8944      8928   60.4176        52    0.336832     1.05789
  592      16      8962      8946   60.4371        72     1.03321       1.058
  593      16      8982      8966   60.4701        80    0.674892     1.05755
  594      16      8997      8981   60.4693        60    0.546432     1.05692
  595      16      9006      8990   60.4281        36     2.02345     1.05733
  596      16      9024      9008   60.4475        72     1.52289     1.05789
  597      16      9034      9018   60.4133        40    0.559424      1.0579
  598      16      9044      9028   60.3791        40    0.295894     1.05781
  599      16      9059      9043   60.3785        60     1.34412     1.05868
2020-02-09 13:13:08.069836 min lat: 0.205941 max lat: 5.63799 avg lat: 1.05868
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      16      9074      9058   60.3779        60     1.33099     1.05868
Total time run:         600.902758
Total writes made:      9075
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     60.4091
Stddev Bandwidth:       16.1433
Max bandwidth (MB/sec): 104
Min bandwidth (MB/sec): 0
Average IOPS:           15
Stddev IOPS:            4
Max IOPS:               26
Min IOPS:               0
Average Latency(s):     1.05936
Stddev Latency(s):      0.528235
Max latency(s):         5.63799
Min latency(s):         0.205941

Sequential Read Test

Total time run:       243.001923
Total reads made:     9075
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   149.382
Average IOPS:         37
Stddev IOPS:          7
Max IOPS:             57
Min IOPS:             14
Average Latency(s):   0.427557
Max latency(s):       3.33833
Min latency(s):       0.0524101

rados bench -p testbench 600 seq | tee --append $(date +%F)-rados-test-read-seq

hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0      16        16         0         0         0           -           0
    1      16        60        44   175.847       176   0.0775366    0.266531
    2      16        95        79   157.914       140    0.577033    0.290436
    3      16       131       115   153.266       144    0.053747    0.344753
    4      16       161       145   144.947       120   0.0933719    0.373127
    5      16       200       184   147.152       156   0.0804509     0.38342
    6      16       235       219   145.957       140     1.06459    0.401643
    7      16       287       271   154.808       208    0.259545    0.402002
    8      16       335       319   159.453       192      0.7613    0.387464
    9      16       381       365   162.177       184   0.0532609    0.380072
   10      16       424       408   163.157       172     1.05304     0.38128
   11      16       478       462   167.958       216   0.0669362    0.369206
   12      16       520       504    167.96       168   0.0750025    0.365122
   13      16       547       531   163.347       108    0.180901    0.366318
   14      16       585       569   162.535       152    0.273444    0.370124
   15      16       625       609   162.365       160     1.14715    0.381174
   16      16       669       653   163.216       176     1.51887    0.379999
   17      16       707       691   162.555       152    0.162649    0.380454
   18      16       747       731   162.411       160    0.133613    0.383509
   19      16       778       762   160.389       124     1.23666     0.38894
2020-02-09 13:19:55.326900 min lat: 0.0532609 max lat: 2.40014 avg lat: 0.390717
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       816       800   159.969       152    0.381071    0.390717
   21      16       864       848   161.491       192    0.145493    0.387432
   22      16       894       878   159.605       120     0.25385    0.390173
   23      16       926       910    158.23       128    0.295835    0.390703
   24      16       960       944   157.303       136   0.0815618    0.394341
   25      16       994       978   156.451       136     2.38637    0.397148
   26      16      1021      1005   154.586       108   0.0533196    0.400685
   27      16      1055      1039   153.897       136     2.06706    0.404393
   28      16      1101      1085   154.972       184     0.12265    0.406813
   29      16      1129      1113   153.489       112     1.63875    0.408063
   30      16      1159      1143   152.373       120    0.134224    0.410572
   31      16      1209      1193   153.908       200    0.200264    0.410413
   32      16      1243      1227   153.348       136    0.305073    0.409512
   33      16      1267      1251    151.61        96     1.06397    0.410404
   34      16      1306      1290   151.738       156    0.635821    0.413746
   35      16      1349      1333   152.316       172    0.265106    0.414041
   36      16      1395      1379   153.195       184    0.169535    0.413877
   37      16      1440      1424   153.919       180    0.198539    0.411154
   38      16      1476      1460   153.657       144    0.468223    0.408878
   39      16      1504      1488   152.589       112    0.564755    0.410133
2020-02-09 13:20:15.329868 min lat: 0.0532609 max lat: 2.63311 avg lat: 0.413358
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1540      1524   152.374       144    0.217399    0.413358
   41      16      1573      1557   151.877       132    0.263172    0.413827
   42      16      1605      1589   151.305       128    0.179301    0.414932
   43      16      1628      1612   149.926        92    0.171398    0.417736
   44      16      1661      1645   149.518       132    0.336689    0.419903
   45      16      1708      1692   150.371       188    0.145116    0.420488
   46      16      1742      1726   150.058       136    0.598588    0.419873
   47      16      1779      1763   150.014       148    0.285508    0.421714
   48      16      1820      1804   150.305       164    0.054415    0.421285
   49      16      1855      1839   150.094       140     1.96416    0.420845
   50      16      1912      1896   151.652       228    0.148299    0.419428
   51      16      1951      1935   151.734       156    0.173392    0.417619
   52      16      1981      1965   151.124       120    0.194782    0.418048
   53      16      2017      2001   150.989       144     2.05148    0.418404
   54      16      2049      2033   150.563       128    0.620795     0.42025
   55      16      2096      2080   151.243       188    0.236774    0.419035
   56      16      2137      2121    151.47       164    0.549282    0.419392
   57      16      2175      2159    151.48       152    0.128907    0.419475
   58      16      2195      2179   150.247        80    0.506913    0.420962
   59      16      2227      2211    149.87       128     0.59782    0.420952
2020-02-09 13:20:35.334493 min lat: 0.0531553 max lat: 2.63311 avg lat: 0.422538
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      2270      2254   150.238       172    0.105766    0.422538
   61      16      2323      2307    151.25       212    0.246775    0.420618
   62      16      2362      2346   151.326       156    0.127161    0.419163
   63      16      2393      2377   150.892       124   0.0817976    0.419073
   64      16      2433      2417   151.034       160    0.120645    0.419084
   65      16      2473      2457   151.171       160    0.209661    0.419329
   66      16      2509      2493   151.062       144    0.267626    0.420065
   67      16      2548      2532   151.135       156    0.141572    0.419748
   68      16      2581      2565   150.852       132     1.36026    0.420943
   69      16      2612      2596   150.462       124    0.188965    0.421119
   70      16      2649      2633   150.427       148     1.85529    0.421967
   71      16      2689      2673   150.561       160     1.51653    0.421757
   72      16      2729      2713   150.692       160    0.360682    0.421859
   73      16      2759      2743   150.272       120     1.85738    0.421937
   74      16      2785      2769   149.646       104    0.316697    0.422557
   75      16      2824      2808   149.731       156    0.259173    0.423537
   76      16      2860      2844   149.655       144    0.373676    0.424584
   77      16      2892      2876   149.374       128     0.16918    0.424612
   78      16      2919      2903   148.843       108    0.348421    0.425454
   79      16      2963      2947   149.187       176    0.283136    0.425862
2020-02-09 13:20:55.338275 min lat: 0.0531553 max lat: 2.63311 avg lat: 0.42543
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      3007      2991   149.521       176    0.204953     0.42543
   81      16      3049      3033   149.749       168    0.231071    0.424879
   82      16      3088      3072   149.825       156    0.500131    0.423616
   83      16      3123      3107   149.707       140    0.288267     0.42383
   84      16      3160      3144   149.686       148     1.97354     0.42396
   85      16      3199      3183    149.76       156    0.372268    0.423324
   86      16      3244      3228   150.111       180    0.405586    0.422984
   87      16      3293      3277   150.639       196    0.157563    0.422653
   88      16      3331      3315   150.654       152    0.123038    0.422012
   89      16      3363      3347   150.399       128     1.09442    0.422538
   90      16      3388      3372   149.839       100    0.496452    0.423152
   91      16      3425      3409   149.819       148    0.760818    0.425024
   92      16      3468      3452    150.06       172    0.138161    0.424631
   93      16      3509      3493   150.209       164    0.160674    0.424151
   94      16      3558      3542   150.696       196   0.0738909    0.423001
   95      16      3592      3576   150.535       136    0.953022    0.422387
   96      16      3627      3611   150.423       140    0.205762    0.422503
   97      16      3663      3647   150.357       144    0.273836    0.423229
   98      16      3698      3682   150.251       140    0.858472    0.423501
   99      16      3750      3734   150.834       208    0.127689    0.421821
2020-02-09 13:21:15.345916 min lat: 0.0531553 max lat: 2.63311 avg lat: 0.421694
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3788      3772   150.845       152   0.0752186    0.421694
  101      16      3829      3813   150.975       164    0.145111    0.421627
  102      16      3872      3856   151.176       172    0.320933     0.42083
  103      16      3929      3913   151.922       228    0.792309    0.419872
  104      16      3969      3953   151.999       160    0.154354    0.419512
  105      16      4006      3990   151.961       148      0.7042    0.419115
  106      16      4034      4018   151.584       112       1.379    0.419524
  107      16      4062      4046   151.214       112     2.03765    0.419363
  108      16      4084      4068   150.629        88    0.679098    0.420921
  109      16      4125      4109   150.751       164    0.114924    0.421896
  110      16      4161      4145    150.69       144    0.888137    0.422524
  111      16      4194      4178   150.519       132    0.511809    0.422612
  112      16      4226      4210   150.318       128     1.89515    0.423278
  113      16      4272      4256   150.616       184    0.323965    0.423004
  114      16      4302      4286   150.347       120     1.68971    0.422566
  115      16      4332      4316   150.083       120     0.10206    0.422606
  116      16      4358      4342   149.686       104    0.106256    0.422926
  117      16      4396      4380   149.705       152    0.123916    0.423597
  118      16      4420      4404    149.25        96    0.149496     0.42524
  119      16      4458      4442   149.273       152    0.261799    0.425465
2020-02-09 13:21:35.353246 min lat: 0.0531303 max lat: 3.30881 avg lat: 0.426789
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      4492      4476   149.162       136     1.75688    0.426789
  121      16      4529      4513   149.153       148   0.0788137    0.426728
  122      16      4567      4551   149.176       152    0.129649    0.426331
  123      16      4609      4593   149.329       168    0.251372    0.426463
  124      16      4654      4638   149.576       180    0.116999    0.426195
  125      16      4704      4688   149.979       200     1.19288    0.425084
  126      16      4760      4744   150.566       224    0.359976     0.42397
  127      16      4800      4784    150.64       160    0.151588    0.423336
  128      16      4845      4829   150.869       180    0.148456    0.422336
  129      16      4877      4861   150.692       128    0.666148    0.422662
  130      16      4913      4897    150.64       144    0.135793    0.422667
  131      16      4947      4931   150.528       136    0.264926    0.423104
  132      16      4972      4956   150.146       100     1.55976    0.423592
  133      16      5001      4985   149.889       116    0.198971    0.423892
  134      16      5033      5017   149.725       128   0.0874727    0.424883
  135      16      5063      5047   149.505       120    0.162228    0.425276
  136      16      5093      5077   149.288       120      0.1525    0.426292
  137      16      5128      5112    149.22       140    0.179021    0.425921
  138      16      5168      5152   149.298       160   0.0896286    0.426577
  139      16      5203      5187   149.231       140     0.11625    0.426265
2020-02-09 13:21:55.355939 min lat: 0.0531303 max lat: 3.30881 avg lat: 0.426662
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      5230      5214   148.936       108    0.137863    0.426662
  141      16      5261      5245   148.759       124   0.0851853    0.427245
  142      16      5296      5280   148.698       140    0.215745    0.428069
  143      16      5329      5313   148.581       132   0.0771022    0.427788
  144      16      5364      5348   148.521       140   0.0797474     0.42885
  145      16      5395      5379   148.352       124     2.16831    0.428771
  146      16      5433      5417   148.377       152    0.541375    0.428757
  147      16      5464      5448   148.211       124    0.184271    0.428186
  148      16      5478      5462   147.588        56     0.17725    0.430023
  149      16      5508      5492   147.403       120    0.160663    0.431688
  150      16      5561      5545   147.833       212    0.198937    0.431373
  151      16      5598      5582   147.834       148     0.75565    0.431367
  152      16      5638      5622   147.914       160   0.0535967    0.431154
  153      16      5687      5671   148.228       196    0.391958    0.430557
  154      16      5728      5712    148.33       164    0.179783    0.429416
  155      16      5763      5747   148.276       140   0.0885647    0.429004
  156      16      5795      5779   148.146       128    0.167609    0.429838
  157      16      5829      5813   148.069       136    0.315391    0.430178
  158      16      5876      5860   148.321       188   0.0664483    0.430024
  159      16      5915      5899    148.37       156    0.176451    0.429568
2020-02-09 13:22:15.358399 min lat: 0.0531303 max lat: 3.33833 avg lat: 0.429197
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      5951      5935   148.342       144      1.7685    0.429197
  161      16      5988      5972    148.34       148     1.61828    0.429532
  162      16      6025      6009   148.338       148   0.0536185    0.429621
  163      16      6070      6054   148.532       180    0.253061    0.429276
  164      16      6124      6108   148.943       216   0.0534622    0.428329
  165      16      6168      6152   149.107       176    0.707872     0.42752
  166      16      6204      6188   149.076       144    0.313464     0.42766
  167      16      6257      6241   149.452       212   0.0583883    0.427309
  168      16      6295      6279   149.468       152    0.077126    0.427123
  169      16      6349      6333   149.861       216    0.230346    0.425888
  170      16      6393      6377   150.015       176    0.437854    0.425142
  171      16      6442      6426   150.283       196    0.203009    0.424346
  172      16      6477      6461   150.223       140     1.80927    0.424414
  173      16      6515      6499   150.234       152    0.128883    0.424008
  174      16      6549      6533   150.152       136    0.665493    0.424011
  175      16      6589      6573   150.208       160    0.369871    0.424103
  176      16      6616      6600   149.968       108     1.82216    0.425042
  177      16      6650      6634   149.889       136     1.13681    0.425122
  178      16      6686      6670   149.856       144    0.366945    0.425318
  179      16      6727      6711   149.935       164    0.264844    0.425365
2020-02-09 13:22:35.360916 min lat: 0.0531303 max lat: 3.33833 avg lat: 0.425429
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      6764      6748   149.924       148      1.2884    0.425429
  181      16      6807      6791   150.046       172    0.227826     0.42546
  182      16      6851      6835   150.188       176    0.989765    0.424759
  183      16      6889      6873   150.198       152    0.153524    0.424457
  184      16      6927      6911   150.208       152     1.72376    0.424285
  185      16      6970      6954   150.325       172     1.43522    0.424324
  186      16      7009      6993   150.356       156    0.117665    0.423928
  187      16      7045      7029   150.321       144     1.39232    0.424155
  188      16      7085      7069   150.372       160     0.18541    0.423924
  189      16      7129      7113   150.508       176    0.400542    0.423455
  190      16      7158      7142   150.326       116     1.81757    0.423803
  191      16      7200      7184   150.419       168       2.155    0.423877
  192      16      7239      7223   150.447       156    0.241659    0.423825
  193      16      7280      7264   150.517       164    0.227747    0.423551
  194      16      7330      7314   150.772       200     0.12798    0.423205
  195      16      7370      7354   150.819       160    0.289021     0.42322
  196      16      7402      7386   150.703       128    0.205269    0.422886
  197      16      7429      7413   150.486       108    0.367656    0.423419
  198      16      7466      7450   150.473       148    0.373544    0.423394
  199      16      7506      7490   150.521       160    0.104452    0.423617
2020-02-09 13:22:55.365401 min lat: 0.0531303 max lat: 3.33833 avg lat: 0.423257
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      7553      7537   150.708       188    0.147034    0.423257
  201      16      7608      7592   151.053       220     1.50131    0.422462
  202      16      7664      7648   151.414       224    0.158317    0.421623
  203      16      7710      7694   151.574       184   0.0540003    0.420806
  204      16      7745      7729   151.517       140    0.185686    0.420671
  205      16      7784      7768   151.539       156    0.264479     0.42058
  206      16      7840      7824   151.891       224    0.328503    0.420301
  207      16      7872      7856   151.775       128    0.200534    0.420101
  208      16      7902      7886   151.622       120    0.210371    0.420415
  209      16      7935      7919   151.528       132     1.79975    0.420584
  210      16      7959      7943   151.264        96    0.162232    0.420895
  211      16      7998      7982   151.286       156    0.455472    0.421297
  212      16      8037      8021   151.308       156     2.19187    0.421391
  213      16      8075      8059   151.312       152    0.141791     0.42122
  214      16      8117      8101    151.39       168    0.123355    0.420982
  215      16      8156      8140   151.411       156     1.94551    0.421002
  216      16      8202      8186   151.562       184    0.399453    0.420992
  217      16      8251      8235   151.766       196    0.251485    0.420436
  218      16      8303      8287   152.024       208    0.159079    0.419871
  219      16      8347      8331   152.134       176   0.0818713    0.419449
2020-02-09 13:23:15.367578 min lat: 0.0531303 max lat: 3.33833 avg lat: 0.419242
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      8377      8361   151.987       120    0.224291    0.419242
  221      16      8404      8388   151.788       108    0.554626    0.419395
  222      16      8431      8415   151.591       108   0.0530849     0.42018
  223      16      8464      8448   151.503       132    0.147943      0.4205
  224      16      8491      8475   151.309       108    0.109361      0.4212
  225      16      8512      8496    151.01        84     1.79258    0.421941
  226      16      8538      8522   150.802       104    0.144007    0.422318
  227      16      8565      8549   150.613       108    0.103389    0.422579
  228      16      8593      8577   150.444       112     0.12866    0.423409
  229      16      8625      8609   150.346       128    0.087667     0.42361
  230      16      8648      8632   150.092        92    0.381609     0.42446
  231      16      8683      8667   150.048       140     0.11909    0.424701
  232      16      8721      8705   150.056       152    0.105297    0.424458
  233      16      8752      8736   149.944       124    0.140606     0.42491
  234      16      8792      8776   149.987       160    0.216735    0.425044
  235      16      8836      8820   150.098       176   0.0852299    0.424983
  236      16      8869      8853   150.021       132    0.180573    0.424986
  237      16      8907      8891    150.03       152     2.16582    0.424848
  238      16      8941      8925   149.971       136    0.114504     0.42462
  239      16      8969      8953   149.812       112    0.380402    0.424974
2020-02-09 13:23:35.369986 min lat: 0.0524101 max lat: 3.33833 avg lat: 0.425338
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      9006      8990   149.804       148     2.44401    0.425338
  241      16      9037      9021   149.697       124    0.188784    0.425254
  242      16      9075      9059   149.706       152    0.241629     0.42553
Total time run:       243.001923
Total reads made:     9075
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   149.382
Average IOPS:         37
Stddev IOPS:          7
Max IOPS:             57
Min IOPS:             14
Average Latency(s):   0.427557
Max latency(s):       3.33833
Min latency(s):       0.0524101

Random Read Test

Total time run:       601.086899
Total reads made:     22341
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   148.671
Average IOPS:         37
Stddev IOPS:          7
Max IOPS:             58
Min IOPS:             16
Average Latency(s):   0.429814
Max latency(s):       4.37235
Min latency(s):       0.0371554

rados bench -p testbench 600 rand | tee --append $(date +%F)-rados-test-read-rand

hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0      16        16         0         0         0           -           0
    1      16        60        44   175.858       176    0.158665    0.267732
    2      16        98        82   163.923       152    0.288999    0.316885
    3      16       124       108   143.949       104    0.943632    0.352532
    4      16       162       146   145.957       152    0.350461     0.37387
    5      16       202       186   148.761       160    0.508248    0.382613
    6      16       234       218   145.157       128     1.36778    0.398673
    7      16       266       250   142.696       128    0.203283    0.401364
    8      16       315       299    149.35       196    0.161564    0.401738
    9      16       343       327   145.202       112    0.312733    0.406985
   10      16       371       355   141.883       112    0.158043    0.416767
   11      16       410       394   143.164       156    0.172161    0.420876
   12      16       455       439    146.23       180    0.121291    0.420175
   13      16       478       462    142.06        92    0.170826    0.428056
   14      16       503       487   139.056       100    0.164159    0.436489
   15      16       546       530    141.25       172    0.306101    0.437643
   16      16       590       574    143.42       176   0.0792493    0.434023
   17      16       627       611   143.688       148   0.0807894    0.435228
   18      16       672       656   145.703       180    0.277246    0.428428
   19      16       717       701   147.507       180    0.452415    0.421986
2020-02-09 13:29:03.131281 min lat: 0.0376865 max lat: 2.61661 avg lat: 0.421167
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       757       741    148.13       160    0.183407    0.421167
   21      16       787       771   146.732       120    0.126201    0.420784
   22      16       826       810   147.152       156   0.0730665    0.424724
   23      16       867       851   147.883       164    0.809074    0.427656
   24      16       908       892   148.553       164   0.0804222    0.424356
   25      16       953       937    149.81       180    0.811713    0.420847
   26      16       987       971   149.278       136    0.135855    0.422013
   27      16      1027      1011   149.674       160   0.0787468    0.421993
   28      16      1075      1059   151.184       192    0.770815    0.418425
   29      16      1124      1108   152.728       196     1.15775    0.413927
   30      16      1165      1149   153.103       164     1.52749    0.410547
   31      16      1204      1188   153.196       156   0.0740682    0.410169
   32      16      1243      1227   153.283       156    0.101785    0.410051
   33      16      1281      1265   153.242       152    0.141857    0.410422
   34      16      1326      1310   154.028       180    0.177054    0.409684
   35      16      1365      1349   154.084       156   0.0819008    0.411697
   36      16      1407      1391   154.467       168    0.124575    0.408415
   37      16      1448      1432   154.724       164    0.277221    0.406714
   38      16      1472      1456   153.179        96    0.115277    0.409442
   39      16      1500      1484   152.123       112    0.730056    0.410855
2020-02-09 13:29:23.142961 min lat: 0.0376865 max lat: 2.61661 avg lat: 0.41539
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1532      1516    151.52       128    0.139827     0.41539
   41      16      1576      1560   152.116       176    0.294529    0.415493
   42      16      1624      1608   153.065       192    0.160342     0.41267
   43      16      1653      1637   152.203       116   0.0773479    0.413331
   44      16      1686      1670   151.744       132      2.2709    0.414801
   45      16      1729      1713   152.193       172    0.419738    0.415394
   46      16      1770      1754    152.45       164    0.107991    0.414208
   47      16      1807      1791   152.354       148    0.260025    0.414528
   48      16      1844      1828   152.263       148    0.119184     0.41535
   49      16      1868      1852   151.113        96     2.15841    0.414248
   50      16      1911      1895    151.53       172    0.334442    0.418284
   51      16      1957      1941   152.166       184    0.322556    0.417423
   52      16      1997      1981   152.317       160    0.117873    0.415896
   53      16      2035      2019    152.31       152    0.129315    0.415912
   54      16      2075      2059   152.452       160    0.143052    0.415983
   55      16      2113      2097   152.443       152    0.857828    0.415232
   56      16      2154      2138   152.649       164   0.0790984    0.415533
   57      16      2180      2164   151.796       104     0.91278    0.416059
   58      16      2201      2185   150.627        84     2.36744    0.418061
   59      16      2224      2208   149.633        92    0.594693    0.420612
2020-02-09 13:29:43.146362 min lat: 0.0376865 max lat: 2.76688 avg lat: 0.421511
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      2254      2238   149.139       120    0.138713    0.421511
   61      16      2292      2276   149.186       152    0.215504    0.422926
   62      16      2318      2302   148.457       104     2.38942     0.42553
   63      16      2353      2337   148.322       140   0.0976515     0.42575
   64      16      2384      2368   147.942       124   0.0859998     0.42803
   65      16      2420      2404   147.881       144     1.58816    0.428678
   66      16      2450      2434   147.459       120     1.79534    0.429239
   67      16      2494      2478   147.884       176    0.126388    0.428008
   68      16      2534      2518   148.062       160    0.241649    0.427744
   69      16      2581      2565    148.64       188   0.0893798    0.426862
   70      16      2618      2602   148.631       148    0.154619    0.427007
   71      16      2645      2629   148.059       108    0.194826     0.42797
   72      16      2677      2661    147.78       128    0.666227    0.428323
   73      16      2709      2693   147.509       128     1.73092    0.429047
   74      16      2746      2730   147.515       148    0.133421    0.429745
   75      16      2777      2761   147.202       124      2.1762     0.43091
   76      16      2812      2796   147.107       140      1.5366    0.431482
   77      16      2836      2820   146.443        96    0.158553    0.431817
   78      16      2872      2856   146.411       144    0.223758    0.433251
   79      16      2906      2890   146.279       136    0.316876    0.434334
2020-02-09 13:30:03.148796 min lat: 0.0376865 max lat: 3.3428 avg lat: 0.433943
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      2937      2921   146.001       124    0.967747    0.433943
   81      16      2971      2955   145.877       136    0.106123    0.434621
   82      16      3003      2987   145.659       128     1.65283    0.436448
   83      16      3049      3033   146.121       184    0.127987    0.435355
   84      16      3081      3065   145.905       128    0.313379     0.43569
   85      16      3122      3106   146.117       164    0.188534    0.434673
   86      16      3162      3146   146.279       160    0.816611     0.43517
   87      16      3198      3182   146.252       144    0.109034    0.434223
   88      16      3231      3215    146.09       132    0.116493     0.43513
   89      16      3259      3243   145.707       112     2.10761    0.435495
   90      16      3298      3282   145.821       156     1.53151    0.436008
   91      16      3340      3324   146.065       168    0.169865    0.434881
   92      16      3380      3364   146.216       160    0.138548    0.434978
   93      16      3430      3414   146.793       200    0.123296    0.433745
   94      16      3461      3445   146.551       124     1.67255    0.433966
   95      16      3489      3473   146.187       112   0.0379765    0.434331
   96      16      3523      3507   146.081       136     2.33516    0.433726
   97      16      3565      3549   146.307       168    0.192651    0.434026
   98      16      3602      3586   146.324       148     0.13436    0.434322
   99      16      3637      3621    146.26       140    0.371054    0.434028
2020-02-09 13:30:23.151396 min lat: 0.0374512 max lat: 3.3428 avg lat: 0.434689
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3670      3654   146.117       132     2.05905    0.434689
  101      16      3706      3690   146.096       144    0.182232    0.435588
  102      16      3740      3724   145.997       136    0.167826    0.435741
  103      16      3766      3750   145.585       104      1.0061    0.436095
  104      16      3789      3773    145.07        92    0.176444    0.436826
  105      16      3822      3806   144.945       132   0.0790594    0.436852
  106      16      3863      3847   145.125       164   0.0379119    0.436696
  107      16      3903      3887   145.264       160    0.328483    0.437669
  108      16      3938      3922   145.215       140   0.0385383    0.437589
  109      16      3969      3953    145.02       124     2.10143    0.438391
  110      16      3996      3980   144.683       108    0.465435    0.438746
  111      16      4022      4006   144.317       104     1.81195    0.440043
  112      16      4044      4028   143.814        88    0.122725    0.440843
  113      16      4082      4066   143.886       152    0.330636    0.440686
  114      16      4118      4102   143.887       144   0.0806929    0.441327
  115      16      4148      4132   143.679       120    0.151149    0.441818
  116      16      4184      4168   143.682       144    0.080323    0.442375
  117      16      4216      4200   143.548       128   0.0744687    0.442939
  118      16      4259      4243   143.789       172    0.605825     0.44214
  119      16      4307      4291   144.194       192   0.0746907    0.442066
2020-02-09 13:30:43.156331 min lat: 0.0374512 max lat: 3.3428 avg lat: 0.44203
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      4340      4324   144.092       132     1.44918     0.44203
  121      16      4379      4363    144.19       156     1.27486    0.441935
  122      16      4417      4401   144.254       152    0.142835    0.441548
  123      16      4447      4431   144.057       120   0.0786897    0.441423
  124      16      4473      4457   143.733       104    0.102837    0.441509
  125      16      4506      4490   143.639       132     0.18636    0.442169
  126      16      4554      4538   144.023       192     2.08191    0.443009
  127      16      4592      4576   144.084       152    0.121165    0.441919
  128      16      4636      4620   144.333       176    0.154682    0.441381
  129      16      4672      4656    144.33       144   0.0383836    0.441397
  130      16      4714      4698   144.512       168    0.124568    0.440771
  131      16      4753      4737     144.6       156    0.482613    0.440308
  132      16      4773      4757    144.11        80    0.354087    0.440609
  133      16      4809      4793   144.109       144    0.257533    0.440989
  134      16      4850      4834   144.257       164    0.126673     0.44116
  135      16      4870      4854   143.781        80     1.63788    0.441457
  136      16      4895      4879   143.459       100   0.0840143    0.442665
  137      16      4932      4916   143.492       148    0.385376    0.443755
  138      16      4988      4972   144.075       224    0.239204    0.442806
  139      16      5038      5022   144.477       200    0.198079    0.441498
2020-02-09 13:31:03.160995 min lat: 0.0374512 max lat: 3.3428 avg lat: 0.440425
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      5083      5067   144.731       180     0.11063    0.440425
  141      16      5122      5106   144.811       156    0.101885    0.439882
  142      16      5164      5148   144.974       168    0.164648    0.439604
  143      16      5197      5181   144.883       132    0.117255    0.439584
  144      16      5236      5220    144.96       156   0.0808238    0.439155
  145      16      5283      5267   145.257       188    0.122624    0.438969
  146      16      5323      5307   145.358       160    0.224853     0.43844
  147      16      5361      5345   145.403       152   0.0380491    0.438389
  148      16      5396      5380   145.366       140   0.0818891    0.437947
  149      16      5422      5406   145.088       104   0.0885778    0.437951
  150      16      5455      5439   145.001       132     2.38527    0.438839
  151      16      5482      5466   144.756       108    0.307687    0.439159
  152      16      5519      5503   144.777       148    0.223955    0.439952
  153      16      5557      5541   144.824       152   0.0383704    0.439732
  154      16      5594      5578   144.845       148    0.213578    0.439746
  155      16      5626      5610   144.736       128      1.9044    0.440315
  156      16      5673      5657   145.013       188    0.445401    0.440056
  157      16      5723      5707   145.362       200   0.0738036     0.43908
  158      16      5767      5751   145.556       176    0.205701    0.438277
  159      16      5804      5788   145.572       148    0.546832    0.438022
2020-02-09 13:31:23.164115 min lat: 0.0374512 max lat: 3.3428 avg lat: 0.438017
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      5842      5826   145.612       152    0.334605    0.438017
  161      16      5886      5870     145.8       176    0.163778    0.437324
  162      16      5941      5925   146.258       220    0.498852    0.436071
  163      16      5993      5977   146.636       208     0.35404    0.435192
  164      16      6041      6025   146.912       192     1.18478    0.434504
  165      16      6090      6074   147.209       196    0.130566     0.43365
  166      16      6135      6119   147.406       180    0.123261    0.432333
  167      16      6176      6160   147.505       164     0.14226    0.431713
  168      16      6211      6195   147.461       140   0.0720339    0.432034
  169      16      6253      6237   147.582       168    0.140557    0.431869
  170      16      6278      6262   147.302       100    0.655208    0.431866
  171      16      6317      6301   147.352       156    0.185645    0.431953
  172      16      6355      6339   147.379       152     2.38242     0.43247
  173      16      6387      6371   147.267       128    0.323564    0.432333
  174      16      6420      6404   147.179       132      0.1495    0.432383
  175      16      6451      6435   147.047       124    0.472103    0.433516
  176      16      6492      6476   147.143       164    0.243844     0.43325
  177      16      6527      6511   147.102       140   0.0758257    0.432862
  178      16      6558      6542   146.972       124    0.159484    0.433098
  179      16      6597      6581   147.023       156   0.0995779    0.433892
2020-02-09 13:31:43.169046 min lat: 0.0373611 max lat: 3.3428 avg lat: 0.433515
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      6638      6622   147.117       164     1.35009    0.433515
  181      16      6677      6661   147.166       156    0.151905    0.433101
  182      16      6707      6691   147.017       120    0.866934    0.433229
  183      16      6750      6734   147.153       172    0.198544    0.433153
  184      16      6789      6773   147.201       156   0.0832541    0.433275
  185      16      6833      6817   147.356       176    0.155824    0.432757
  186      16      6873      6857   147.424       160    0.203863    0.432418
  187      16      6919      6903    147.62       184    0.169655    0.432303
  188      16      6960      6944   147.706       164    0.399398    0.431848
  189      16      7001      6985   147.792       164    0.120394     0.43144
  190      16      7045      7029   147.941       176    0.407866    0.430779
  191      16      7079      7063   147.878       136   0.0792319    0.431384
  192      16      7116      7100   147.878       148    0.387026     0.43102
  193      16      7150      7134   147.817       136    0.296334    0.431241
  194      16      7190      7174    147.88       160    0.119027    0.431118
  195      16      7220      7204   147.736       120    0.336479    0.430837
  196      16      7264      7248   147.881       176    0.121619    0.431127
  197      16      7296      7280    147.78       128   0.0916305    0.431389
  198      16      7330      7314    147.72       136     0.39415    0.431362
  199      16      7376      7360   147.902       184    0.323069    0.431474
2020-02-09 13:32:03.172450 min lat: 0.0372422 max lat: 3.3428 avg lat: 0.431176
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      7420      7404   148.043       176   0.0489673    0.431176
  201      16      7461      7445   148.122       164    0.273936    0.430824
  202      16      7497      7481   148.101       144    0.206799    0.430592
  203      16      7533      7517   148.081       144    0.154888    0.430434
  204      16      7574      7558   148.159       164    0.166686    0.430103
  205      16      7610      7594   148.139       144    0.135203    0.430414
  206      16      7650      7634   148.196       160      1.9791    0.430173
  207      16      7687      7671   148.195       148    0.187932    0.429802
  208      16      7737      7721   148.444       200    0.368127    0.429801
  209      16      7782      7766   148.595       180   0.0771454    0.429503
  210      16      7810      7794    148.42       112    0.197459     0.42953
  211      16      7843      7827   148.343       132    0.154616    0.429829
  212      16      7883      7867   148.397       160     1.26504    0.429809
  213      16      7918      7902   148.358       140    0.112432    0.429789
  214      16      7939      7923   148.057        84    0.115489    0.430173
  215      16      7969      7953   147.927       120     1.87709    0.430906
  216      16      8015      7999   148.093       184   0.0791509    0.431065
  217      16      8058      8042   148.203       172    0.392373    0.430655
  218      16      8100      8084   148.294       168    0.267835    0.430392
  219      16      8134      8118   148.238       136   0.0934882    0.430519
2020-02-09 13:32:23.175328 min lat: 0.0372422 max lat: 3.3428 avg lat: 0.430025
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      8179      8163   148.382       180    0.146404    0.430025
  221      16      8201      8185   148.109        88    0.102131    0.430244
  222      16      8247      8231    148.27       184    0.151443    0.430281
  223      16      8277      8261   148.144       120    0.113215    0.430249
  224      16      8309      8293   148.054       128     1.90335    0.430535
  225      16      8348      8332   148.089       156    0.137005    0.430501
  226      16      8393      8377    148.23       180     1.47221    0.430566
  227      16      8440      8424   148.405       188   0.0736608    0.430109
  228      16      8482      8466   148.491       168    0.310622     0.42972
  229      16      8522      8506   148.541       160    0.113297    0.429471
  230      16      8563      8547   148.608       164   0.0804247    0.429462
  231      16      8615      8599   148.865       208    0.152066    0.429022
  232      16      8666      8650   149.102       204    0.235952    0.428382
  233      16      8715      8699   149.303       196    0.278372    0.427459
  234      16      8760      8744   149.435       180    0.252322    0.427162
  235      16      8803      8787    149.53       172     0.10573    0.426811
  236      16      8841      8825   149.541       152     1.57202    0.426538
  237      16      8869      8853   149.382       112    0.308915    0.426468
  238      16      8898      8882   149.242       116     2.07807    0.427112
  239      16      8928      8912    149.12       120    0.353452    0.427552
2020-02-09 13:32:43.178247 min lat: 0.0372422 max lat: 3.3428 avg lat: 0.427817
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      8953      8937   148.915       100    0.359972    0.427817
  241      16      8983      8967   148.795       120    0.195311    0.428462
  242      16      9018      9002   148.759       140   0.0790235    0.428465
  243      16      9061      9045   148.854       172    0.131788    0.428893
  244      16      9114      9098   149.113       212     0.18917    0.428431
  245      16      9161      9145   149.271       188    0.568679    0.427854
  246      16      9207      9191   149.412       184    0.878253    0.427395
  247      16      9253      9237   149.552       184    0.290427    0.427006
  248      16      9290      9274   149.546       148   0.0996611    0.426601
  249      16      9329      9313   149.572       156    0.633384    0.426593
  250      16      9366      9350   149.565       148    0.201538    0.426602
  251      16      9399      9383   149.495       132     1.96777    0.426704
  252      16      9433      9417   149.442       136     0.16124    0.426873
  253      16      9472      9456   149.468       156       0.185    0.426775
  254      16      9506      9490   149.415       136    0.398723    0.426595
  255      16      9545      9529    149.44       156    0.183321    0.426623
  256      16      9587      9571   149.513       168    0.119509    0.426472
  257      16      9625      9609   149.522       152    0.385211     0.42689
  258      16      9664      9648   149.547       156    0.109105    0.426757
  259      16      9692      9676   149.402       112   0.0799794    0.426747
2020-02-09 13:33:03.181754 min lat: 0.0372422 max lat: 3.3428 avg lat: 0.42702
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      9728      9712   149.381       144     2.10264     0.42702
  261      16      9762      9746    149.33       136     2.72966     0.42707
  262      16      9787      9771   149.141       100     1.01599    0.427525
  263      16      9818      9802   149.046       124    0.136691    0.427962
  264      16      9860      9844   149.117       168     2.22855    0.427719
  265      16      9894      9878   149.068       136    0.178066    0.427983
  266      16      9923      9907   148.944       116    0.157223    0.428009
  267      16      9971      9955   149.105       192    0.193973    0.427733
  268      16     10017     10001   149.234       184    0.330592    0.427638
  269      16     10056     10040   149.259       156     1.13164    0.427578
  270      16     10090     10074    149.21       136   0.0725154    0.427934
  271      16     10125     10109   149.176       140   0.0797251    0.427801
  272      16     10156     10140   149.084       124    0.353268    0.427621
  273      16     10193     10177    149.08       148     0.10865    0.427679
  274      16     10233     10217   149.119       160     2.20318    0.428075
  275      16     10265     10249   149.043       128    0.200518    0.428026
  276      16     10300     10284    149.01       140    0.343428     0.42797
  277      16     10329     10313    148.89       116    0.865722    0.427923
  278      16     10356     10340   148.743       108     0.14778     0.42834
  279      16     10386     10370    148.64       120   0.0793715    0.428726
2020-02-09 13:33:23.185357 min lat: 0.0372422 max lat: 3.3428 avg lat: 0.429261
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16     10409     10393   148.438        92     2.88688    0.429261
  281      16     10434     10418   148.265       100     2.72532    0.429702
  282      16     10456     10440   148.052        88     0.16576    0.429962
  283      16     10504     10488   148.207       192    0.258442    0.430629
  284      16     10548     10532   148.305       176     0.32889    0.430555
  285      16     10583     10567   148.275       140     0.20307    0.430647
  286      16     10619     10603    148.26       144    0.312288    0.430365
  287      16     10664     10648    148.37       180    0.084919    0.430287
  288      16     10697     10681   148.313       132     0.97315    0.430161
  289      16     10735     10719   148.326       152    0.147487    0.430278
  290      16     10764     10748   148.215       116   0.0726217    0.430693
  291      16     10808     10792    148.31       176    0.125523    0.430387
  292      16     10848     10832    148.35       160     1.10635    0.430431
  293      16     10883     10867   148.321       140     0.79381    0.430366
  294      16     10931     10915    148.47       192   0.0885528    0.430116
  295      16     10972     10956   148.522       164    0.460296    0.429907
  296      16     11009     10993   148.521       148     1.27302    0.429746
  297      16     11049     11033   148.559       160    0.178288     0.42945
  298      16     11087     11071   148.571       152     2.30158    0.429519
  299      16     11129     11113   148.636       168    0.185823    0.429486
2020-02-09 13:33:43.188854 min lat: 0.0371554 max lat: 3.3428 avg lat: 0.429263
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16     11172     11156   148.713       172    0.167343    0.429263
  301      16     11210     11194   148.724       152    0.160433    0.429062
  302      16     11242     11226   148.656       128    0.212049    0.429122
  303      16     11287     11271   148.759       180    0.311421    0.429094
  304      16     11330     11314   148.835       172    0.235107    0.428883
  305      16     11372     11356   148.898       168    0.445686    0.428719
  306      16     11402     11386   148.804       120    0.397662     0.42869
  307      16     11440     11424   148.814       152    0.117302      0.4287
  308      16     11483     11467   148.889       172    0.171616    0.428745
  309      16     11518     11502    148.86       140     0.53991     0.42876
  310      16     11555     11539   148.858       148    0.168095    0.428708
  311      16     11593     11577   148.868       152    0.128028    0.428571
  312      16     11632     11616    148.89       156     1.94476    0.428652
  313      16     11665     11649   148.836       132    0.122182    0.428571
  314      16     11696     11680   148.757       124    0.112441    0.428616
  315      16     11721     11705   148.603       100    0.126284     0.42908
  316      16     11757     11741   148.588       144     0.31772    0.429367
  317      16     11799     11783   148.649       168   0.0836561    0.429329
  318      16     11829     11813   148.559       120     1.79385    0.429552
  319      16     11865     11849   148.544       144    0.122584    0.429648
2020-02-09 13:34:03.191367 min lat: 0.0371554 max lat: 3.3428 avg lat: 0.429959
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16     11901     11885    148.53       144   0.0860015    0.429959
  321      16     11939     11923   148.541       152   0.0772369    0.429541
  322      16     11972     11956    148.49       132     2.15216    0.429568
  323      16     12015     11999   148.562       172    0.169144    0.429662
  324      16     12060     12044   148.659       180    0.315283    0.429562
  325      16     12109     12093   148.805       196    0.323831    0.429345
  326      16     12145     12129    148.79       144     1.02612    0.429222
  327      16     12192     12176    148.91       188   0.0861337    0.428925
  328      16     12234     12218   148.968       168     1.51141    0.428639
  329      16     12263     12247   148.864       116    0.191959     0.42866
  330      16     12295     12279   148.801       128    0.083131    0.428944
  331      16     12333     12317    148.81       152     0.26636    0.428935
  332      16     12378     12362   148.904       180   0.0726336    0.428572
  333      16     12413     12397   148.877       140     1.60141    0.428564
  334      16     12446     12430   148.827       132    0.352465    0.428788
  335      16     12487     12471   148.872       164    0.239609    0.428926
  336      16     12508     12492   148.679        84    0.124293    0.428794
  337      16     12524     12508   148.428        64    0.222529    0.429045
  338      16     12549     12533   148.284       100   0.0383554    0.429388
  339      16     12570     12554   148.095        84    0.136781    0.429586
2020-02-09 13:34:23.202129 min lat: 0.0371554 max lat: 4.06014 avg lat: 0.430108
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16     12597     12581   147.977       108    0.691739    0.430108
  341      16     12620     12604   147.813        92    0.119019    0.431232
  342      16     12662     12646   147.872       168     2.43334    0.431444
  343      16     12697     12681   147.849       140    0.298401    0.431646
  344      16     12730     12714   147.802       132    0.221078     0.43167
  345      16     12764     12748   147.768       136     0.69765    0.431912
  346      16     12792     12776   147.665       112    0.137977    0.432159
  347      16     12818     12802   147.539       104    0.879846    0.432386
  348      16     12856     12840   147.552       152    0.125793    0.432321
  349      16     12890     12874   147.519       136   0.0863322    0.432704
  350      16     12920     12904    147.44       120   0.0381378    0.432904
  351      16     12962     12946   147.498       168     1.71075    0.432837
  352      16     13008     12992   147.602       184    0.149601     0.43253
  353      16     13045     13029   147.603       148     0.21414    0.432497
  354      16     13082     13066   147.604       148     0.20458     0.43238
  355      16     13116     13100   147.571       136    0.138245    0.432553
  356      16     13137     13121   147.393        84    0.800156    0.432668
  357      16     13167     13151   147.316       120    0.402946     0.43294
  358      16     13203     13187   147.307       144    0.199824    0.433138
  359      16     13236     13220   147.264       132    0.169808    0.433227
2020-02-09 13:34:43.205412 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.43338
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     13274     13258   147.277       152     0.20655     0.43338
  361      16     13300     13284   147.157       104    0.139172    0.433395
  362      16     13329     13313   147.071       116    0.240137    0.433706
  363      16     13367     13351   147.084       152    0.167571    0.433971
  364      16     13416     13400   147.219       196    0.537863    0.433818
  365      16     13464     13448   147.341       192    0.152654    0.433449
  366      16     13510     13494   147.441       184      1.4697    0.433321
  367      16     13547     13531   147.443       148    0.129573    0.433134
  368      16     13585     13569   147.455       152    0.961605    0.433204
  369      16     13633     13617   147.576       192    0.463153     0.43295
  370      16     13675     13659   147.631       168    0.180883    0.432731
  371      16     13703     13687   147.535       112    0.247103     0.43261
  372      16     13724     13708   147.364        84    0.590489    0.432958
  373      16     13751     13735   147.258       108     1.29877    0.433341
  374      16     13787     13771    147.25       144   0.0733255    0.433393
  375      16     13828     13812   147.294       164    0.134886    0.433677
  376      16     13875     13859   147.402       188    0.437709    0.433403
  377      16     13916     13900   147.446       164    0.909453    0.433289
  378      16     13962     13946   147.543       184    0.118898    0.432914
  379      16     14000     13984   147.555       152      0.2024    0.432853
2020-02-09 13:35:03.208344 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.432938
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     14035     14019   147.535       140    0.188379    0.432938
  381      16     14077     14061   147.588       168    0.187073    0.432745
  382      16     14105     14089   147.495       112    0.161368    0.432808
  383      16     14142     14126   147.496       148    0.623856    0.432817
  384      16     14176     14160   147.466       136    0.875272    0.432911
  385      16     14206     14190   147.395       120    0.159145    0.433174
  386      16     14237     14221   147.334       124     0.87072    0.433067
  387      16     14261     14245   147.202        96    0.350214    0.433437
  388      16     14311     14295   147.338       200    0.214412    0.433347
  389      16     14347     14331   147.329       144    0.258318    0.433424
  390      16     14389     14373   147.382       168     1.04781    0.433427
  391      16     14423     14407   147.353       136     0.72955    0.433336
  392      16     14462     14446   147.375       156    0.136917    0.433231
  393      16     14500     14484   147.386       152    0.453865    0.433165
  394      16     14521     14505   147.225        84    0.288692    0.433401
  395      16     14565     14549   147.298       176    0.276463    0.433538
  396      16     14598     14582   147.259       132       0.199    0.433735
  397      16     14635     14619   147.261       148    0.310182    0.433533
  398      16     14668     14652   147.223       132    0.878727    0.433452
  399      16     14700     14684   147.175       128    0.128098    0.433778
2020-02-09 13:35:23.212174 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.433684
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     14736     14720   147.167       144   0.0787955    0.433684
  401      16     14777     14761   147.209       164    0.295965    0.433589
  402      16     14814     14798   147.211       148    0.238222    0.433627
  403      16     14845     14829   147.153       124   0.0379297    0.433697
  404      16     14887     14871   147.204       168    0.120115    0.433773
  405      16     14936     14920   147.325       196    0.174135    0.433583
  406      16     14973     14957   147.326       148    0.348998     0.43344
  407      16     15004     14988   147.269       124    0.103221    0.433579
  408      16     15032     15016   147.183       112    0.438134    0.433809
  409      16     15065     15049   147.145       132   0.0854022    0.433849
  410      16     15097     15081   147.099       128   0.0913248    0.433897
  411      16     15133     15117   147.091       144    0.256522    0.433963
  412      16     15155     15139   146.948        88     2.12797    0.434136
  413      16     15190     15174   146.931       140    0.162838    0.434406
  414      16     15214     15198   146.808        96     1.47845    0.434799
  415      16     15252     15236    146.82       152     2.02248    0.434969
  416      16     15302     15286   146.948       200     1.46194    0.434657
  417      16     15338     15322   146.941       144     1.50939     0.43469
  418      16     15389     15373   147.077       204     1.12122    0.434352
  419      16     15428     15412   147.098       156    0.118907     0.43418
2020-02-09 13:35:43.215705 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.433971
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     15466     15450    147.11       152    0.825257    0.433971
  421      16     15495     15479   147.036       116    0.341953    0.434172
  422      16     15532     15516   147.038       148    0.180862    0.434288
  423      16     15575     15559   147.097       172     1.90854    0.434159
  424      16     15611     15595    147.09       144   0.0796344    0.434075
  425      16     15639     15623   147.007       112     2.04723    0.434116
  426      16     15675     15659       147       144     0.14453      0.4344
  427      16     15712     15696   147.003       148    0.303808    0.434444
  428      16     15741     15725    146.93       116     0.14508     0.43437
  429      16     15773     15757   146.886       128      2.3327    0.434543
  430      16     15804     15788   146.833       124    0.105307    0.434564
  431      16     15849     15833    146.91       180    0.448747    0.434632
  432      16     15879     15863   146.847       120    0.545647    0.434767
  433      16     15903     15887    146.73        96    0.147497    0.434895
  434      16     15938     15922   146.714       140    0.170654    0.434879
  435      16     15965     15949   146.625       108     1.06293    0.435043
  436      16     16007     15991   146.674       168    0.141577    0.435458
  437      16     16050     16034   146.732       172   0.0860765    0.435429
  438      16     16092     16076   146.781       168   0.0846797    0.435217
  439      16     16150     16134   146.975       232    0.506564    0.434777
2020-02-09 13:36:03.218220 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.434446
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     16192     16176   147.022       168    0.116958    0.434446
  441      16     16225     16209   146.988       132    0.166428    0.434419
  442      16     16257     16241   146.945       128     1.74999    0.434407
  443      16     16288     16272   146.893       124    0.554927    0.434489
  444      16     16315     16299   146.806       108    0.156289    0.434705
  445      16     16344     16328   146.737       116    0.387818    0.434832
  446      16     16375     16359   146.686       124    0.108814    0.435006
  447      16     16407     16391   146.644       128    0.294962    0.435501
  448      16     16430     16414   146.522        92   0.0737029    0.435745
  449      16     16463     16447   146.489       132    0.238807    0.435815
  450      16     16511     16495    146.59       192    0.204294    0.435873
  451      16     16548     16532   146.594       148    0.192584    0.435602
  452      16     16581     16565   146.561       132    0.106459    0.435719
  453      16     16624     16608   146.617       172   0.0717083    0.435704
  454      16     16670     16654     146.7       184     0.25675    0.435577
  455      16     16702     16686   146.658       128    0.153033    0.435571
  456      16     16738     16722   146.652       144   0.0997808    0.435646
  457      16     16785     16769   146.743       188    0.264005    0.435227
  458      16     16832     16816   146.833       188    0.160897    0.434993
  459      16     16874     16858   146.879       168    0.125864    0.434873
2020-02-09 13:36:23.221646 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.434652
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     16916     16900   146.925       168     1.53706    0.434652
  461      16     16950     16934   146.901       136    0.201656    0.434849
  462      16     17000     16984   147.016       200    0.439712    0.434664
  463      16     17039     17023   147.035       156    0.709505    0.434492
  464      16     17081     17065    147.08       168    0.312886    0.434325
  465      16     17121     17105   147.108       160    0.132298    0.434227
  466      16     17169     17153   147.204       192    0.238188    0.434038
  467      16     17202     17186   147.172       132     0.26168    0.434034
  468      16     17241     17225   147.191       156    0.109434    0.433921
  469      16     17272     17256   147.141       124    0.134454    0.433852
  470      16     17306     17290   147.117       136     2.16778    0.433897
  471      16     17345     17329   147.136       156    0.099281    0.433875
  472      16     17387     17371    147.18       168    0.123631    0.433757
  473      16     17423     17407   147.174       144   0.0376549    0.433794
  474      16     17471     17455   147.268       192    0.144803    0.433684
  475      16     17521     17505   147.379       200    0.110836    0.433521
  476      16     17568     17552   147.464       188     0.12726    0.433235
  477      16     17605     17589   147.465       148   0.0792458    0.433075
  478      16     17631     17615   147.374       104   0.0791533    0.433028
  479      16     17664     17648   147.342       132   0.0786209    0.433127
2020-02-09 13:36:43.224242 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.433168
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     17699     17683   147.327       140    0.428667    0.433168
  481      16     17740     17724   147.362       164     1.94898    0.433473
  482      16     17775     17759   147.346       140    0.083168    0.433377
  483      16     17811     17795   147.339       144    0.186749    0.433481
  484      16     17851     17835   147.365       160    0.835302    0.433621
  485      16     17903     17887    147.49       208    0.357078    0.433234
  486      16     17954     17938   147.607       204    0.220655    0.432858
  487      16     18000     17984   147.681       184    0.174906    0.432674
  488      16     18043     18027   147.731       172    0.136553    0.432389
  489      16     18088     18072   147.797       180    0.077572    0.432149
  490      16     18123     18107   147.781       140    0.282586    0.432009
  491      16     18161     18145    147.79       152    0.803411    0.432241
  492      16     18199     18183   147.798       152    0.654894     0.43224
  493      16     18231     18215   147.758       128    0.050921    0.432242
  494      16     18264     18248   147.726       132    0.232159    0.432271
  495      16     18283     18267   147.581        76    0.207232    0.432398
  496      16     18308     18292   147.485       100     2.29834    0.432985
  497      16     18343     18327    147.47       140    0.252852    0.433067
  498      16     18377     18361   147.447       136    0.162768    0.433042
  499      16     18423     18407    147.52       184     2.11463    0.433013
2020-02-09 13:37:03.226717 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.432696
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     18475     18459   147.641       208    0.349235    0.432696
  501      16     18513     18497    147.65       152    0.225278    0.432492
  502      16     18550     18534    147.65       148    0.319968    0.432492
  503      16     18586     18570   147.643       144    0.189793    0.432397
  504      16     18618     18602   147.604       128    0.125309    0.432459
  505      16     18648     18632   147.549       120    0.623812    0.432631
  506      16     18688     18672   147.574       160     0.30456    0.432762
  507      16     18733     18717   147.638       180    0.207914    0.432615
  508      16     18770     18754   147.639       148     1.45656    0.432581
  509      16     18810     18794   147.663       160    0.151149    0.432756
  510      16     18844     18828    147.64       136    0.437577     0.43261
  511      16     18892     18876   147.727       192    0.277336    0.432416
  512      16     18935     18919   147.774       172    0.202626    0.432224
  513      16     18981     18965   147.845       184    0.216221    0.432172
  514      16     19031     19015   147.946       200    0.323121     0.43195
  515      16     19061     19045   147.892       120    0.437502    0.431928
  516      16     19097     19081   147.884       144    0.266963    0.431934
  517      16     19143     19127   147.954       184    0.158258    0.431787
  518      16     19179     19163   147.946       144    0.285614    0.431802
  519      16     19227     19211   148.031       192    0.633673    0.431557
2020-02-09 13:37:23.230160 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.431264
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     19278     19262   148.138       204     0.25527    0.431264
  521      16     19326     19310   148.222       192     0.62698    0.430973
  522      16     19368     19352    148.26       168    0.100481    0.430859
  523      16     19407     19391   148.275       156    0.588045     0.43076
  524      16     19453     19437   148.343       184   0.0795811    0.430628
  525      16     19501     19485   148.426       192    0.159733    0.430423
  526      16     19548     19532   148.502       188    0.122866    0.430144
  527      16     19581     19565    148.47       132    0.136998    0.430003
  528      16     19611     19595   148.416       120    0.118945    0.430175
  529      16     19651     19635   148.438       160    0.128759    0.430271
  530      16     19674     19658   148.332        92    0.199943     0.43042
  531      16     19707     19691   148.301       132    0.134716    0.430539
  532      16     19743     19727   148.293       144     0.12066    0.430661
  533      16     19780     19764   148.292       148    0.169253    0.430572
  534      16     19807     19791   148.217       108     0.12908    0.430593
  535      16     19837     19821   148.164       120    0.633963    0.430943
  536      16     19880     19864   148.208       172     2.01449    0.430868
  537      16     19911     19895   148.163       124    0.785238    0.431099
  538      16     19960     19944   148.252       196    0.190768    0.430953
  539      16     20000     19984   148.274       160    0.165297    0.430835
2020-02-09 13:37:43.233537 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.4307
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     20049     20033   148.362       196    0.185534      0.4307
  541      16     20096     20080   148.435       188    0.327209    0.430413
  542      16     20139     20123   148.479       172    0.140309    0.430267
  543      16     20178     20162   148.492       156     0.40302    0.430212
  544      16     20231     20215   148.609       212     0.40325     0.43004
  545      16     20277     20261   148.674       184    0.189831     0.42974
  546      16     20324     20308   148.746       188    0.121609     0.42961
  547      16     20380     20364   148.883       224     0.24303    0.429192
  548      16     20423     20407   148.926       172    0.115516    0.428976
  549      16     20457     20441   148.902       136    0.148409    0.429049
  550      16     20496     20480   148.915       156    0.170918    0.429071
  551      16     20519     20503   148.811        92    0.881173    0.429094
  552      16     20553     20537   148.788       136      1.9161    0.429276
  553      16     20589     20573   148.779       144   0.0777044    0.429302
  554      16     20624     20608   148.764       140     2.00163    0.429387
  555      16     20667     20651   148.805       172    0.535448    0.429334
  556      16     20718     20702   148.905       204     0.52049    0.429135
  557      16     20750     20734   148.867       128    0.432513    0.429163
  558      16     20789     20773    148.88       156    0.606554    0.429139
  559      16     20820     20804   148.835       124     0.18241    0.429114
2020-02-09 13:38:03.238318 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.429272
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     20858     20842    148.84       152    0.150577    0.429272
  561      16     20893     20877   148.825       140    0.524481    0.429042
  562      16     20930     20914   148.823       148    0.450568    0.429132
  563      16     20979     20963   148.907       196   0.0823911    0.429005
  564      16     21023     21007   148.955       176    0.277439    0.428965
  565      16     21076     21060   149.066       212    0.152311    0.428674
  566      16     21115     21099   149.079       156    0.294167    0.428444
  567      16     21154     21138   149.091       156    0.079664    0.428444
  568      16     21186     21170   149.054       128     0.92728    0.428489
  569      16     21221     21205   149.038       140    0.152299    0.428494
  570      16     21262     21246   149.064       164    0.155614    0.428482
  571      16     21302     21286   149.083       160   0.0882518    0.428503
  572      16     21336     21320    149.06       136    0.176632    0.428527
  573      16     21391     21375   149.184       220    0.256822    0.428336
  574      16     21443     21427   149.286       208     0.43319    0.428055
  575      16     21488     21472    149.34       180    0.163608    0.427911
  576      16     21524     21508    149.33       144    0.551366     0.42787
  577      16     21560     21544   149.321       144    0.173311    0.427742
  578      16     21599     21583   149.333       156    0.179705    0.427757
  579      16     21644     21628   149.386       180     1.35689    0.427686
2020-02-09 13:38:23.240958 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.427513
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     21695     21679    149.48       204    0.220089    0.427513
  581      16     21731     21715    149.47       144    0.144685    0.427329
  582      16     21754     21738   149.371        92   0.0904589    0.427396
  583      16     21790     21774   149.362       144    0.435873    0.427551
  584      16     21823     21807   149.332       132     1.57209    0.427718
  585      16     21856     21840   149.303       132    0.164843    0.427771
  586      16     21889     21873   149.273       132    0.154224    0.427897
  587      16     21923     21907   149.251       136     1.71284    0.427955
  588      16     21948     21932   149.167       100    0.443082    0.428027
  589      16     21981     21965   149.138       132    0.149436    0.428017
  590      16     22023     22007    149.17       168      0.0753     0.42812
  591      16     22054     22038   149.127       124    0.138113    0.428243
  592      16     22091     22075   149.125       148     1.31866    0.428397
  593      16     22130     22114   149.137       156    0.153368    0.428261
  594      16     22161     22145   149.094       124    0.675096    0.428256
  595      16     22183     22167   148.992        88    0.119917    0.428565
  596      16     22218     22202   148.976       140    0.125538    0.428624
  597      16     22245     22229   148.908       108   0.0840039    0.428937
  598      16     22287     22271    148.94       168    0.313115    0.428859
  599      16     22311     22295   148.851        96    0.250167    0.428968
2020-02-09 13:38:43.243589 min lat: 0.0371554 max lat: 4.37235 avg lat: 0.429032
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      16     22341     22325   148.803       120     2.24232    0.429032
Total time run:       601.086899
Total reads made:     22341
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   148.671
Average IOPS:         37
Stddev IOPS:          7
Max IOPS:             58
Min IOPS:             16
Average Latency(s):   0.429814
Max latency(s):       4.37235
Min latency(s):       0.0371554

Cleanup

[root@ceph-mon-01 ~]# rados -p testbench cleanup
Removed 11790 objects

Ceph RBD Performance Measurements

Prepare libvirt

Since one can use libvirt with Ceph RBD, I created 2 vols on the libvirt side. Note that I set up libvirt with a pool.

root@hypervisor ~ # virsh vol-list --pool Ceph-HouseNet-libvirt-pool
 Name                 Path                                    
------------------------------------------------------------------------------
 banana               libvirt-pool/banana                     
 banana2              libvirt-pool/banana2                    

Prepare Test VM

Attached the 500 GiB RBD (banana2) to an existing VM as a virtio block device, formatted it with XFS, mounted it and ran fio.

[root@centos-stream ~]# dmesg 
[...]
[11506.366077] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)
[11506.371877] virtio-pci 0000:00:0b.0: virtio_pci: leaving for legacy driver
[11506.379137] virtio_blk virtio6: [vdc] 1048576000 512-byte logical blocks (537 GB/500 GiB)
[root@centos-stream ~]# lsblk
NAME                 MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                   11:0    1 1024M  0 rom  
vda                  252:0    0    1G  0 disk 
└─vda1               252:1    0 1023M  0 part /boot
vdb                  252:16   0   30G  0 disk 
└─vdb1               252:17   0   30G  0 part 
  ├─VG_cs-LV_root    253:0    0    8G  0 lvm  /
  ├─VG_cs-LV_swap    253:1    0  3.1G  0 lvm  [SWAP]
  ├─VG_cs-LV_home    253:2    0    1G  0 lvm  /home
  └─VG_cs-LV_var_log 253:3    0    1G  0 lvm  /var/log
vdc                  252:32   0  500G  0 disk 
[root@centos-stream ~]# mkfs.xfs /dev/vdc
meta-data=/dev/vdc               isize=512    agcount=4, agsize=32768000 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=131072000, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=64000, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@centos-stream ~]# mount /dev/vdc /mnt/
[root@centos-stream ~]# df -h /mnt/
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdc        500G  3.6G  497G   1% /mnt

Sequential Read, 4k Blocksize

  • READ: bw=84.9MiB/s
  • iops: avg=21742.01

fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4k

[root@centos-stream ~]# fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4k
readtest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
readtest: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [R(1)][99.9%][r=74.1MiB/s,w=0KiB/s][r=18.0k,w=0 IOPS][eta 00m:01s]
readtest: (groupid=0, jobs=1): err= 0: pid=32571: Sat Feb  8 17:45:56 2020
   read: IOPS=21.7k, BW=84.9MiB/s (89.1MB/s)(160GiB/1928858msec)
    clat (nsec): min=1040, max=602104k, avg=44325.67, stdev=1608935.76
     lat (nsec): min=1390, max=602105k, avg=44692.57, stdev=1608936.85
    clat percentiles (nsec):
     |  1.00th=[    1128],  5.00th=[    1144], 10.00th=[    1176],
     | 20.00th=[    1208], 30.00th=[    1240], 40.00th=[    1288],
     | 50.00th=[    1400], 60.00th=[    2040], 70.00th=[    3824],
     | 80.00th=[    3984], 90.00th=[    4256], 95.00th=[    4640],
     | 99.00th=[    6560], 99.50th=[   11712], 99.90th=[ 7897088],
     | 99.95th=[34340864], 99.99th=[78118912]
   bw (  KiB/s): min= 4096, max=143360, per=99.99%, avg=86968.07, stdev=17899.79, samples=3857
   iops        : min= 1024, max=35840, avg=21742.01, stdev=4474.94, samples=3857
  lat (usec)   : 2=59.88%, 4=20.96%, 10=18.54%, 20=0.44%, 50=0.05%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.02%, 50=0.04%
  lat (msec)   : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=1.75%, sys=8.46%, ctx=51940, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=41943040,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=84.9MiB/s (89.1MB/s), 84.9MiB/s-84.9MiB/s (89.1MB/s-89.1MB/s), io=160GiB (172GB), run=1928858-1928858msec

Disk stats (read/write):
  vdc: ios=171695/7, merge=1/0, ticks=19105195/1381, in_queue=19020794, util=10.11%

Sequential Read, 4m Blocksize

  • READ: bw=74.7MiB/s
  • iops: avg=18.65

fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4m

[root@centos-stream ~]# fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4m
readtest: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=76.1MiB/s,w=0KiB/s][r=19,w=0 IOPS][eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=10007: Sun Feb  9 10:39:59 2020
   read: IOPS=18, BW=74.7MiB/s (78.3MB/s)(160GiB/2193474msec)
    clat (usec): min=425, max=672398, avg=53545.47, stdev=63278.39
     lat (usec): min=425, max=672398, avg=53546.45, stdev=63278.33
    clat percentiles (usec):
     |  1.00th=[   515],  5.00th=[   775], 10.00th=[  1336], 20.00th=[  2089],
     | 30.00th=[  2835], 40.00th=[  4359], 50.00th=[ 24773], 60.00th=[ 49021],
     | 70.00th=[ 79168], 80.00th=[111674], 90.00th=[147850], 95.00th=[177210],
     | 99.00th=[233833], 99.50th=[263193], 99.90th=[320865], 99.95th=[354419],
     | 99.99th=[492831]
   bw (  KiB/s): min=16351, max=147456, per=99.99%, avg=76480.58, stdev=18909.26, samples=4386
   iops        : min=    3, max=   36, avg=18.65, stdev= 4.61, samples=4386
  lat (usec)   : 500=0.84%, 750=3.94%, 1000=2.20%
  lat (msec)   : 2=11.67%, 4=20.31%, 10=4.90%, 20=4.20%, 50=12.20%
  lat (msec)   : 100=15.90%, 250=23.17%, 500=0.65%, 750=0.01%
  cpu          : usr=0.02%, sys=4.33%, ctx=41070, majf=0, minf=524
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40960,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=74.7MiB/s (78.3MB/s), 74.7MiB/s-74.7MiB/s (78.3MB/s-78.3MB/s), io=160GiB (172GB), run=2193474-2193474msec

Disk stats (read/write):
  vdc: ios=174951/0, merge=17/0, ticks=21141738/0, in_queue=21054323, util=9.14%

Sequential Write, 4k Blocksize

  • WRITE: bw=31.4MiB/s
  • iops: avg=8041.64

fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k

[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
writetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
writetest: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=30.3MiB/s][r=0,w=7750 IOPS][eta 00m:00s]    
writetest: (groupid=0, jobs=1): err= 0: pid=505: Sat Feb  8 19:15:35 2020
  write: IOPS=8041, BW=31.4MiB/s (32.9MB/s)(160GiB/5215542msec)
    clat (usec): min=2, max=101237, avg=122.62, stdev=1027.96
     lat (usec): min=2, max=101237, avg=123.03, stdev=1027.96
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    4], 80.00th=[    6], 90.00th=[    7], 95.00th=[    8],
     | 99.00th=[ 6783], 99.50th=[ 9372], 99.90th=[11600], 99.95th=[11600],
     | 99.99th=[11600]
   bw (  KiB/s): min= 8976, max=612672, per=100.00%, avg=32166.57, stdev=12683.70, samples=10431
   iops        : min= 2244, max=153168, avg=8041.64, stdev=3170.93, samples=10431
  lat (usec)   : 4=70.58%, 10=26.51%, 20=1.40%, 50=0.11%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=1.09%, 20=0.29%, 50=0.01%
  lat (msec)   : 250=0.01%
  cpu          : usr=0.63%, sys=3.84%, ctx=579301, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,41943040,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=31.4MiB/s (32.9MB/s), 31.4MiB/s-31.4MiB/s (32.9MB/s-32.9MB/s), io=160GiB (172GB), run=5215542-5215542msec

Disk stats (read/write):
  vdc: ios=0/175534, merge=0/4, ticks=0/998233048, in_queue=998145387, util=0.50%

Sequential Write, 4m Blocksize

  • WRITE: bw=57.5MiB/s
  • iops: avg=14.34

fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m

[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
writetest: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=56.1MiB/s][r=0,w=14 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=10351: Sun Feb  9 11:28:38 2020
  write: IOPS=14, BW=57.5MiB/s (60.3MB/s)(160GiB/2851095msec)
    clat (usec): min=1410, max=155763, avg=69470.49, stdev=11483.13
     lat (usec): min=1482, max=155890, avg=69601.41, stdev=11480.40
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   51], 10.00th=[   60], 20.00th=[   63],
     | 30.00th=[   64], 40.00th=[   70], 50.00th=[   70], 60.00th=[   72],
     | 70.00th=[   72], 80.00th=[   81], 90.00th=[   82], 95.00th=[   88],
     | 99.00th=[   90], 99.50th=[  100], 99.90th=[  102], 99.95th=[  110],
     | 99.99th=[  136]
   bw (  KiB/s): min=32768, max=2105344, per=99.99%, avg=58839.67, stdev=29089.37, samples=5702
   iops        : min=    8, max=  514, avg=14.34, stdev= 7.11, samples=5702
  lat (msec)   : 2=0.62%, 4=0.18%, 10=0.01%, 50=4.55%, 100=94.55%
  lat (msec)   : 250=0.10%
  cpu          : usr=0.23%, sys=3.49%, ctx=347036, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,40960,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=57.5MiB/s (60.3MB/s), 57.5MiB/s-57.5MiB/s (60.3MB/s-60.3MB/s), io=160GiB (172GB), run=2851095-2851095msec

Disk stats (read/write):
  vdc: ios=0/177589, merge=0/1, ticks=0/544638715, in_queue=544549875, util=0.91%

Sequential, 80% Read, 20% Write, 4k Blocksize

  • READ: bw=65.0MiB/s
  • iops: avg=16775.73
  • WRITE: bw=16.2MiB/s
  • iops: avg=4194.47

fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k

[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k
read80write20test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=84.0MiB/s,w=21.1MiB/s][r=21.5k,w=5403 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=3708: Sat Feb  8 23:32:33 2020
   read: IOPS=16.6k, BW=65.0MiB/s (68.2MB/s)(128GiB/2016416msec)
    clat (nsec): min=1060, max=6496.7M, avg=56898.90, stdev=2814777.43
     lat (nsec): min=1400, max=6496.7M, avg=57266.32, stdev=2814777.63
    clat percentiles (nsec):
     |  1.00th=[     1128],  5.00th=[     1160], 10.00th=[     1176],
     | 20.00th=[     1224], 30.00th=[     1256], 40.00th=[     1304],
     | 50.00th=[     1400], 60.00th=[     1704], 70.00th=[     3792],
     | 80.00th=[     3984], 90.00th=[     4256], 95.00th=[     4576],
     | 99.00th=[     6624], 99.50th=[    11840], 99.90th=[  7766016],
     | 99.95th=[ 38010880], 99.99th=[102236160]
   bw (  KiB/s): min=   32, max=143360, per=100.00%, avg=67102.92, stdev=29982.08, samples=3999
   iops        : min=    8, max=35840, avg=16775.73, stdev=7495.52, samples=3999
  write: IOPS=4159, BW=16.2MiB/s (17.0MB/s)(31.0GiB/2016416msec)
    clat (nsec): min=1800, max=11521k, avg=4234.74, stdev=11813.18
     lat (usec): min=2, max=11521, avg= 4.64, stdev=11.83
    clat percentiles (nsec):
     |  1.00th=[ 2064],  5.00th=[ 2352], 10.00th=[ 2448], 20.00th=[ 2544],
     | 30.00th=[ 2608], 40.00th=[ 2736], 50.00th=[ 2992], 60.00th=[ 3792],
     | 70.00th=[ 5472], 80.00th=[ 5856], 90.00th=[ 6688], 95.00th=[ 7840],
     | 99.00th=[14144], 99.50th=[18304], 99.90th=[28800], 99.95th=[33536],
     | 99.99th=[44288]
   bw (  KiB/s): min=  784, max=35936, per=100.00%, avg=16777.90, stdev=7500.81, samples=3998
   iops        : min=  196, max= 8984, avg=4194.47, stdev=1875.20, samples=3998
  lat (usec)   : 2=50.45%, 4=26.89%, 10=21.75%, 20=0.67%, 50=0.13%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
  lat (msec)   : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=1.79%, sys=8.47%, ctx=40855, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=33555281,8387759,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=65.0MiB/s (68.2MB/s), 65.0MiB/s-65.0MiB/s (68.2MB/s-68.2MB/s), io=128GiB (137GB), run=2016416-2016416msec
  WRITE: bw=16.2MiB/s (17.0MB/s), 16.2MiB/s-16.2MiB/s (17.0MB/s-17.0MB/s), io=31.0GiB (34.4GB), run=2016416-2016416msec

Disk stats (read/write):
  vdc: ios=133994/34031, merge=0/10, ticks=18502416/22797554, in_queue=41215836, util=8.15%

Sequential, 80% Read, 20% Write, 4m Blocksize

  • READ: bw=55.0MiB/s
  • iops: avg=14.46
  • WRITE: bw=14.2MiB/s
  • iops: avg=4.84

fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m

[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
read80write20test: (g=0): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=80.1MiB/s,w=32.0MiB/s][r=20,w=8 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=10903: Sun Feb  9 12:19:34 2020
   read: IOPS=13, BW=55.0MiB/s (58.7MB/s)(128GiB/2335537msec)
    clat (usec): min=435, max=1555.6k, avg=70791.35, stdev=107394.86
     lat (usec): min=436, max=1555.6k, avg=70792.35, stdev=107394.84
    clat percentiles (usec):
     |  1.00th=[    537],  5.00th=[    758], 10.00th=[   1205],
     | 20.00th=[   1893], 30.00th=[   2704], 40.00th=[   9372],
     | 50.00th=[  35914], 60.00th=[  58983], 70.00th=[  86508],
     | 80.00th=[ 121111], 90.00th=[ 175113], 95.00th=[ 238027],
     | 99.00th=[ 541066], 99.50th=[ 675283], 99.90th=[1027605],
     | 99.95th=[1132463], 99.99th=[1300235]
   bw (  KiB/s): min= 8143, max=131072, per=100.00%, avg=59278.21, stdev=26040.27, samples=4517
   iops        : min=    1, max=   32, avg=14.46, stdev= 6.36, samples=4517
  write: IOPS=3, BW=14.2MiB/s (14.9MB/s)(32.3GiB/2335537msec)
    clat (usec): min=840, max=7833, avg=2477.47, stdev=929.71
     lat (usec): min=885, max=7933, avg=2570.40, stdev=932.81
    clat percentiles (usec):
     |  1.00th=[  996],  5.00th=[ 1369], 10.00th=[ 1450], 20.00th=[ 1582],
     | 30.00th=[ 1778], 40.00th=[ 2024], 50.00th=[ 2311], 60.00th=[ 2638],
     | 70.00th=[ 2966], 80.00th=[ 3326], 90.00th=[ 3785], 95.00th=[ 4178],
     | 99.00th=[ 4817], 99.50th=[ 4948], 99.90th=[ 5604], 99.95th=[ 5800],
     | 99.99th=[ 7832]
   bw (  KiB/s): min= 8159, max=106496, per=100.00%, avg=19894.69, stdev=12748.10, samples=3406
   iops        : min=    1, max=   26, avg= 4.84, stdev= 3.11, samples=3406
  lat (usec)   : 500=0.44%, 750=3.48%, 1000=2.49%
  lat (msec)   : 2=18.60%, 4=23.11%, 10=4.19%, 20=2.71%, 50=10.22%
  lat (msec)   : 100=14.03%, 250=17.16%, 500=2.67%, 750=0.66%, 1000=0.17%
  cpu          : usr=0.05%, sys=3.77%, ctx=32499, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32688,8272,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=55.0MiB/s (58.7MB/s), 55.0MiB/s-55.0MiB/s (58.7MB/s-58.7MB/s), io=128GiB (137GB), run=2335537-2335537msec
  WRITE: bw=14.2MiB/s (14.9MB/s), 14.2MiB/s-14.2MiB/s (14.9MB/s-14.9MB/s), io=32.3GiB (34.7GB), run=2335537-2335537msec

Disk stats (read/write):
  vdc: ios=145237/37789, merge=12/0, ticks=22152096/27634303, in_queue=49694677, util=7.44%

Conclusions

  • As expected, Fedora works fine on the hardware.
  • As expected, a quick Ceph smoke test was fine.
  • The Ceph performance is already a bit better than with ODROID-HC2 OSDs only.

So, apart from budget considerations, I am ready to buy 3 more of these and a 10Gig switch. And then remove the ODROID-HC2 boards from my cluster.

Addendum 2020-02-14

Added three SATA HDD to the F5-422 (1x 1TB, 2x 160G. Those were the last old drives I had on the shelf), removed the old HDD only osd.7, created 4 OSDs each with 1 physical SATA HDD and 1 LV on the SATA SSD.

Volume Groups

[root@f5-422-01 ~]# vgs
  VG             #PV #LV #SN Attr   VSize    VFree   
  VG_Ceph_HDD_00   1   1   0 wz--n-   <1.82t       0 
  VG_Ceph_HDD_01   1   1   0 wz--n- <931.51g       0 
  VG_Ceph_HDD_02   1   1   0 wz--n- <149.05g       0 
  VG_Ceph_HDD_03   1   1   0 wz--n- <149.05g       0 
  VG_Ceph_SSD_01   1   4   0 wz--n- <399.17g <223.17g
  VG_f5-422-01     1   5   0 wz--n-   65.00g   39.00g

Logical Volumes

[root@f5-422-01 ~]# lvs
  LV                 VG             Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  LV_Ceph_HDD_OSD_00 VG_Ceph_HDD_00 -wi-ao----   <1.82t                                                    
  LV_Ceph_HDD_OSD_01 VG_Ceph_HDD_01 -wi-ao---- <931.51g                                                    
  LV_Ceph_HDD_OSD_02 VG_Ceph_HDD_02 -wi-ao---- <149.05g                                                    
  LV_Ceph_HDD_OSD_03 VG_Ceph_HDD_03 -wi-ao---- <149.05g                                                    
  LV_Ceph_SSD_OSD_00 VG_Ceph_SSD_01 -wi-ao----   80.00g                                                    
  LV_Ceph_SSD_OSD_01 VG_Ceph_SSD_01 -wi-ao----   80.00g                                                    
  LV_Ceph_SSD_OSD_02 VG_Ceph_SSD_01 -wi-ao----    8.00g                                                    
  LV_Ceph_SSD_OSD_03 VG_Ceph_SSD_01 -wi-ao----    8.00g                                                    
  LV_home            VG_f5-422-01   -wi-ao----    4.00g                                                    
  LV_root            VG_f5-422-01   -wi-ao----    8.00g                                                    
  LV_swap            VG_f5-422-01   -wi-ao----    4.00g                                                    
  LV_var             VG_f5-422-01   -wi-ao----    8.00g                                                    
  LV_var_log         VG_f5-422-01   -wi-ao----    2.00g                                                    

Physical Volumes

[root@f5-422-01 ~]# pvs
  PV         VG             Fmt  Attr PSize    PFree   
  /dev/sda3  VG_f5-422-01   lvm2 a--    65.00g   39.00g
  /dev/sda4  VG_Ceph_SSD_01 lvm2 a--  <399.17g <223.17g
  /dev/sdb1  VG_Ceph_HDD_01 lvm2 a--  <931.51g       0 
  /dev/sdc1  VG_Ceph_HDD_00 lvm2 a--    <1.82t       0 
  /dev/sdd1  VG_Ceph_HDD_02 lvm2 a--  <149.05g       0 
  /dev/sde1  VG_Ceph_HDD_03 lvm2 a--  <149.05g       0

[root@f5-422-01 ~]# ceph-volume lvm create --data VG_Ceph_HDD_00/LV_Ceph_HDD_OSD_00 --block.db  VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_00 --dmcrypt --bluestore
[root@f5-422-01 ~]# ceph-volume lvm create --data VG_Ceph_HDD_01/LV_Ceph_HDD_OSD_01 --block.db  VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_01 --dmcrypt --bluestore
[root@f5-422-01 ~]# ceph-volume lvm create --data VG_Ceph_HDD_02/LV_Ceph_HDD_OSD_02 --block.db  VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_02 --dmcrypt --bluestore
[root@f5-422-01 ~]# ceph-volume lvm create --data VG_Ceph_HDD_03/LV_Ceph_HDD_OSD_03 --block.db  VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_03 --dmcrypt --bluestore

System load, memory use and io wait were still OK when it was sorting misplaced objects.

FIXME: the above really should used better LV naming. See https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#devices AFAIK the WAL is placed with the DB, if not specified separately. so no need to add --block.wal <wal-device>. https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/administration_guide/osd-bluestore

snapshot of top

top - 17:52:20 up 29 min,  1 user,  load average: 0.67, 0.50, 0.36
Tasks: 201 total,   2 running, 199 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.9 us,  7.0 sy,  0.0 ni, 76.0 id,  8.7 wa,  1.5 hi,  3.8 si,  0.0 st
MiB Mem :  11831.6 total,  10763.2 free,    744.4 used,    324.0 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  10828.4 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                
   6887 ceph      20   0  860548 228292  30420 S  12.3   1.9   1:10.03 ceph-osd                   
   1993 ceph      20   0  858528 210648  29928 S  11.3   1.7   1:40.37 ceph-osd                   
   7498 root       0 -20       0      0      0 I   2.3   0.0   0:03.31 kworker/u9:5-kcryptd/253:17
   4738 root       0 -20       0      0      0 I   2.0   0.0   0:09.87 kworker/u9:0-kcryptd/253:14
   4739 root       0 -20       0      0      0 I   2.0   0.0   0:07.08 kworker/u9:3-kcryptd/253:14
   1732 root       0 -20       0      0      0 I   1.7   0.0   0:13.80 kworker/u9:1-kcryptd/253:14
   1214 root      20   0  269468  28304  13812 S   0.7   0.2   0:01.95 tuned                   
[...]                                                        

ceph osd tree

[root@ceph-mon-01 ~]# ceph osd tree
ID  CLASS WEIGHT   TYPE NAME              STATUS REWEIGHT PRI-AFF 
 -1       13.02634 root default                                   
-17        3.02019     host f5-422-01                             
  7   hdd  1.81929         osd.7              up  1.00000 1.00000 
  8   hdd  0.90970         osd.8              up  1.00000 1.00000 
  9   hdd  0.14560         osd.9              up  1.00000 1.00000 
 10   hdd  0.14560         osd.10             up  1.00000 1.00000 
-13        0.90959     host odroid-hc2-00                         
  6   hdd  0.90959         osd.6              up  1.00000 1.00000 
-11        0.90970     host odroid-hc2-01                         
  0   hdd  0.90970         osd.0              up  1.00000 1.00000 
 -9        0.90970     host odroid-hc2-02                         
  4   hdd  0.90970         osd.4              up  1.00000 1.00000 
-15        1.81929     host odroid-hc2-03                         
  5   hdd  1.81929         osd.5              up  1.00000 1.00000 
 -3        1.81929     host odroid-hc2-04                         
  1   hdd  1.81929         osd.1              up  1.00000 1.00000 
 -5        1.81929     host odroid-hc2-05                         
  2   hdd  1.81929         osd.2              up  1.00000 1.00000 
 -7        1.81929     host odroid-hc2-06                         
  3   hdd  1.81929         osd.3              up  1.00000 1.00000 

ceph-volume lvm list

[root@f5-422-01 ~]# ceph-volume lvm list


====== osd.9 =======

  [block]    /dev/VG_Ceph_HDD_02/LV_Ceph_HDD_OSD_02

      type                      block
      osd id                    9
      cluster fsid              154b75c2-1767-45c5-b7f6-ba7e3a36fc12
      cluster name              ceph
      osd fsid                  ebb82241-6760-421f-9715-577b04046089
      db device                 /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_02
      encrypted                 1
      db uuid                   EmDgfU-hhKz-1Pe5-ua8x-Hsl3-F0TH-ythz10
      cephx lockbox secret      [REDACTED]
      block uuid                ibDiUQ-frjH-TvUt-th4C-tbwB-dYwS-9yIYJP
      block device              /dev/VG_Ceph_HDD_02/LV_Ceph_HDD_OSD_02
      vdo                       0
      crush device class        None
      devices                   /dev/sdd1

  [  db]    /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_02

      type                      db
      osd id                    9
      cluster fsid              154b75c2-1767-45c5-b7f6-ba7e3a36fc12
      cluster name              ceph
      osd fsid                  ebb82241-6760-421f-9715-577b04046089
      db device                 /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_02
      encrypted                 1
      db uuid                   EmDgfU-hhKz-1Pe5-ua8x-Hsl3-F0TH-ythz10
      cephx lockbox secret      [REDACTED]
      block uuid                ibDiUQ-frjH-TvUt-th4C-tbwB-dYwS-9yIYJP
      block device              /dev/VG_Ceph_HDD_02/LV_Ceph_HDD_OSD_02
      vdo                       0
      crush device class        None
      devices                   /dev/sda4

====== osd.8 =======

  [block]    /dev/VG_Ceph_HDD_01/LV_Ceph_HDD_OSD_01

      type                      block
      osd id                    8
      cluster fsid              154b75c2-1767-45c5-b7f6-ba7e3a36fc12
      cluster name              ceph
      osd fsid                  78f5d058-b0b5-4a7b-988f-3f96332b1246
      db device                 /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_01
      encrypted                 1
      db uuid                   q8tPpn-1wSg-ais7-7cac-k5eB-HLd1-UItqkV
      cephx lockbox secret      [REDACTED]
      block uuid                z1rEFf-Xx9B-81FV-sYuG-x21C-S9hW-TN752D
      block device              /dev/VG_Ceph_HDD_01/LV_Ceph_HDD_OSD_01
      vdo                       0
      crush device class        None
      devices                   /dev/sdb1

  [  db]    /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_01

      type                      db
      osd id                    8
      cluster fsid              154b75c2-1767-45c5-b7f6-ba7e3a36fc12
      cluster name              ceph
      osd fsid                  78f5d058-b0b5-4a7b-988f-3f96332b1246
      db device                 /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_01
      encrypted                 1
      db uuid                   q8tPpn-1wSg-ais7-7cac-k5eB-HLd1-UItqkV
      cephx lockbox secret      [REDACTED]
      block uuid                z1rEFf-Xx9B-81FV-sYuG-x21C-S9hW-TN752D
      block device              /dev/VG_Ceph_HDD_01/LV_Ceph_HDD_OSD_01
      vdo                       0
      crush device class        None
      devices                   /dev/sda4

====== osd.7 =======

  [block]    /dev/VG_Ceph_HDD_00/LV_Ceph_HDD_OSD_00

      type                      block
      osd id                    7
      cluster fsid              154b75c2-1767-45c5-b7f6-ba7e3a36fc12
      cluster name              ceph
      osd fsid                  30b5fe84-7b76-4fcb-9eae-af7a375779b0
      db device                 /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_00
      encrypted                 1
      db uuid                   YGRkoK-BgOK-CqU7-GUih-Jc6m-gvjH-qw593H
      cephx lockbox secret      [REDACTED]
      block uuid                kqcXz2-VDHF-KNCu-9kXc-vgeP-doYB-6d6fEs
      block device              /dev/VG_Ceph_HDD_00/LV_Ceph_HDD_OSD_00
      vdo                       0
      crush device class        None
      devices                   /dev/sdc1

  [  db]    /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_00

      type                      db
      osd id                    7
      cluster fsid              154b75c2-1767-45c5-b7f6-ba7e3a36fc12
      cluster name              ceph
      osd fsid                  30b5fe84-7b76-4fcb-9eae-af7a375779b0
      db device                 /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_00
      encrypted                 1
      db uuid                   YGRkoK-BgOK-CqU7-GUih-Jc6m-gvjH-qw593H
      cephx lockbox secret      [REDACTED]
      block uuid                kqcXz2-VDHF-KNCu-9kXc-vgeP-doYB-6d6fEs
      block device              /dev/VG_Ceph_HDD_00/LV_Ceph_HDD_OSD_00
      vdo                       0
      crush device class        None
      devices                   /dev/sda4

====== osd.10 ======

  [block]    /dev/VG_Ceph_HDD_03/LV_Ceph_HDD_OSD_03

      type                      block
      osd id                    10
      cluster fsid              154b75c2-1767-45c5-b7f6-ba7e3a36fc12
      cluster name              ceph
      osd fsid                  9c00d813-7676-4083-a1d4-d64b76f8f4ab
      db device                 /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_03
      encrypted                 1
      db uuid                   jtXDYp-nG6y-bKsp-okzW-5Aj3-zgJQ-5cDWz1
      cephx lockbox secret      [REDACTED]
      block uuid                Jptyt5-RmI5-1KRD-fXaO-h5OH-dqLa-g7f9nN
      block device              /dev/VG_Ceph_HDD_03/LV_Ceph_HDD_OSD_03
      vdo                       0
      crush device class        None
      devices                   /dev/sde1

  [  db]    /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_03

      type                      db
      osd id                    10
      cluster fsid              154b75c2-1767-45c5-b7f6-ba7e3a36fc12
      cluster name              ceph
      osd fsid                  9c00d813-7676-4083-a1d4-d64b76f8f4ab
      db device                 /dev/VG_Ceph_SSD_01/LV_Ceph_SSD_OSD_03
      encrypted                 1
      db uuid                   jtXDYp-nG6y-bKsp-okzW-5Aj3-zgJQ-5cDWz1
      cephx lockbox secret      [REDACTED]
      block uuid                Jptyt5-RmI5-1KRD-fXaO-h5OH-dqLa-g7f9nN
      block device              /dev/VG_Ceph_HDD_03/LV_Ceph_HDD_OSD_03
      vdo                       0
      crush device class        None
      devices                   /dev/sda4

Again, all rados bench tests were run on ceph-mon-01.

Sequential Write Test

Total time run:         600.652078
Total writes made:      11790
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     78.5147
Stddev Bandwidth:       13.7158
Max bandwidth (MB/sec): 116
Min bandwidth (MB/sec): 28
Average IOPS:           19
Stddev IOPS:            3
Max IOPS:               29
Min IOPS:               7
Average Latency(s):     0.815131
Stddev Latency(s):      0.339359
Max latency(s):         2.53383
Min latency(s):         0.21279

rados bench -p testbench 600 write --no-cleanup | tee --append $(date +%F)-rados-test-write

hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_ceph-mon-01.internal.pcfe.net_20282
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0      16        16         0         0         0           -           0
    1      16        26        10   39.9614        40    0.934416     0.70472
    2      16        48        32   63.9641        88    0.506183    0.782824
    3      16        72        56   74.6361        96    0.763491     0.77637
    4      16        93        77   76.9744        84    0.410092    0.760004
    5      16       110        94   75.1785        68    0.427527    0.769987
    6      16       125       109   72.6482        60    0.744231    0.778053
    7      16       145       129   73.6971        80    0.284302    0.793663
    8      16       165       149   74.4837        80    0.468672      0.7996
    9      16       178       162   71.9852        52     1.37299    0.818159
   10      16       195       179   71.5857        68    0.405016    0.853723
   11      16       212       196   71.2591        68    0.420537    0.852709
   12      16       234       218   72.6533        88    0.358446    0.847286
   13      16       247       231   71.0643        52    0.374069    0.843661
   14      16       268       252   71.9876        84    0.473908    0.860645
   15      16       285       269   71.7212        68    0.483529    0.856597
   16      16       301       285   71.2383        64     1.59227    0.856457
   17      16       317       301   70.8121        64    0.442328     0.85968
   18      16       331       315    69.989        56     0.58816     0.88181
   19      16       353       337   70.9364        88      1.3939     0.88399
2020-02-14 22:41:13.556889 min lat: 0.249889 max lat: 2.26579 avg lat: 0.883883
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       369       353    70.589        64    0.312343    0.883883
   21      16       384       368   70.0839        60     1.36119    0.885663
   22      16       401       385   69.9887        68    0.295768    0.888053
   23      16       424       408    70.945        92    0.712418    0.891446
   24      16       437       421   70.1554        52    0.493172    0.887942
   25      16       458       442   70.7088        84    0.698938    0.893216
   26      16       473       457   70.2967        60     0.76582    0.891686
   27      16       495       479    70.952        88    0.556024    0.886044
   28      16       512       496   70.8463        68    0.574363    0.880923
   29      16       531       515   71.0237        76      1.3633    0.887491
   30      16       548       532   70.9227        68    0.569357    0.891555
   31      16       572       556   71.7312        96    0.495083     0.88095
   32      16       587       571   71.3645        60    0.661303    0.881039
   33      16       608       592   71.7471        84    0.289608    0.878793
   34      16       629       613   72.1072        84    0.588759    0.880177
   35      16       648       632   72.2182        76    0.540896    0.874191
   36      16       672       656   72.8785        96    0.648527    0.867963
   37      16       690       674   72.8543        72    0.686326    0.864152
   38      16       712       696   73.2526        88    0.554647    0.864158
   39      16       724       708    72.605        48    0.292921    0.857932
2020-02-14 22:41:33.559446 min lat: 0.247503 max lat: 2.26579 avg lat: 0.862447
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16       730       714   71.3899        24      1.6527    0.862447
   41      16       754       738   71.9898        96     0.93162      0.8824
   42      16       772       756   71.9899        72    0.823795    0.879136
   43      16       794       778    72.362        88     0.72404    0.874639
   44      16       813       797   72.4445        76    0.768016    0.871235
   45      16       835       819   72.7899        88       0.696    0.867428
   46      16       856       840   73.0334        84    0.860067     0.86868
   47      16       873       857   72.9262        68    0.473534    0.866583
   48      16       889       873   72.7401        64     1.48436    0.868305
   49      16       913       897   73.2145        96    0.486223    0.866772
   50      16       929       913   73.0301        64    0.630302    0.863038
   51      16       954       938   73.5587       100    0.732801    0.864161
   52      16       972       956   73.5286        72    0.550323    0.863083
   53      16       994       978   73.8015        88    0.902776    0.860538
   54      16      1012       996   73.7679        72     1.03256     0.86029
   55      16      1032      1016   73.8809        80    0.569524    0.858338
   56      16      1049      1033   73.7758        68      1.4483    0.859223
   57      16      1067      1051   73.7444        72    0.658675     0.85949
   58      16      1091      1075    74.128        96    0.401844    0.857796
   59      16      1113      1097   74.3628        88    0.798294    0.855516
2020-02-14 22:41:53.561836 min lat: 0.223072 max lat: 2.5726 avg lat: 0.852987
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      1134      1118   74.5233        84    0.959303    0.852987
   61      16      1154      1138   74.6129        80    0.744574    0.851366
   62      16      1173      1157   74.6351        76    0.867636    0.850559
   63      16      1192      1176   74.6567        76    0.873523    0.849874
   64      16      1214      1198    74.865        88    0.588496    0.847659
   65      16      1239      1223   75.2515       100     0.79085    0.846227
   66      16      1258      1242   75.2627        76    0.794235    0.843374
   67      16      1279      1263    75.393        84    0.826382    0.843282
   68      16      1298      1282   75.4018        76    0.896996    0.842269
   69      16      1321      1305   75.6422        92    0.673371    0.841708
   70      16      1339      1323     75.59        72    0.437782    0.840853
   71      16      1364      1348   75.9337       100    0.391643    0.838934
   72      16      1378      1362   75.6568        56     0.34302    0.839368
   73      16      1402      1386   75.9353        96    0.350094    0.840248
   74      16      1417      1401   75.7199        60     1.05723    0.839819
   75      16      1440      1424   75.9368        92    0.321802    0.838751
   76      16      1463      1447    76.148        92    0.853417    0.835822
   77      16      1479      1463   75.9902        64    0.913123    0.835253
   78      16      1501      1485    76.144        88    0.824506    0.836111
   79      16      1524      1508   76.3446        92    0.513667    0.834243
2020-02-14 22:42:13.564122 min lat: 0.223072 max lat: 2.5726 avg lat: 0.834991
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      1540      1524   76.1901        64    0.917013    0.834991
   81      16      1557      1541   76.0889        68     1.44861    0.835767
   82      16      1571      1555   75.8438        56    0.345051    0.836941
   83      16      1587      1571   75.7011        64     0.55372    0.838406
   84      16      1608      1592   75.7988        84    0.930096    0.840216
   85      16      1621      1605   75.5187        52     1.38569    0.841744
   86      16      1636      1620   75.3382        60    0.351551    0.841674
   87      16      1656      1640   75.3917        80    0.502591    0.842249
   88      16      1676      1660   75.4439        80    0.671843    0.843994
   89      16      1694      1678   75.4052        72    0.690537    0.844436
   90      16      1711      1695   75.3228        68    0.314688    0.843794
   91      16      1728      1712   75.2422        68    0.284384    0.845079
   92      16      1738      1722   74.8591        40     1.92035    0.845002
   93      16      1749      1733   74.5273        44     2.23773    0.852311
   94      16      1766      1750   74.4577        68     0.27145    0.853821
   95      16      1786      1770    74.516        80    0.688935    0.854932
   96      16      1802      1786   74.4063        64    0.415568    0.854633
   97      16      1819      1803   74.3402        68    0.371044    0.856848
   98      16      1842      1826   74.5203        92    0.440917    0.855608
   99      16      1861      1845   74.5352        76    0.297912    0.854583
2020-02-14 22:42:33.567498 min lat: 0.223072 max lat: 2.86714 avg lat: 0.852756
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      1883      1867   74.6697        88    0.795727    0.852756
  101      16      1905      1889   74.8016        88    0.453008    0.852565
  102      16      1921      1905   74.6957        64     1.09684     0.85211
  103      16      1936      1920   74.5529        60    0.404247     0.85102
  104      16      1955      1939   74.5668        76    0.381559    0.854327
  105      16      1975      1959   74.6184        80    0.333144    0.855177
  106      16      1991      1975   74.5182        64      1.1419    0.854361
  107      16      2011      1995   74.5693        80    0.892053    0.854394
  108      16      2029      2013   74.5455        72    0.879552    0.855386
  109      16      2051      2035   74.6688        88    0.868937    0.853935
  110      16      2074      2058   74.8263        92    0.741671    0.851699
  111      16      2092      2076   74.8007        72     0.25916    0.851671
  112      16      2110      2094   74.7757        72    0.649942     0.85182
  113      16      2135      2119   74.9988       100    0.472506    0.851276
  114      16      2149      2133    74.832        56    0.932414    0.850445
  115      16      2173      2157    75.016        96    0.887637    0.849942
  116      16      2190      2174   74.9554        68    0.399664    0.849994
  117      16      2208      2192   74.9301        72    0.293087    0.850524
  118      16      2227      2211   74.9391        76    0.832736    0.851101
  119      16      2244      2228   74.8807        68     1.51699    0.851084
2020-02-14 22:42:53.569832 min lat: 0.223072 max lat: 2.86714 avg lat: 0.851065
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      2264      2248   74.9233        80    0.741264    0.851065
  121      16      2284      2268   74.9652        80    0.682165    0.851271
  122      16      2301      2285    74.908        68    0.316093     0.85003
  123      16      2323      2307   75.0144        88    0.556807    0.850731
  124      16      2343      2327   75.0545        80    0.522033    0.849193
  125      16      2360      2344    74.998        68    0.541304    0.849782
  126      16      2384      2368   75.1646        96    0.297488    0.848757
  127      16      2399      2383   75.0451        60     1.33596     0.84915
  128      16      2420      2404    75.115        84     0.56683    0.848436
  129      16      2436      2420   75.0288        64     1.37316    0.850204
  130      16      2457      2441   75.0977        84     1.38193    0.849683
  131      16      2476      2460   75.1045        76    0.464722    0.848705
  132      16      2498      2482   75.2021        88     1.15378    0.848154
  133      16      2520      2504   75.2983        88     0.39472    0.846382
  134      16      2536      2520   75.2139        64     1.19732    0.846436
  135      16      2564      2548   75.4863       112     0.72259    0.846479
  136      16      2582      2566   75.4606        72     1.17366    0.845311
  137      16      2594      2578   75.2601        48     1.28357    0.846153
  138      16      2615      2599   75.3233        84     1.31443    0.846796
  139      16      2632      2616   75.2706        68    0.473917    0.846719
2020-02-14 22:43:13.572303 min lat: 0.223072 max lat: 2.86714 avg lat: 0.84659
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      2653      2637   75.3329        84    0.456113     0.84659
  141      16      2673      2657   75.3659        80     1.27685    0.847157
  142      16      2688      2672   75.2576        60    0.392893    0.846492
  143      16      2708      2692   75.2907        80     1.48618     0.84626
  144      16      2723      2707   75.1844        60    0.490787    0.846729
  145      16      2743      2727   75.2176        80    0.836494    0.847327
  146      16      2769      2753   75.4146       104    0.679884    0.846507
  147      16      2786      2770   75.3641        68    0.597976    0.846749
  148      16      2804      2788   75.3413        72    0.517633    0.846419
  149      16      2827      2811   75.4531        92    0.650622    0.845517
  150      16      2845      2829     75.43        72     1.49757    0.845187
  151      16      2863      2847   75.4072        72    0.996583    0.845987
  152      16      2888      2872    75.569       100     0.49177    0.845011
  153      16      2909      2893    75.624        84    0.826943    0.844609
  154      16      2930      2914   75.6779        84    0.614527    0.843205
  155      16      2947      2931   75.6283        68    0.290206    0.842965
  156      16      2964      2948   75.5793        68    0.619257     0.84366
  157      16      2986      2970   75.6584        88    0.506869    0.843539
  158      16      3002      2986   75.5846        64    0.265475     0.84354
  159      16      3025      3009   75.6877        92    0.741666    0.843646
2020-02-14 22:43:33.575695 min lat: 0.223072 max lat: 2.86714 avg lat: 0.843319
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      3042      3026   75.6396        68    0.671463    0.843319
  161      16      3059      3043   75.5921        68    0.379707    0.843159
  162      16      3085      3069   75.7674       104    0.360053    0.843197
  163      16      3103      3087   75.7443        72    0.491305    0.842371
  164      16      3124      3108   75.7945        84    0.861325    0.842085
  165      16      3142      3126   75.7715        72    0.686406    0.841676
  166      16      3164      3148   75.8451        88     0.92492    0.841545
  167      16      3183      3167    75.846        76    0.336833    0.840663
  168      16      3197      3181   75.7278        56    0.297921     0.84153
  169      16      3217      3201    75.753        80    0.384566    0.842507
  170      16      3240      3224   75.8486        92    0.476998    0.842379
  171      16      3260      3244   75.8727        80    0.371012    0.841434
  172      16      3278      3262   75.8502        72    0.547964    0.841449
  173      16      3294      3278   75.7816        64    0.477398    0.841627
  174      16      3317      3301   75.8748        92    0.314621    0.841515
  175      16      3335      3319   75.8526        72    0.885499    0.841042
  176      16      3356      3340   75.8988        84     0.75498    0.840854
  177      16      3370      3354   75.7864        56    0.420594    0.841091
  178      16      3391      3375   75.8324        84     1.23795    0.842172
  179      16      3412      3396    75.878        84    0.651285    0.842088
2020-02-14 22:43:53.578085 min lat: 0.223072 max lat: 2.86714 avg lat: 0.841129
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      3431      3415   75.8786        76     1.23181    0.841129
  181      16      3451      3435   75.9013        80    0.443923    0.841411
  182      16      3472      3456   75.9458        84     1.29829    0.840641
  183      16      3494      3478   76.0116        88    0.411632    0.840323
  184      16      3513      3497   76.0115        76    0.933453    0.839895
  185      16      3536      3520   76.0979        92    0.834135    0.839512
  186      16      3554      3538   76.0758        72    0.843441     0.83922
  187      16      3570      3554   76.0112        64    0.516185    0.838325
  188      16      3591      3575   76.0536        84    0.640002    0.839774
  189      16      3611      3595   76.0744        80    0.342578     0.83935
  190      16      3629      3613    76.053        72     1.01039    0.839876
  191      16      3642      3626    75.927        52    0.450038    0.839891
  192      16      3654      3638   75.7815        48      2.0488    0.840513
  193      16      3675      3659    75.824        84    0.494598    0.842343
  194      16      3701      3685   75.9692       104    0.698002    0.841863
  195      16      3716      3700   75.8873        60    0.476102    0.841201
  196      16      3739      3723   75.9694        92    0.279008     0.84099
  197      16      3757      3741   75.9492        72    0.507472    0.840353
  198      16      3778      3762   75.9898        84    0.882544    0.840057
  199      16      3798      3782   76.0099        80    0.620276    0.840043
2020-02-14 22:44:13.580545 min lat: 0.212625 max lat: 2.86714 avg lat: 0.839361
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      3816      3800   75.9898        72    0.475999    0.839361
  201      16      3836      3820   76.0097        80    0.696399    0.839865
  202      16      3859      3843   76.0888        92    0.871214    0.839276
  203      16      3872      3856   75.9701        52     1.38602    0.839964
  204      16      3894      3878    76.029        88    0.428129    0.840022
  205      16      3912      3896   76.0094        72      0.6636    0.840007
  206      16      3935      3919   76.0864        92    0.729706    0.839551
  207      16      3957      3941    76.144        88    0.585139     0.83898
  208      16      3976      3960   76.1432        76    0.753602    0.838972
  209      16      3993      3977   76.1042        68     1.11383    0.838931
  210      16      4015      3999   76.1608        88    0.391903    0.838574
  211      16      4035      4019    76.179        80     1.09045    0.838659
  212      16      4053      4037   76.1592        72    0.915773    0.838531
  213      16      4074      4058    76.196        84    0.588361    0.837971
  214      16      4092      4076   76.1763        72    0.504845     0.83826
  215      16      4110      4094   76.1569        72    0.460312     0.83834
  216      16      4133      4117   76.2302        92    0.852069    0.837708
  217      16      4144      4128   76.0816        44     1.40087     0.83876
  218      16      4164      4148   76.0996        80     1.28114    0.839468
  219      16      4181      4165   76.0625        68    0.389401    0.839008
2020-02-14 22:44:33.584182 min lat: 0.212625 max lat: 2.86714 avg lat: 0.839435
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      4201      4185   76.0804        80    0.419191    0.839435
  221      16      4219      4203   76.0619        72    0.752501    0.839156
  222      16      4240      4224   76.0976        84    0.490634    0.839171
  223      16      4258      4242   76.0792        72    0.376712    0.839591
  224      16      4277      4261   76.0788        76    0.304429    0.839477
  225      16      4300      4284   76.1495        92    0.669913    0.839032
  226      16      4315      4299    76.078        60     1.41747    0.838542
  227      16      4335      4319   76.0952        80     1.37994    0.839772
  228      16      4345      4329   75.9369        40    0.648106    0.839553
  229      16      4369      4353   76.0245        96    0.777862    0.840525
  230      16      4389      4373   76.0417        80    0.667933    0.840261
  231      16      4410      4394   76.0761        84    0.372537     0.83969
  232      16      4432      4416   76.1275        88    0.512308    0.839019
  233      16      4452      4436   76.1441        80    0.335746    0.838827
  234      16      4467      4451   76.0751        60    0.624963      0.8389
  235      16      4485      4469   76.0577        72    0.328452    0.838649
  236      16      4505      4489   76.0742        80    0.706252    0.839467
  237      16      4528      4512   76.1413        92    0.898602    0.839011
  238      16      4547      4531   76.1407        76    0.802389    0.838821
  239      16      4568      4552   76.1735        84    0.869376    0.838655
2020-02-14 22:44:53.587037 min lat: 0.212625 max lat: 2.86714 avg lat: 0.838695
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      4575      4559   75.9728        28     1.77838    0.838695
  241      16      4604      4588   76.1388       116    0.494749    0.839428
  242      16      4617      4601   76.0391        52     1.34318    0.839286
  243      16      4635      4619   76.0224        72     1.22263    0.840278
  244      16      4651      4635   75.9731        64     1.73266    0.840841
  245      16      4663      4647   75.8589        48    0.894494    0.841124
  246      16      4688      4672    75.957       100    0.828727    0.841557
  247      16      4698      4682   75.8114        40    0.855654    0.841427
  248      16      4723      4707   75.9089       100    0.358043    0.841654
  249      16      4746      4730   75.9735        92    0.638163    0.841106
  250      16      4767      4751   76.0056        84     1.12893    0.840728
  251      16      4786      4770   76.0055        76     1.28614    0.840777
  252      16      4806      4790   76.0213        80    0.886302    0.840207
  253      16      4829      4813   76.0844        92    0.786548    0.839753
  254      16      4847      4831    76.068        72     1.04593    0.840361
  255      16      4860      4844   75.9736        52    0.441618    0.840384
  256      16      4876      4860   75.9268        64    0.433653    0.840757
  257      16      4894      4878   75.9114        72     1.25133    0.841587
  258      16      4913      4897   75.9117        76    0.372412    0.841376
  259      16      4930      4914   75.8812        68     1.23798    0.842325
2020-02-14 22:45:13.590482 min lat: 0.212625 max lat: 2.86714 avg lat: 0.842371
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      4949      4933   75.8816        76    0.391742    0.842371
  261      16      4970      4954   75.9126        84    0.579967    0.841828
  262      16      4992      4976   75.9587        88    0.830126    0.841368
  263      16      5013      4997   75.9893        84    0.417882    0.841165
  264      16      5031      5015   75.9741        72    0.891979    0.840782
  265      16      5052      5036   76.0044        84    0.702783    0.840884
  266      16      5071      5055   76.0043        76    0.394352    0.840849
  267      16      5091      5075   76.0192        80    0.463335    0.840202
  268      16      5112      5096    76.049        84    0.702678    0.839889
  269      16      5132      5116   76.0636        80     1.19854    0.839907
  270      16      5143      5127   75.9449        44    0.577941    0.839567
  271      16      5162      5146   75.9451        76    0.788984    0.841106
  272      16      5181      5165   75.9452        76    0.905767    0.841102
  273      16      5204      5188    76.004        92    0.626781    0.840968
  274      16      5215      5199   75.8872        44    0.881823    0.841126
  275      16      5237      5221   75.9312        88    0.576911    0.841937
  276      16      5260      5244   75.9894        92     1.02934    0.841472
  277      16      5280      5264   76.0038        80     0.88656    0.841138
  278      16      5299      5283   76.0037        76    0.431861    0.840377
  279      16      5319      5303    76.018        80    0.730097    0.840368
2020-02-14 22:45:33.592903 min lat: 0.212625 max lat: 2.86714 avg lat: 0.840195
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      5342      5326   76.0751        92    0.825522    0.840195
  281      16      5362      5346    76.089        80     1.47364    0.839971
  282      16      5384      5368   76.1312        88    0.718996    0.839561
  283      16      5407      5391   76.1873        92    0.455541    0.838959
  284      16      5426      5410   76.1866        76    0.339565    0.838931
  285      16      5445      5429   76.1859        76    0.997768    0.838967
  286      16      5459      5443   76.1153        56      0.6453    0.839156
  287      16      5481      5465   76.1566        88    0.339332    0.839123
  288      16      5501      5485   76.1699        80     0.54045    0.838988
  289      16      5520      5504   76.1693        76    0.608298    0.838836
  290      16      5534      5518   76.0997        56    0.521014     0.83875
  291      16      5558      5542   76.1681        96    0.694046    0.838985
  292      16      5575      5559   76.1401        68     1.35632    0.839131
  293      16      5599      5583   76.2078        96     1.10081    0.838568
  294      16      5612      5596   76.1255        52    0.765025    0.838712
  295      16      5631      5615    76.125        76    0.899501    0.839244
  296      16      5647      5631    76.084        64    0.467947    0.839035
  297      16      5671      5655    76.151        96     0.30348    0.839383
  298      16      5689      5673   76.1369        72    0.758561    0.839404
  299      16      5712      5696   76.1899        92    0.551772    0.838726
2020-02-14 22:45:53.596067 min lat: 0.212625 max lat: 2.86714 avg lat: 0.838389
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      5737      5721   76.2692       100    0.704915    0.838389
  301      16      5758      5742   76.2949        84    0.785669    0.837782
  302      16      5781      5765   76.3469        92    0.429183    0.837098
  303      16      5797      5781   76.3061        64    0.494189    0.837037
  304      16      5822      5806    76.384       100    0.708051    0.836388
  305      16      5835      5819    76.304        52    0.440783    0.836516
  306      16      5844      5828   76.1723        36     2.03004    0.837712
  307      16      5859      5843   76.1196        60    0.644313    0.839269
  308      16      5879      5863   76.1322        80    0.724387    0.839088
  309      16      5897      5881   76.1188        72    0.813836    0.839241
  310      16      5917      5901   76.1313        80    0.748287    0.839477
  311      16      5939      5923   76.1694        88    0.301734    0.839049
  312      16      5960      5944   76.1945        84     0.57288    0.838689
  313      16      5976      5960   76.1555        64    0.255404    0.838917
  314      16      5983      5967   76.0021        28    0.373497    0.838771
  315      16      6001      5985   75.9894        72    0.527997    0.840387
  316      16      6020      6004   75.9894        76    0.387321    0.840964
  317      16      6035      6019   75.9389        60    0.589032    0.840863
  318      16      6051      6035   75.9014        64      1.5921    0.842122
  319      16      6062      6046   75.8014        44     1.75864    0.842892
2020-02-14 22:46:13.598325 min lat: 0.212625 max lat: 2.86714 avg lat: 0.843093
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16      6075      6059    75.727        52    0.510625    0.843093
  321      16      6095      6079   75.7402        80     1.33221    0.843825
  322      16      6112      6096   75.7162        68    0.531465    0.843635
  323      16      6134      6118   75.7542        88    0.583068    0.843745
  324      16      6152      6136   75.7425        72    0.402024    0.843517
  325      16      6175      6159   75.7925        92    0.443608    0.843279
  326      16      6192      6176   75.7686        68    0.368951    0.842888
  327      16      6214      6198    75.806        88    0.594562     0.84354
  328      16      6228      6212   75.7456        56    0.412328    0.843309
  329      16      6251      6235    75.795        92    0.603614    0.843651
  330      16      6267      6251   75.7592        64    0.339706    0.843608
  331      16      6282      6266   75.7116        60    0.653862     0.84355
  332      16      6297      6281   75.6642        60     1.66479       0.844
  333      16      6318      6302   75.6892        84    0.658443    0.844422
  334      16      6333      6317   75.6422        60    0.712221    0.844198
  335      16      6354      6338   75.6672        84    0.392954    0.844561
  336      16      6371      6355   75.6443        68     0.43528    0.844609
  337      16      6392      6376   75.6691        84    0.883243    0.844805
  338      16      6404      6388   75.5872        48     1.66772    0.845132
  339      16      6419      6403   75.5412        60      0.9948    0.845651
2020-02-14 22:46:33.600618 min lat: 0.212625 max lat: 2.86714 avg lat: 0.845656
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16      6440      6424   75.5661        84    0.280988    0.845656
  341      16      6460      6444    75.579        80     1.12206    0.845706
  342      16      6476      6460   75.5452        64     1.01699    0.845659
  343      16      6496      6480   75.5581        80     1.05166    0.845939
  344      16      6512      6496   75.5245        64    0.784998    0.846013
  345      16      6536      6520   75.5838        96    0.910466    0.846027
  346      16      6549      6533   75.5156        52    0.561443    0.845922
  347      16      6569      6553   75.5285        80     1.53521    0.846466
  348      16      6588      6572   75.5299        76    0.852849    0.846206
  349      16      6610      6594   75.5656        88    0.747423    0.846219
  350      16      6629      6613   75.5668        76     1.00139    0.846136
  351      16      6654      6638   75.6363       100    0.330358     0.84538
  352      16      6668      6652   75.5805        56     1.05996    0.845172
  353      16      6689      6673   75.6044        84       0.453    0.845399
  354      16      6710      6694    75.628        84    0.978247    0.845216
  355      16      6731      6715   75.6516        84    0.449951    0.844757
  356      16      6754      6738   75.6975        92    0.999364    0.844656
  357      16      6769      6753   75.6535        60    0.257063    0.844173
  358      16      6793      6777   75.7103        96    0.643254     0.84433
  359      16      6818      6802   75.7779       100    0.630813    0.843743
2020-02-14 22:46:53.603057 min lat: 0.212625 max lat: 2.86714 avg lat: 0.843784
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16      6834      6818   75.7452        64    0.509345    0.843784
  361      16      6854      6838   75.7569        80    0.628862    0.844216
  362      16      6876      6860   75.7907        88    0.991614    0.843604
  363      16      6902      6886   75.8684       104     1.01746    0.842847
  364      16      6923      6907   75.8907        84    0.434695    0.842675
  365      16      6940      6924   75.8691        68    0.863234    0.842366
  366      16      6959      6943   75.8694        76    0.899639      0.8426
  367      16      6974      6958   75.8262        60    0.335408    0.842511
  368      16      6993      6977   75.8266        76    0.827961    0.843044
  369      16      7017      7001   75.8812        96    0.655636    0.842712
  370      16      7037      7021   75.8924        80    0.418677    0.842272
  371      16      7059      7043    75.925        88    0.827429    0.842055
  372      16      7078      7062   75.9251        76    0.624113    0.841904
  373      16      7096      7080   75.9146        72    0.276808    0.841758
  374      16      7116      7100   75.9255        80    0.334143     0.84183
  375      16      7137      7121    75.947        84    0.463178    0.841999
  376      16      7157      7141   75.9578        80    0.679909    0.841811
  377      16      7181      7165   76.0109        96    0.716504    0.841136
  378      16      7195      7179   75.9579        56    0.834678    0.841002
  379      16      7214      7198    75.958        76    0.645061    0.841464
2020-02-14 22:47:13.605469 min lat: 0.212625 max lat: 2.86714 avg lat: 0.840743
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16      7240      7224   76.0318       104    0.512115    0.840743
  381      16      7264      7248   76.0841        96    0.341694     0.84039
  382      16      7287      7271   76.1258        92     0.44678    0.840084
  383      16      7305      7289    76.115        72     1.15374    0.840066
  384      16      7316      7300   76.0313        44    0.719197    0.840029
  385      16      7339      7323   76.0728        92    0.342887    0.840672
  386      16      7354      7338   76.0311        60     1.17534    0.840714
  387      16      7376      7360    76.062        88    0.434123    0.840524
  388      16      7397      7381   76.0824        84    0.470167    0.840056
  389      16      7418      7402   76.1028        84    0.554076    0.839718
  390      16      7442      7426   76.1538        96    0.508361    0.839625
  391      16      7460      7444   76.1431        72    0.387758    0.839367
  392      16      7477      7461   76.1223        68    0.384375    0.839285
  393      16      7493      7477   76.0915        64     1.37253    0.839787
  394      16      7520      7504   76.1724       108    0.262682     0.83959
  395      16      7535      7519   76.1315        60    0.985886     0.83948
  396      16      7549      7533   76.0806        56     1.56169     0.83965
  397      16      7562      7546   76.0199        52     1.49123    0.841198
  398      16      7577      7561   75.9797        60    0.629233    0.841186
  399      16      7600      7584   76.0198        92    0.863336    0.841071
2020-02-14 22:47:33.607833 min lat: 0.212625 max lat: 2.86714 avg lat: 0.841177
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16      7611      7595   75.9397        44     1.81077    0.841177
  401      16      7630      7614   75.9399        76    0.721655     0.84179
  402      16      7649      7633     75.94        76      0.4543    0.841562
  403      16      7668      7652   75.9401        76    0.297391    0.841552
  404      16      7686      7670   75.9303        72    0.499471    0.841649
  405      16      7702      7686   75.9009        64    0.612848    0.842062
  406      16      7717      7701   75.8617        60     1.33675    0.842256
  407      16      7734      7718   75.8423        68    0.516045    0.842413
  408      16      7756      7740   75.8721        88       1.272     0.84252
  409      16      7776      7760   75.8822        80    0.536753    0.842253
  410      16      7802      7786   75.9507       104    0.358715    0.841835
  411      16      7823      7807   75.9703        84    0.989766    0.841568
  412      16      7842      7826   75.9703        76     1.33159    0.841513
  413      16      7856      7840    75.922        56    0.339558    0.841386
  414      16      7874      7858   75.9125        72     1.38033    0.841934
  415      16      7892      7876    75.903        72    0.339063    0.842166
  416      16      7907      7891   75.8648        60    0.407453    0.842302
  417      16      7928      7912   75.8843        84    0.607168    0.842308
  418      16      7944      7928   75.8558        64    0.484209    0.841998
  419      16      7959      7943    75.818        60    0.357573    0.842675
2020-02-14 22:47:53.610169 min lat: 0.212625 max lat: 2.86714 avg lat: 0.843013
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16      7974      7958   75.7803        60    0.399992    0.843013
  421      16      7990      7974   75.7523        64    0.436496    0.842993
  422      16      8014      7998   75.8003        96    0.271399    0.843754
  423      16      8037      8021   75.8385        92    0.542647    0.843353
  424      16      8055      8039   75.8295        72    0.847937    0.842989
  425      16      8077      8061   75.8581        88    0.444809    0.843029
  426      16      8094      8078   75.8396        68    0.372791      0.8428
  427      16      8106      8090   75.7744        48     0.43486    0.842745
  428      16      8122      8106   75.7469        64    0.567038    0.843194
  429      16      8149      8133    75.822       108    0.859567    0.843703
  430      16      8170      8154    75.841        84    0.928082    0.843454
  431      16      8185      8169   75.8042        60     1.08726    0.843415
  432      16      8199      8183   75.7584        56     0.42473    0.843196
  433      16      8219      8203   75.7681        80     1.23468    0.843669
  434      16      8236      8220   75.7502        68    0.523934    0.843578
  435      16      8258      8242   75.7784        88    0.794599    0.843634
  436      16      8283      8267   75.8339       100    0.815178    0.843314
  437      16      8293      8277   75.7519        40    0.400676    0.843102
  438      16      8311      8295   75.7433        72    0.464273    0.843948
  439      16      8331      8315    75.753        80      1.0573    0.844153
2020-02-14 22:48:13.612601 min lat: 0.212625 max lat: 2.86714 avg lat: 0.843745
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16      8351      8335   75.7626        80    0.615956    0.843745
  441      16      8365      8349   75.7178        56    0.407184      0.8441
  442      16      8387      8371   75.7455        88     1.13163    0.844343
  443      16      8407      8391   75.7551        80    0.680602    0.843787
  444      16      8424      8408   75.7376        68      1.1486    0.844123
  445      16      8443      8427   75.7382        76    0.997138    0.844205
  446      16      8465      8449   75.7656        88    0.391728    0.843891
  447      16      8489      8473   75.8109        96    0.633176    0.843387
  448      16      8499      8483    75.731        40    0.301932     0.84342
  449      16      8520      8504   75.7493        84    0.272483    0.844305
  450      16      8537      8521   75.7321        68    0.454225    0.844124
  451      16      8561      8545    75.777        96    0.637149    0.843772
  452      16      8583      8567    75.804        88     0.62366    0.843568
  453      16      8607      8591   75.8486        96    0.407124    0.843282
  454      16      8623      8607   75.8225        64    0.637014    0.843114
  455      16      8649      8633   75.8844       104    0.931033    0.842717
  456      16      8662      8646    75.832        52    0.900576    0.842646
  457      16      8678      8662   75.8061        64    0.336761    0.842805
  458      16      8697      8681   75.8065        76    0.281588    0.843104
  459      16      8717      8701   75.8156        80    0.256536    0.842813
2020-02-14 22:48:33.615049 min lat: 0.212625 max lat: 2.86714 avg lat: 0.842962
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16      8741      8725   75.8595        96    0.683358    0.842962
  461      16      8761      8745   75.8684        80    0.893701    0.842652
  462      16      8782      8766    75.886        84    0.471468    0.842877
  463      16      8801      8785   75.8862        76    0.530747    0.842645
  464      16      8823      8807   75.9123        88     0.53591    0.842144
  465      16      8848      8832   75.9641       100    0.518616    0.842071
  466      16      8867      8851   75.9642        76    0.801448    0.841763
  467      16      8889      8873   75.9899        88    0.392658    0.841542
  468      16      8909      8893   75.9984        80    0.950495    0.841361
  469      16      8927      8911   75.9899        72    0.585551    0.841366
  470      16      8948      8932   76.0069        84    0.437067     0.84104
  471      16      8967      8951   76.0069        76     1.06616    0.841255
  472      16      8988      8972   76.0238        84    0.425995    0.840832
  473      16      9008      8992   76.0322        80     0.68746    0.840736
  474      16      9032      9016   76.0743        96     1.10766    0.840699
  475      16      9052      9036   76.0825        80     0.47195    0.840593
  476      16      9068      9052   76.0571        64    0.418375    0.840568
  477      16      9086      9070   76.0486        72     1.48098    0.840785
  478      16      9104      9088   76.0401        72     1.25832    0.840848
  479      16      9126      9110    76.065        88    0.491488    0.840799
2020-02-14 22:48:53.617620 min lat: 0.212625 max lat: 2.86714 avg lat: 0.840342
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16      9151      9135   76.1149       100    0.698806    0.840342
  481      16      9168      9152    76.098        68     0.58216    0.840193
  482      16      9188      9172   76.1061        80    0.560368    0.840225
  483      16      9207      9191   76.1058        76    0.381203    0.839948
  484      16      9225      9209   76.0973        72    0.743648    0.840094
  485      16      9244      9228   76.0971        76    0.432193    0.840173
  486      16      9265      9249   76.1134        84     0.48358    0.840069
  487      16      9284      9268   76.1131        76    0.451657    0.840074
  488      16      9299      9283   76.0801        60     1.19697    0.840331
  489      16      9319      9303   76.0881        80    0.685949    0.840382
  490      16      9340      9324   76.1042        84     1.00618     0.84031
  491      16      9357      9341   76.0877        68       1.326     0.84027
  492      16      9378      9362   76.1037        84    0.341335     0.84042
  493      16      9397      9381   76.1035        76    0.479356    0.840153
  494      16      9420      9404   76.1357        92    0.653924    0.839953
  495      16      9441      9425   76.1515        84    0.599013    0.839917
  496      16      9460      9444   76.1512        76    0.856523    0.839647
  497      16      9479      9463   76.1508        76    0.731891     0.83966
  498      16      9498      9482   76.1505        76     0.80608    0.839343
  499      16      9522      9506   76.1902        96    0.641915    0.839182
2020-02-14 22:49:13.620631 min lat: 0.212625 max lat: 2.86714 avg lat: 0.838865
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16      9544      9528   76.2138        88    0.552168    0.838865
  501      16      9561      9545   76.1974        68     1.46231    0.838976
  502      16      9587      9571   76.2527       104    0.853431     0.83882
  503      16      9608      9592   76.2681        84      0.3187     0.83858
  504      16      9626      9610   76.2596        72    0.427209    0.838259
  505      16      9646      9630    76.267        80     1.44886      0.8386
  506      16      9661      9645   76.2349        60    0.894211    0.838683
  507      16      9676      9660   76.2028        60    0.732685    0.838561
  508      16      9697      9681   76.2181        84    0.523891    0.838956
  509      16      9717      9701   76.2256        80    0.606057     0.83892
  510      16      9737      9721   76.2329        80    0.495583    0.838771
  511      16      9755      9739   76.2246        72    0.638669    0.838589
  512      16      9776      9760   76.2398        84     0.98839    0.838794
  513      16      9798      9782   76.2627        88    0.460856    0.838662
  514      16      9817      9801   76.2622        76    0.904328    0.838565
  515      16      9834      9818   76.2461        68    0.515633    0.838494
  516      16      9858      9842   76.2844        96    0.672811    0.838266
  517      16      9880      9864    76.307        88    0.511715    0.838065
  518      16      9901      9885   76.3219        84    0.462689    0.837795
  519      16      9920      9904   76.3212        76    0.618191    0.837672
2020-02-14 22:49:33.623101 min lat: 0.212625 max lat: 2.86714 avg lat: 0.837568
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16      9940      9924   76.3283        80    0.575968    0.837568
  521      16      9963      9947   76.3581        92     1.25087    0.837546
  522      16      9986      9970   76.3881        92    0.673702    0.837267
  523      16     10004      9988   76.3797        72    0.747442     0.83711
  524      16     10025     10009   76.3942        84    0.724471    0.836893
  525      16     10048     10032   76.4239        92    0.836856    0.836764
  526      16     10071     10055   76.4535        92    0.638493    0.836527
  527      16     10092     10076   76.4678        84    0.580833    0.836362
  528      16     10113     10097    76.482        84    0.916029    0.836109
  529      16     10125     10109   76.4282        48    0.319385    0.835826
  530      16     10141     10125   76.4047        64    0.295395    0.836421
  531      16     10163     10147   76.4266        88    0.254517    0.836641
  532      16     10176     10160   76.3806        52    0.516377    0.836685
  533      16     10197     10181   76.3949        84    0.345804    0.836966
  534      16     10222     10206   76.4391       100    0.843198    0.836562
  535      16     10235     10219   76.3934        52    0.934714    0.836863
  536      16     10256     10240   76.4076        84     0.75833    0.836771
  537      16     10273     10257   76.3919        68    0.509247     0.83682
  538      16     10293     10277   76.3986        80    0.427609     0.83696
  539      16     10307     10291   76.3607        56      1.6499    0.837218
2020-02-14 22:49:53.626796 min lat: 0.212625 max lat: 2.86714 avg lat: 0.837241
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     10331     10315   76.3971        96    0.799426    0.837241
  541      16     10352     10336   76.4111        84     1.18009    0.836969
  542      16     10369     10353   76.3956        68    0.841291    0.836936
  543      16     10388     10372   76.3948        76    0.700394    0.836782
  544      16     10410     10394   76.4161        88    0.569031    0.836636
  545      16     10437     10421   76.4741       108    0.767351    0.836383
  546      16     10456     10440   76.4732        76    0.423961    0.836316
  547      16     10475     10459   76.4723        76     1.09299    0.836371
  548      16     10489     10473   76.4349        56     1.18874    0.836428
  549      16     10514     10498   76.4778       100    0.866337    0.836398
  550      16     10532     10516   76.4697        72    0.357434    0.836193
  551      16     10550     10534   76.4616        72    0.474326    0.836106
  552      16     10570     10554   76.4679        80    0.640883    0.836397
  553      16     10588     10572   76.4599        72    0.969547    0.836349
  554      16     10611     10595   76.4879        92    0.332298    0.836141
  555      16     10632     10616   76.5014        84    0.391454    0.835754
  556      16     10654     10638   76.5221        88    0.583742    0.835846
  557      16     10669     10653   76.4924        60     1.04246    0.835872
  558      16     10683     10667   76.4556        56    0.848037    0.836048
  559      16     10697     10681    76.419        56     1.69668    0.836434
2020-02-14 22:50:13.629206 min lat: 0.212625 max lat: 2.86714 avg lat: 0.836778
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     10719     10703   76.4397        88    0.334409    0.836778
  561      16     10734     10718   76.4104        60    0.916444    0.836752
  562      16     10752     10736   76.4025        72    0.512184    0.836799
  563      16     10775     10759   76.4302        92    0.402044    0.836845
  564      16     10796     10780   76.4436        84     1.02154    0.836821
  565      16     10811     10795   76.4145        60    0.512833    0.836755
  566      16     10828     10812   76.3996        68    0.339695    0.836879
  567      16     10849     10833    76.413        84    0.649832    0.836802
  568      16     10870     10854   76.4263        84    0.683415    0.836686
  569      16     10887     10871   76.4115        68     1.32962    0.836689
  570      16     10905     10889   76.4038        72    0.351057    0.836775
  571      16     10930     10914   76.4451       100    0.328477    0.836584
  572      16     10953     10937   76.4722        92    0.627209    0.836351
  573      16     10972     10956   76.4714        76     1.06608     0.83628
  574      16     10990     10974   76.4636        72      1.6052    0.836164
  575      16     11007     10991   76.4489        68     0.43336    0.836122
  576      16     11028     11012    76.462        84     1.04019    0.836395
  577      16     11042     11026   76.4265        56    0.513065    0.836387
  578      16     11067     11051   76.4673       100    0.889956    0.836545
  579      16     11089     11073   76.4872        88    0.477137    0.836189
2020-02-14 22:50:33.631481 min lat: 0.197662 max lat: 2.86714 avg lat: 0.835974
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     11110     11094   76.5001        84    0.498484    0.835974
  581      16     11129     11113   76.4992        76    0.602696    0.835825
  582      16     11152     11136   76.5258        92    0.785831    0.835818
  583      16     11165     11149   76.4838        52    0.564088    0.835714
  584      16     11181     11165   76.4624        64    0.354155    0.835988
  585      16     11194     11178   76.4205        52    0.350508    0.836059
  586      16     11209     11193   76.3925        60    0.340539    0.836568
  587      16     11236     11220   76.4463       108    0.900205    0.836523
  588      16     11255     11239   76.4456        76    0.476522     0.83651
  589      16     11276     11260   76.4584        84    0.350573    0.836479
  590      16     11292     11276   76.4372        64    0.385901    0.836453
  591      16     11307     11291   76.4094        60    0.440175    0.836418
  592      16     11326     11310   76.4087        76    0.764558    0.836915
  593      16     11342     11326   76.3878        64    0.460326    0.836891
  594      16     11360     11344   76.3804        72     1.57908    0.837281
  595      16     11371     11355    76.326        44    0.312726     0.83699
  596      16     11393     11377   76.3455        88    0.870125    0.837722
  597      16     11410     11394   76.3315        68     1.67967    0.838146
  598      16     11427     11411   76.3176        68     1.02629    0.838155
  599      16     11439     11423   76.2703        48     0.39205    0.837914
2020-02-14 22:50:53.633829 min lat: 0.197662 max lat: 2.86714 avg lat: 0.8383
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      16     11461     11445   76.2898        88    0.395209      0.8383
Total time run:         600.568810
Total writes made:      11462
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     76.341
Stddev Bandwidth:       14.5922
Max bandwidth (MB/sec): 116
Min bandwidth (MB/sec): 24
Average IOPS:           19
Stddev IOPS:            3
Max IOPS:               29
Min IOPS:               6
Average Latency(s):     0.838328
Stddev Latency(s):      0.38524
Max latency(s):         2.86714
Min latency(s):         0.197662

Sequential Read Test

Total time run:       281.951843
Total reads made:     11790
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   167.263
Average IOPS:         41
Stddev IOPS:          8
Max IOPS:             63
Min IOPS:             22
Average Latency(s):   0.382004
Max latency(s):       2.87508
Min latency(s):       0.0520804

rados bench -p testbench 600 seq | tee --append $(date +%F)-rados-test-read-seq

hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0      16        16         0         0         0           -           0
    1      16        57        41   163.762       164    0.129241    0.231886
    2      16        97        81   161.869       160    0.184329    0.292812
    3      16       151       135   179.847       216    0.125822    0.307539
    4      16       198       182   181.878       188      0.9544     0.32508
    5      16       249       233   186.295       204    0.683037    0.320846
    6      16       300       284    189.24       204    0.699804    0.325389
    7      16       346       330   188.489       184    0.302995     0.32117
    8      16       384       368   183.927       152     1.46503    0.322451
    9      16       421       405   179.935       148    0.150797    0.329013
   10      16       461       445    177.94       160     0.37658    0.344374
   11      16       515       499   181.396       216    0.116553    0.345734
   12      16       571       555   184.944       224   0.0848365    0.338793
   13      16       621       605   186.101       200   0.0890021    0.337184
   14      16       664       648   185.086       172    0.311093    0.332959
   15      16       714       698   186.079       200    0.125842    0.332219
   16      16       766       750   187.436       208    0.237308    0.331745
   17      16       814       798   187.702       192   0.0555165    0.331155
   18      16       857       841   186.828       172     1.47429    0.332085
   19      16       910       894   188.151       212    0.123422    0.331608
2020-02-14 22:52:16.780349 min lat: 0.0536279 max lat: 2.31221 avg lat: 0.332673
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       946       930   185.942       144    0.190618    0.332673
   21      16       986       970   184.705       160     1.57822    0.338705
   22      16      1029      1013   184.117       172    0.287843    0.339931
   23      16      1086      1070   186.013       228    0.134183    0.339682
   24      16      1135      1119   186.426       196    0.399202    0.337652
   25      16      1173      1157   185.048       152   0.0534487    0.337519
   26      16      1214      1198   184.238       164    0.273811    0.339897
   27      16      1243      1227   181.711       116   0.0752925     0.34182
   28      16      1269      1253   178.936       104   0.0830936    0.346966
   29      16      1300      1284    177.04       124    0.119312    0.349288
   30      16      1352      1336   178.071       208    0.101779    0.354258
   31      16      1425      1409   181.744       292    0.218366      0.3494
   32      16      1472      1456   181.939       188    0.184445    0.347525
   33      16      1504      1488   180.304       128    0.950406    0.348341
   34      16      1542      1526   179.472       152    0.053534     0.35154
   35      16      1582      1566   178.915       160    0.352144    0.350598
   36      16      1611      1595   177.167       116   0.0530931    0.352296
   37      16      1657      1641   177.351       184    0.115921    0.355825
   38      16      1700      1684   177.209       172   0.0883738    0.355616
   39      16      1728      1712   175.537       112    0.147714    0.357109
2020-02-14 22:52:36.785894 min lat: 0.0530931 max lat: 2.7562 avg lat: 0.358581
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1762      1746   174.549       136    0.118512    0.358581
   41      16      1802      1786   174.193       160    0.215248    0.360302
   42      16      1841      1825   173.759       156    0.135572    0.362869
   43      16      1887      1871   173.997       184    0.227326    0.362961
   44      16      1926      1910   173.587       156    0.210927    0.364243
   45      16      1966      1950   173.285       160    0.114966    0.364115
   46      16      2007      1991   173.083       164   0.0544832    0.364528
   47      16      2038      2022   172.038       124    0.232185    0.366951
   48      16      2081      2065   172.036       172     1.43975    0.368035
   49      16      2126      2110   172.198       180     1.22513    0.368547
   50      16      2171      2155   172.354       180   0.0792847    0.366669
   51      16      2217      2201   172.562       184    0.195685     0.36519
   52      16      2247      2231   171.551       120    0.203033    0.366924
   53      16      2292      2276    171.71       180     1.89268    0.368939
   54      16      2330      2314   171.345       152    0.110699    0.368449
   55      16      2365      2349   170.774       140    0.140208    0.369659
   56      16      2411      2395    171.01       184    0.171001    0.370589
   57      16      2461      2445   171.518       200   0.0850895    0.368717
   58      16      2503      2487   171.457       168    0.145698    0.369726
   59      16      2544      2528    171.33       164    0.223245     0.36941
2020-02-14 22:52:56.794747 min lat: 0.0527629 max lat: 2.7562 avg lat: 0.368929
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      2587      2571   171.341       172    0.106952    0.368929
   61      16      2626      2610   171.089       156    0.176093    0.369504
   62      16      2669      2653   171.104       172    0.113683    0.370376
   63      16      2719      2703   171.562       200    0.279275    0.370493
   64      16      2756      2740   171.194       148    0.378959    0.369076
   65      16      2805      2789   171.575       196    0.172054    0.370365
   66      16      2838      2822   170.975       132    0.205098    0.370206
   67      16      2870      2854   170.333       128     1.84009    0.371148
   68      16      2908      2892   170.063       152     0.11592    0.371699
   69      16      2933      2917   169.048       100    0.529488    0.372645
   70      16      2969      2953    168.69       144    0.117464    0.373865
   71      16      3020      3004   169.187       204   0.0532694    0.374888
   72      16      3059      3043   169.003       156    0.660874    0.374974
   73      16      3112      3096   169.592       212    0.578829    0.373939
   74      16      3151      3135   169.408       156    0.208941    0.374238
   75      16      3195      3179   169.495       176   0.0873402    0.374424
   76      16      3230      3214   169.107       140   0.0862061    0.374801
   77      16      3265      3249   168.728       140     1.30653    0.376305
   78      16      3320      3304   169.385       220    0.106878     0.37544
   79      16      3372      3356   169.874       208    0.170799    0.374659
2020-02-14 22:53:16.797724 min lat: 0.0527051 max lat: 2.93153 avg lat: 0.373459
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      3418      3402    170.05       184   0.0943033    0.373459
   81      16      3469      3453   170.469       204   0.0885231     0.37343
   82      16      3506      3490   170.194       148   0.0960379    0.373806
   83      16      3558      3542   170.648       208   0.0745853    0.372642
   84      16      3590      3574   170.141       128    0.131342    0.373684
   85      16      3627      3611    169.88       148    0.167081    0.373973
   86      16      3666      3650   169.718       156   0.0833341    0.373552
   87      16      3713      3697   169.928       188   0.0763592    0.373354
   88      16      3759      3743   170.088       184    0.370167    0.373698
   89      16      3796      3780   169.839       148    0.126625    0.374097
   90      16      3829      3813   169.419       132    0.152632    0.374099
   91      16      3863      3847   169.051       136     0.10849     0.37519
   92      16      3902      3886   168.909       156    0.647038    0.375851
   93      16      3930      3914   168.297       112   0.0834804    0.377148
   94      16      3966      3950   168.039       144   0.0715648    0.376628
   95      16      4002      3986   167.785       144    0.148089    0.378237
   96      16      4026      4010   167.038        96    0.557916    0.379146
   97      16      4054      4038    166.47       112     2.42487    0.380597
   98      16      4084      4068   165.996       120      0.7027    0.381661
   99      16      4107      4091   165.249        92     2.55571    0.383561
2020-02-14 22:53:36.800824 min lat: 0.052318 max lat: 2.93153 avg lat: 0.384145
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      4145      4129   165.116       152     2.55698    0.384145
  101      16      4172      4156    164.55       108    0.109822    0.386184
  102      16      4209      4193   164.388       148   0.0825119    0.386484
  103      16      4234      4218   163.763       100   0.0544984    0.386711
  104      16      4267      4251   163.457       132    0.118032    0.387648
  105      16      4305      4289   163.348       152     2.48676    0.388984
  106      16      4336      4320   162.977       124   0.0748831    0.389095
  107      16      4380      4364   163.098       176     2.18509    0.389482
  108      16      4422      4406   163.143       168    0.191207    0.390398
  109      16      4461      4445   163.078       156   0.0549118    0.389534
  110      16      4493      4477   162.759       128    0.109156    0.389802
  111      16      4532      4516   162.697       156   0.0811559     0.39023
  112      16      4570      4554   162.602       152   0.0779582    0.390543
  113      16      4621      4605   162.968       204      0.2167    0.390451
  114      16      4664      4648   163.047       172    0.352362    0.390079
  115      16      4714      4698   163.368       200    0.326338    0.389573
  116      16      4748      4732   163.132       136    0.467122    0.389743
  117      16      4763      4747    162.25        60    0.576711    0.390551
  118      16      4802      4786   162.197       156   0.0927156    0.391904
  119      16      4846      4830   162.313       176    0.053048    0.391875
2020-02-14 22:53:56.803423 min lat: 0.0522791 max lat: 3.33647 avg lat: 0.392533
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      4880      4864   162.094       136    0.204178    0.392533
  121      16      4924      4908   162.209       176    0.143887    0.392196
  122      16      4958      4942   161.994       136      0.9467    0.392456
  123      16      5008      4992   162.302       200    0.183683    0.392599
  124      16      5055      5039   162.509       188   0.0553233    0.391997
  125      16      5086      5070   162.201       124   0.0835045    0.392034
  126      16      5129      5113   162.279       172   0.0974787    0.392105
  127      16      5172      5156   162.355       172   0.0898608    0.391403
  128      16      5222      5206   162.649       200    0.114492    0.391675
  129      16      5265      5249   162.721       172     0.18769    0.391338
  130      16      5302      5286   162.608       148    0.125773     0.39083
  131      16      5334      5318   162.343       128     1.33508    0.392092
  132      16      5366      5350   162.083       128     2.01918    0.392181
  133      16      5405      5389   162.037       156    0.176213    0.392289
  134      16      5434      5418   161.694       116    0.171706    0.392735
  135      16      5466      5450   161.444       128    0.173966    0.393531
  136      16      5512      5496    161.61       184     0.14199    0.394112
  137      16      5543      5527   161.335       124     0.16925    0.394203
  138      16      5586      5570   161.412       172    0.158254    0.394102
  139      16      5611      5595    160.97       100    0.218431     0.39481
2020-02-14 22:54:16.806208 min lat: 0.0522791 max lat: 3.33647 avg lat: 0.395208
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      5641      5625   160.677       120    0.379187    0.395208
  141      16      5695      5679    161.07       216    0.837289    0.396139
  142      16      5742      5726   161.259       188    0.149888    0.395531
  143      16      5787      5771    161.39       180   0.0804189    0.394532
  144      16      5826      5810   161.353       156    0.244229    0.394356
  145      16      5865      5849   161.315       156   0.0596529     0.39442
  146      16      5907      5891   161.361       168    0.246574    0.394459
  147      16      5954      5938   161.542       188    0.128342    0.394449
  148      16      5994      5978   161.531       160     2.04655    0.394368
  149      16      6026      6010   161.306       128   0.0537127    0.394071
  150      16      6043      6027   160.684        68     2.22473    0.395402
  151      16      6071      6055   160.362       112   0.0753619    0.396245
  152      16      6107      6091   160.254       144   0.0830832    0.397646
  153      16      6156      6140   160.487       196    0.181396    0.396943
  154      16      6210      6194   160.847       216    0.165789    0.396172
  155      16      6247      6231   160.764       148     1.41704    0.396191
  156      16      6295      6279   160.965       192      0.1358    0.395222
  157      16      6346      6330   161.238       204    0.131862    0.395398
  158      16      6393      6377   161.408       188     1.85512    0.394768
  159      16      6423      6407   161.147       120   0.0604151    0.394635
2020-02-14 22:54:36.809585 min lat: 0.0522791 max lat: 3.33647 avg lat: 0.395652
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      6447      6431   160.739        96     0.20415    0.395652
  161      16      6485      6469   160.685       152    0.389969    0.395521
  162      16      6517      6501   160.482       128    0.082259    0.396498
  163      16      6561      6545   160.578       176     1.56243    0.396758
  164      16      6606      6590   160.696       180    0.448176    0.396284
  165      16      6658      6642   160.982       208    0.257093    0.395946
  166      16      6695      6679   160.904       148    0.145277     0.39643
  167      16      6751      6735   161.281       224     0.19578    0.395502
  168      16      6805      6789   161.607       216    0.201983    0.394765
  169      16      6854      6838    161.81       196     0.26439    0.394122
  170      16      6896      6880   161.846       168    0.163156    0.393913
  171      16      6941      6925   161.952       180    0.191756    0.393503
  172      16      6987      6971    162.08       184    0.383641    0.393252
  173      16      7025      7009   162.022       152    0.056629    0.393576
  174      16      7065      7049    162.01       160    0.151143    0.393185
  175      16      7105      7089   161.999       160   0.0823962    0.393213
  176      16      7142      7126   161.919       148    0.164627    0.393362
  177      16      7184      7168   161.953       168    0.162608    0.393676
  178      16      7223      7207    161.92       156    0.122373    0.393718
  179      16      7267      7251   161.998       176    0.183036    0.393734
2020-02-14 22:54:56.813178 min lat: 0.0522791 max lat: 3.33647 avg lat: 0.393105
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      7317      7301   162.209       200    0.192651    0.393105
  181      16      7348      7332   161.998       124    0.456634    0.393201
  182      16      7380      7364   161.811       128    0.228963    0.393637
  183      16      7409      7393   161.561       116   0.0783955    0.393641
  184      16      7458      7442   161.748       196    0.314484    0.394076
  185      16      7508      7492   161.954       200    0.252043    0.393712
  186      16      7549      7533   161.965       164    0.153378    0.393834
  187      16      7583      7567   161.826       136    0.963346    0.393743
  188      16      7606      7590   161.455        92   0.0742535    0.394345
  189      16      7635      7619   161.214       116    0.117178    0.394577
  190      16      7679      7663   161.292       176     2.24898    0.394879
  191      16      7715      7699   161.201       144   0.0540177    0.395185
  192      16      7755      7739   161.195       160   0.0695364     0.39562
  193      16      7792      7776   161.127       148   0.0838847    0.395552
  194      16      7830      7814   161.079       152     2.13427    0.395695
  195      16      7884      7868   161.361       216    0.172549    0.395417
  196      16      7922      7906   161.313       152    0.129746    0.394778
  197      16      7968      7952   161.428       184   0.0835599    0.395084
  198      16      8016      8000   161.582       192    0.449698    0.394556
  199      16      8065      8049   161.755       196    0.299544    0.394278
2020-02-14 22:55:16.815903 min lat: 0.0521896 max lat: 3.33647 avg lat: 0.394235
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      8100      8084   161.646       140   0.0802843    0.394235
  201      16      8133      8117   161.499       132     2.46951    0.394181
  202      16      8169      8153   161.412       144     2.26804    0.394491
  203      16      8206      8190   161.346       148    0.112469    0.394644
  204      16      8244      8228     161.3       152    0.185016    0.395308
  205      16      8291      8275    161.43       188    0.324219    0.395135
  206      16      8330      8314   161.403       156    0.117121      0.3951
  207      16      8373      8357   161.455       172    0.189234    0.394841
  208      16      8403      8387   161.255       120   0.0831808     0.39493
  209      16      8448      8432   161.345       180   0.0805557    0.395204
  210      16      8496      8480   161.491       192    0.291093    0.395131
  211      16      8547      8531   161.691       204    0.135836    0.394552
  212      16      8592      8576   161.778       180   0.0848451    0.394114
  213      16      8624      8608   161.619       128    0.164053    0.394098
  214      16      8649      8633   161.331       100    0.667972     0.39441
  215      16      8693      8677   161.399       176    0.221579    0.394985
  216      16      8745      8729   161.615       208    0.159074    0.394847
  217      16      8786      8770   161.626       164   0.0703386    0.394809
  218      16      8832      8816   161.728       184     1.15555    0.394505
  219      16      8880      8864   161.866       192    0.161807    0.394192
2020-02-14 22:55:36.819200 min lat: 0.0521896 max lat: 3.33647 avg lat: 0.394078
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      8910      8894   161.676       120    0.196688    0.394078
  221      16      8939      8923   161.469       116    0.572676    0.394209
  222      16      8988      8972   161.625       196    0.167857    0.394582
  223      16      9034      9018   161.725       184    0.167536    0.394446
  224      16      9078      9062   161.789       176       1.086    0.394315
  225      16      9105      9089   161.549       108    0.598772    0.394075
  226      16      9138      9122   161.419       132    0.132201    0.394585
  227      16      9173      9157   161.324       140     2.24971    0.395072
  228      16      9214      9198   161.336       164   0.0542014    0.395063
  229      16      9256      9240   161.365       168   0.0648941    0.395354
  230      16      9287      9271   161.202       124    0.392404    0.395527
  231      16      9320      9304   161.076       132   0.0854787    0.395517
  232      16      9366      9350   161.174       184   0.0846252    0.395867
  233      16      9407      9391   161.187       164   0.0947143    0.395694
  234      16      9442      9426   161.096       140   0.0752536    0.395713
  235      16      9483      9467   161.108       164     0.36338    0.395553
  236      16      9525      9509   161.137       168    0.055651    0.395424
  237      16      9566      9550   161.149       164      0.1813    0.396044
  238      16      9609      9593   161.195       172   0.0759368     0.39573
  239      16      9654      9638   161.273       180    0.723772    0.395581
2020-02-14 22:55:56.822604 min lat: 0.0521896 max lat: 3.33647 avg lat: 0.395281
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      9698      9682   161.334       176   0.0660823    0.395281
  241      16      9734      9718   161.262       144    0.306338     0.39526
  242      16      9766      9750   161.125       128    0.119929    0.395607
  243      16      9804      9788   161.087       152    0.140319    0.395859
  244      16      9838      9822   160.984       136    0.303154    0.396034
  245      16      9879      9863   160.996       164   0.0816013    0.396404
  246      16      9919      9903   160.992       160    0.149237    0.396281
  247      16      9953      9937   160.891       136    0.267473     0.39654
  248      16     10003      9987   161.048       200    0.361956    0.396246
  249      16     10039     10023    160.98       144    0.358183    0.396181
  250      16     10077     10061   160.944       152    0.145689    0.396173
  251      16     10110     10094   160.829       132     1.91914    0.396246
  252      16     10138     10122   160.635       112    0.103371     0.39651
  253      16     10191     10175   160.838       212    0.253354    0.396717
  254      16     10224     10208   160.724       132     0.13338    0.396788
  255      16     10255     10239    160.58       124    0.786144    0.397227
  256      16     10283     10267    160.39       112    0.304134     0.39737
  257      16     10324     10308   160.404       164     2.21629    0.397522
  258      16     10368     10352   160.464       176    0.194805    0.397428
  259      16     10401     10385   160.354       132    0.142079    0.397412
2020-02-14 22:56:16.825355 min lat: 0.0521896 max lat: 3.33647 avg lat: 0.397512
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16     10434     10418   160.245       132    0.165367    0.397512
  261      16     10472     10456   160.214       152    0.291335    0.397748
  262      16     10514     10498   160.243       168    0.111663    0.397835
  263      16     10550     10534   160.181       144    0.175979    0.398148
  264      16     10579     10563   160.014       116    0.746378    0.398033
  265      16     10615     10599   159.954       144    0.118493    0.398358
  266      16     10642     10626   159.758       108     1.27238    0.399306
  267      16     10683     10667   159.774       164    0.116568    0.399076
  268      16     10717     10701   159.685       136    0.361172    0.398987
  269      16     10742     10726   159.463       100    0.344828    0.399539
  270      16     10785     10769    159.51       172     2.06475    0.399972
  271      16     10832     10816   159.615       188     1.56799    0.399788
  272      16     10884     10868   159.792       208    0.208132     0.39934
  273      16     10933     10917   159.925       196   0.0763828    0.398955
  274      16     10970     10954   159.881       148     1.65138    0.399037
  275      16     11005     10989   159.809       140     0.19707    0.399036
  276      16     11045     11029    159.81       160    0.185064    0.399161
  277      16     11076     11060    159.68       124    0.132775    0.398999
  278      16     11116     11100   159.681       160    0.379721    0.399561
  279      16     11159     11143   159.726       172    0.232735    0.399369
2020-02-14 22:56:36.828003 min lat: 0.0521896 max lat: 3.33647 avg lat: 0.399405
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16     11200     11184   159.741       164    0.111902    0.399405
  281      16     11236     11220   159.685       144   0.0533658    0.399468
  282      16     11280     11264   159.742       176   0.0548492    0.399446
  283      16     11326     11310   159.828       184    0.118763    0.399322
  284      16     11358     11342   159.715       128     2.06554    0.399361
  285      16     11395     11379   159.674       148    0.379412    0.399419
  286      16     11438     11422   159.717       172    0.186498    0.399545
  287       7     11462     11455    159.62       132     1.25933    0.399789
Total time run:       287.165268
Total reads made:     11462
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   159.657
Average IOPS:         39
Stddev IOPS:          8
Max IOPS:             73
Min IOPS:             15
Average Latency(s):   0.400031
Max latency(s):       3.33647
Min latency(s):       0.0521896

Random Read Test

Total time run:       600.851506
Total reads made:     23638
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   157.363
Average IOPS:         39
Stddev IOPS:          7
Max IOPS:             63
Min IOPS:             17
Average Latency(s):   0.406183
Max latency(s):       3.04659
Min latency(s):       0.0372216

rados bench -p testbench 600 rand | tee --append $(date +%F)-rados-test-read-rand

hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0      16        16         0         0         0           -           0
    1      16        60        44   175.852       176    0.123451    0.284632
    2      16        98        82   163.903       152    0.863719    0.297597
    3      16       139       123   163.925       164   0.0834313    0.332324
    4      16       165       149   148.945       104    0.122863     0.37049
    5      16       208       192   153.527       172    0.128988    0.386854
    6      16       250       234   155.867       168    0.150761    0.378545
    7      16       302       286   163.296       208   0.0558394    0.375426
    8      16       345       329    164.38       172     1.35706    0.367942
    9      16       403       387   171.886       232     0.37122    0.361504
   10      16       440       424   169.497       148   0.0809844    0.356292
   11      16       486       470   170.812       184   0.0809962    0.351863
   12      16       529       513    170.91       172    0.150925    0.359943
   13      16       586       570   175.297       228   0.0743829    0.357099
   14      16       629       613    175.06       172   0.0376657    0.352007
   15      16       672       656   174.855       172    0.186661    0.350654
   16      16       704       688   171.926       128    0.435999    0.356971
   17      16       746       730   171.694       168     1.55275    0.362453
   18      16       782       766   170.155       144    0.205478    0.363302
   19      16       825       809   170.251       172    0.164933    0.362713
2020-02-14 22:57:40.315788 min lat: 0.0373792 max lat: 2.35095 avg lat: 0.368081
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       858       842   168.337       132     1.64804    0.368081
   21      16       907       891   169.651       196    0.244302    0.369134
   22      16       948       932   169.393       164   0.0379065    0.367461
   23      16       980       964   167.593       128   0.0777448    0.370489
   24      16      1011       995   165.777       124    0.142145    0.374253
   25      16      1053      1037   165.865       168    0.126759    0.377375
   26      16      1078      1062   163.332       100   0.0824319    0.381133
   27      16      1127      1111   164.541       196    0.131991    0.382737
   28      16      1167      1151   164.378       160    0.321278    0.382344
   29      16      1200      1184   163.261       132   0.0375643    0.380588
   30      16      1248      1232   164.218       192    0.274339    0.378903
   31      16      1289      1273   164.208       164   0.0375938    0.380145
   32      16      1326      1310     163.7       148    0.189664    0.384006
   33      16      1369      1353    163.95       172    0.103673    0.384065
   34      16      1406      1390   163.481       148    0.143072    0.382786
   35      16      1448      1432   163.609       168    0.313507    0.381893
   36      16      1481      1465   162.731       132     0.11719    0.386529
   37      16      1510      1494   161.468       116    0.112235    0.387758
   38      16      1549      1533   161.323       156    0.160397    0.387332
   39      16      1583      1567   160.674       136    0.146137    0.388776
2020-02-14 22:58:00.319211 min lat: 0.0373792 max lat: 3.3282 avg lat: 0.391134
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1622      1606   160.556       156    0.365622    0.391134
   41      16      1669      1653   161.225       188     2.06127    0.392502
   42      16      1706      1690    160.91       148     0.45716    0.391878
   43      16      1737      1721   160.051       124    0.180832    0.392935
   44      16      1774      1758   159.777       148    0.123429    0.394191
   45      16      1815      1799    159.87       164   0.0928125    0.393497
   46      16      1852      1836   159.612       148    0.134454    0.396411
   47      16      1909      1893   161.066       228    0.105589    0.393923
   48      16      1949      1933   161.043       160    0.109947    0.392866
   49      16      1994      1978    161.43       180    0.761015    0.393902
   50      16      2042      2026   162.041       192    0.154462    0.391935
   51      16      2084      2068   162.157       168   0.0373792    0.389744
   52      16      2116      2100     161.5       128     2.10556    0.390347
   53      16      2146      2130   160.717       120    0.230573    0.391941
   54      16      2192      2176   161.147       184    0.189078     0.39364
   55      16      2232      2216   161.126       160     1.12757    0.394596
   56      16      2278      2262   161.534       184    0.370259    0.392672
   57      16      2332      2316   162.489       216    0.164873    0.391194
   58      16      2364      2348   161.894       128     1.65333    0.390781
   59      16      2408      2392   162.133       176    0.273545    0.391056
2020-02-14 22:58:20.321771 min lat: 0.0373792 max lat: 3.3282 avg lat: 0.390399
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      2460      2444   162.897       208    0.187673    0.390399
   61      16      2500      2484   162.849       160    0.118758    0.388975
   62      16      2536      2520   162.543       144    0.133707    0.388743
   63      16      2586      2570   163.137       200    0.432153    0.388444
   64      16      2614      2598   162.338       112     1.94405    0.389075
   65      16      2646      2630    161.81       128      1.9782    0.390416
   66      16      2691      2675   162.085       180     1.96478    0.391036
   67      16      2742      2726    162.71       204    0.221265    0.390446
   68      16      2806      2790   164.081       256    0.246395    0.388779
   69      16      2853      2837   164.428       188   0.0392827     0.38626
   70      16      2890      2874   164.193       148   0.0791218    0.385767
   71      16      2937      2921   164.528       188    0.201076    0.385815
   72      16      2968      2952   163.965       124    0.183829    0.386282
   73      16      3007      2991   163.855       156    0.144999    0.386893
   74      16      3043      3027   163.587       144     2.01545    0.387724
   75      16      3084      3068   163.592       164   0.0382203    0.388117
   76      16      3121      3105   163.384       148    0.796388    0.387997
   77      16      3158      3142   163.184       148    0.158849     0.38875
   78      16      3189      3173   162.682       124    0.529617    0.389487
   79      16      3227      3211   162.546       152    0.151977    0.389479
2020-02-14 22:58:40.325971 min lat: 0.0373792 max lat: 3.3282 avg lat: 0.391062
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      3263      3247   162.314       144    0.144937    0.391062
   81      16      3306      3290   162.434       172    0.204456    0.390011
   82      16      3342      3326   162.209       144     1.52041    0.391783
   83      16      3387      3371   162.423       180   0.0884655    0.391357
   84      16      3439      3423   162.965       208    0.213099     0.39052
   85      16      3480      3464   162.977       164   0.0870987    0.390159
   86      16      3517      3501   162.803       148    0.150084    0.389389
   87      16      3548      3532   162.357       124     2.00885    0.390456
   88      16      3579      3563   161.917       124     1.47279    0.392481
   89      16      3627      3611   162.254       192    0.240604    0.391945
   90      16      3671      3655   162.407       176     1.42438    0.391685
   91      16      3718      3702   162.688       188    0.112953     0.39162
   92      16      3761      3745   162.787       172     1.60579    0.390767
   93      16      3802      3786     162.8       164    0.259297    0.391066
   94      16      3837      3821   162.557       140    0.824924    0.390758
   95      16      3879      3863   162.614       168    0.191065    0.390873
   96      16      3908      3892   162.129       116    0.222965    0.391379
   97      16      3944      3928   161.942       144     1.70787    0.392335
   98      16      3993      3977   162.289       196    0.135496    0.391873
   99      16      4037      4021   162.427       176    0.263083    0.392134
2020-02-14 22:59:00.331314 min lat: 0.0373792 max lat: 3.3282 avg lat: 0.391696
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      4079      4063   162.483       168    0.456687    0.391696
  101      16      4107      4091   161.983       112    0.122295    0.391489
  102      16      4140      4124   161.689       132    0.334466    0.392019
  103      16      4174      4158   161.439       136    0.652835    0.392803
  104      16      4205      4189   161.079       124    0.644701      0.3945
  105      16      4252      4236   161.335       188    0.164368    0.394728
  106      16      4290      4274   161.247       152    0.258417    0.394372
  107      16      4326      4310   161.086       144    0.197363    0.394643
  108      16      4375      4359   161.409       196     1.30522    0.394651
  109      16      4411      4395   161.249       144    0.533143     0.39413
  110      16      4452      4436   161.274       164    0.309631    0.393722
  111      16      4499      4483   161.514       188     2.58867    0.394114
  112      16      4542      4526   161.606       172    0.112219    0.394117
  113      16      4589      4573    161.84       188    0.259756    0.393166
  114      16      4621      4605   161.543       128    0.130345    0.393359
  115      16      4662      4646   161.564       164    0.110149    0.393394
  116      16      4694      4678   161.274       128    0.130513    0.393399
  117      16      4733      4717   161.229       156    0.243465    0.394524
  118      16      4770      4754   161.117       148     0.46596    0.394992
  119      16      4817      4801   161.342       188    0.225602    0.394268
2020-02-14 22:59:20.334784 min lat: 0.0373792 max lat: 3.3282 avg lat: 0.39362
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      4875      4859   161.931       232    0.200754     0.39362
  121      16      4918      4902   162.014       172   0.0860285     0.39267
  122      16      4954      4938   161.866       144   0.0937885    0.392753
  123      16      4981      4965   161.428       108    0.128134    0.393598
  124      16      5019      5003   161.352       152    0.109069    0.393893
  125      16      5056      5040   161.244       148    0.106514    0.394211
  126      16      5094      5078   161.171       152     0.15851      0.3945
  127      16      5131      5115   161.067       148    0.150916    0.394999
  128      16      5173      5157   161.121       168   0.0402077    0.394969
  129      16      5201      5185    160.74       112     1.70641     0.39541
  130      16      5226      5210   160.273       100   0.0372624    0.394885
  131      16      5255      5239   159.935       116      0.1341    0.395679
  132      16      5287      5271   159.693       128    0.142474    0.396775
  133      16      5339      5323   160.056       208    0.144811     0.39812
  134      16      5379      5363   160.055       160    0.119604    0.397925
  135      16      5410      5394   159.788       124    0.213036    0.398429
  136      16      5454      5438   159.907       176    0.143864    0.398316
  137      16      5509      5493   160.346       220    0.111212    0.397563
  138      16      5554      5538   160.488       180    0.224602    0.396967
  139      16      5608      5592   160.887       216    0.138324    0.396405
2020-02-14 22:59:40.337965 min lat: 0.0372624 max lat: 3.7977 avg lat: 0.395409
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      5659      5643   161.194       204    0.125527    0.395409
  141      16      5709      5693   161.469       200   0.0801818    0.394802
  142      16      5751      5735   161.515       168   0.0769544    0.394184
  143      16      5790      5774   161.477       156     0.25902    0.393683
  144      16      5830      5814   161.466       160     1.29159    0.394733
  145      16      5871      5855   161.484       164    0.728232    0.394127
  146      16      5908      5892   161.391       148    0.451801    0.394311
  147      16      5955      5939   161.572       188    0.283553    0.394551
  148      16      5997      5981   161.615       168    0.295431    0.394385
  149      16      6047      6031   161.873       200    0.132618    0.393483
  150      16      6077      6061   161.593       120      0.5996    0.393727
  151      16      6110      6094   161.397       132    0.863301    0.393567
  152      16      6136      6120    161.02       104    0.179123    0.395037
  153      16      6177      6161   161.039       164     0.22298    0.395764
  154      16      6225      6209    161.24       192    0.200634    0.395399
  155      16      6262      6246   161.154       148    0.108137    0.395019
  156      16      6293      6277   160.916       124     1.84742    0.395523
  157      16      6323      6307   160.655       120    0.205733    0.396288
  158      16      6367      6351   160.752       176   0.0844001    0.396248
  159      16      6409      6393   160.798       168     0.50952    0.395807
2020-02-14 23:00:00.340584 min lat: 0.0372624 max lat: 3.7977 avg lat: 0.395747
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      6454      6438   160.918       180   0.0794568    0.395747
  161      16      6498      6482   161.011       176    0.149318    0.395461
  162      16      6530      6514   160.807       128     2.30604    0.395735
  163      16      6568      6552   160.753       152   0.0762198    0.395978
  164      16      6610      6594   160.797       168     2.47345    0.395859
  165      16      6656      6640   160.938       184    0.231167     0.39583
  166      16      6693      6677    160.86       148    0.925372     0.39579
  167      16      6733      6717   160.854       160    0.124948    0.396171
  168      16      6784      6768   161.111       204   0.0710534    0.395863
  169      16      6831      6815    161.27       188    0.322445    0.395155
  170      16      6870      6854   161.238       156    0.161645    0.395137
  171      16      6919      6903   161.441       196   0.0847008    0.394955
  172      16      6965      6949   161.572       184    0.228725    0.394588
  173      16      6996      6980   161.355       124     0.31388    0.394539
  174      16      7029      7013   161.186       132     1.94047    0.394871
  175      16      7068      7052   161.155       156    0.493038     0.39515
  176      16      7104      7088   161.058       144     1.74579    0.395366
  177      16      7149      7133   161.165       180     1.46472    0.395465
  178      16      7189      7173   161.158       160    0.338431    0.395736
  179      16      7232      7216   161.218       172    0.310278    0.395447
2020-02-14 23:00:20.345500 min lat: 0.037177 max lat: 3.7977 avg lat: 0.395254
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      7283      7267   161.456       204    0.899701    0.395254
  181      16      7320      7304   161.381       148    0.890878    0.394698
  182      16      7347      7331   161.088       108    0.091821    0.395509
  183      16      7380      7364   160.929       132    0.385622     0.39591
  184      16      7416      7400   160.837       144    0.181629    0.396281
  185      16      7444      7428   160.573       112    0.143413    0.396619
  186      16      7480      7464   160.483       144    0.145263    0.396627
  187      16      7522      7506   160.523       168    0.204332    0.396952
  188      16      7567      7551   160.626       180   0.0373661    0.396876
  189      16      7598      7582   160.433       124    0.235743     0.39709
  190      16      7631      7615   160.283       132     0.14132    0.397291
  191      16      7678      7662   160.428       188    0.126972    0.397361
  192      16      7726      7710   160.592       192     1.82952    0.397006
  193      16      7775      7759   160.775       196    0.128593    0.396893
  194      16      7818      7802   160.833       172   0.0381462     0.39635
  195      16      7873      7857   161.136       220    0.228944    0.396338
  196      16      7919      7903   161.253       184    0.168294    0.395813
  197      16      7975      7959   161.571       224    0.720571    0.394958
  198      16      8025      8009   161.765       200    0.235503    0.394653
  199      16      8068      8052   161.816       172    0.130741    0.394229
2020-02-14 23:00:40.349181 min lat: 0.037177 max lat: 3.7977 avg lat: 0.394002
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      8109      8093   161.827       164     1.74702    0.394002
  201      16      8147      8131   161.778       152    0.081515    0.393947
  202      16      8189      8173   161.809       168    0.190654    0.394281
  203      16      8227      8211    161.76       152      1.1193    0.394449
  204      16      8269      8253   161.791       168   0.0375981      0.3943
  205      16      8313      8297    161.86       176    0.363053     0.39391
  206      16      8355      8339    161.89       168    0.321838    0.393566
  207      16      8390      8374   161.784       140     2.76282    0.393724
  208      16      8429      8413   161.756       156    0.286283    0.393847
  209      16      8467      8451   161.709       152      1.8951    0.394293
  210      16      8503      8487   161.625       144    0.322017    0.394304
  211      16      8527      8511   161.314        96    0.158668    0.394721
  212      16      8561      8545   161.194       136    0.371475    0.395156
  213      16      8596      8580   161.095       140    0.221386    0.395608
  214      16      8648      8632   161.314       208    0.257262    0.395427
  215      16      8690      8674   161.345       168    0.173525     0.39527
  216      16      8730      8714   161.338       160    0.155188    0.395224
  217      16      8780      8764   161.516       200   0.0805348    0.394928
  218      16      8823      8807   161.564       172    0.241537    0.394776
  219      16      8865      8849   161.594       168    0.288854    0.394875
2020-02-14 23:01:00.351723 min lat: 0.037177 max lat: 3.7977 avg lat: 0.394669
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      8904      8888   161.568       156     1.67448    0.394669
  221      16      8954      8938   161.742       200     2.06527    0.394348
  222      16      8990      8974   161.662       144    0.109834    0.394328
  223      16      9024      9008   161.547       136   0.0802081    0.394393
  224      16      9066      9050   161.576       168    0.121965     0.39467
  225      16      9109      9093   161.622       172     0.13277    0.394397
  226      16      9154      9138   161.703       180     1.82424    0.394558
  227      16      9194      9178   161.695       160    0.163132    0.394204
  228      16      9242      9226   161.828       192    0.141296    0.394206
  229      16      9280      9264   161.785       152    0.297481    0.394122
  230      16      9322      9306   161.812       168    0.193375    0.393794
  231      16      9362      9346   161.804       160     0.44246    0.394038
  232      16      9404      9388   161.831       168    0.116987    0.394084
  233      16      9447      9431   161.874       172    0.169001    0.394086
  234      16      9482      9466   161.781       140    0.108825    0.394071
  235      16      9517      9501   161.688       140      0.1079    0.394095
  236      16      9555      9539   161.647       152    0.422795    0.394104
  237      16      9591      9575   161.572       144    0.238435    0.394421
  238      16      9629      9613   161.532       152    0.297895    0.394591
  239      16      9677      9661   161.659       192    0.127349    0.394341
2020-02-14 23:01:20.354282 min lat: 0.037177 max lat: 3.7977 avg lat: 0.39472
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      9704      9688   161.436       108    0.119046     0.39472
  241      16      9742      9726   161.397       152    0.242698    0.395238
  242      16      9786      9770   161.457       176    0.131604    0.395315
  243      16      9824      9808   161.418       152    0.124007    0.395359
  244      16      9864      9848   161.412       160     1.03336    0.395257
  245      16      9902      9886   161.373       152   0.0845803    0.395326
  246      16      9927      9911   161.124       100   0.0842903    0.395676
  247      16      9956      9940   160.941       116    0.126562     0.39554
  248      16     10000      9984   161.002       176     2.64659    0.395845
  249      16     10029     10013   160.821       116    0.750484    0.396193
  250      16     10070     10054   160.833       164   0.0835146    0.396892
  251      16     10105     10089    160.75       140    0.834029    0.396741
  252      16     10135     10119   160.589       120    0.245787    0.396793
  253      16     10170     10154   160.507       140    0.108867    0.397467
  254      16     10203     10187   160.395       132    0.160558    0.397559
  255      16     10228     10212   160.158       100    0.172588    0.397845
  256      16     10258     10242   160.001       120    0.145178    0.398206
  257      16     10299     10283   160.016       164    0.122252     0.39863
  258      16     10345     10329   160.109       184    0.117493    0.398606
  259      16     10393     10377   160.232       192   0.0877103    0.398321
2020-02-14 23:01:40.358503 min lat: 0.037177 max lat: 3.7977 avg lat: 0.398218
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16     10427     10411   160.138       136    0.190907    0.398218
  261      16     10472     10456   160.214       180      0.2608    0.398185
  262      16     10526     10510   160.427       216    0.310256    0.398138
  263      16     10566     10550   160.425       160    0.144514    0.397781
  264      16     10587     10571   160.136        84   0.0781688    0.397929
  265      16     10627     10611   160.135       160    0.144636    0.398461
  266      16     10656     10640   159.969       116     2.08071    0.398472
  267      16     10684     10668    159.79       112   0.0776477    0.398869
  268      16     10728     10712    159.85       176    0.254517    0.398997
  269      16     10769     10753   159.865       164    0.209268    0.399161
  270      16     10800     10784   159.733       124   0.0794802    0.399231
  271      16     10846     10830   159.822       184    0.208712    0.399197
  272      16     10887     10871   159.837       164   0.0382633    0.399228
  273      16     10925     10909   159.808       152    0.111541    0.399158
  274      16     10967     10951   159.838       168     0.24793     0.39922
  275      16     11016     11000    159.97       196     0.14281      0.3991
  276      16     11060     11044   160.028       176    0.164773    0.398994
  277      16     11097     11081   159.984       148   0.0851005    0.398718
  278      16     11141     11125   160.042       176    0.253983    0.398676
  279      16     11187     11171   160.127       184    0.118874    0.398613
2020-02-14 23:02:00.361371 min lat: 0.037177 max lat: 3.7977 avg lat: 0.398341
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16     11235     11219   160.241       192     0.44555    0.398341
  281      16     11264     11248   160.084       116     0.23781     0.39855
  282      16     11301     11285   160.041       148    0.111666    0.398319
  283      16     11340     11324   160.026       156    0.113925    0.398296
  284      16     11367     11351   159.843       108    0.695603    0.398577
  285      16     11397     11381   159.703       120   0.0756108    0.399306
  286      16     11428     11412   159.578       124     1.59796    0.399759
  287      16     11460     11444   159.468       128    0.668478    0.399606
  288      16     11482     11466    159.22        88    0.290673    0.399763
  289      16     11527     11511   159.292       180    0.233183    0.400323
  290      16     11568     11552   159.308       164    0.181735    0.400387
  291      16     11612     11596   159.365       176    0.219351    0.400347
  292      16     11656     11640   159.422       176    0.392674    0.400289
  293      16     11701     11685   159.492       180      0.2016    0.400232
  294      16     11743     11727   159.521       168    0.300506      0.4001
  295      16     11782     11766   159.509       156    0.137937    0.400092
  296      16     11815     11799   159.416       132    0.121194    0.400266
  297      16     11853     11837   159.391       152     1.11964    0.400313
  298      16     11871     11855   159.098        72   0.0792226     0.40053
  299      16     11906     11890   159.034       140    0.114346     0.40103
2020-02-14 23:02:20.364637 min lat: 0.037177 max lat: 3.7977 avg lat: 0.40089
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16     11961     11945   159.237       220   0.0830606     0.40089
  301      16     12006     11990   159.306       180    0.108408    0.400556
  302      16     12058     12042   159.467       208    0.191387    0.400489
  303      16     12099     12083   159.482       164    0.128356    0.400154
  304      16     12134     12118   159.418       140   0.0962723    0.400069
  305      16     12164     12148   159.288       120    0.127027    0.400342
  306      16     12194     12178    159.16       120     0.16311    0.400608
  307      16     12233     12217    159.15       156    0.416693    0.400806
  308      16     12277     12261   159.204       176     1.86024    0.400724
  309      16     12313     12297   159.155       144     0.13534    0.400972
  310      16     12365     12349   159.312       208    0.167827     0.40068
  311      16     12414     12398    159.43       196   0.0956161    0.400265
  312      16     12467     12451   159.598       212     0.16342    0.399962
  313      16     12505     12489   159.574       152   0.0875105    0.399805
  314      16     12559     12543   159.753       216    0.134114    0.399705
  315      16     12603     12587   159.805       176    0.198315    0.399431
  316      16     12649     12633   159.881       184    0.209453    0.399291
  317      16     12702     12686   160.046       212   0.0782464    0.398918
  318      16     12740     12724    160.02       152   0.0758649    0.398652
  319      16     12780     12764    160.02       160     1.07874    0.398634
2020-02-14 23:02:40.368051 min lat: 0.037177 max lat: 3.7977 avg lat: 0.398741
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16     12820     12804    160.02       160   0.0793648    0.398741
  321      16     12855     12839   159.958       140    0.102653    0.398813
  322      16     12899     12883   160.007       176   0.0853117    0.398653
  323      16     12940     12924    160.02       164    0.114371    0.398739
  324      16     12991     12975   160.155       204    0.355505     0.39867
  325      16     13032     13016   160.167       164    0.149489    0.398462
  326      16     13079     13063   160.253       188    0.139107    0.398219
  327      16     13120     13104   160.264       164    0.422033    0.398165
  328      16     13167     13151   160.348       188    0.925685    0.398264
  329      16     13216     13200   160.457       196     1.40942    0.397964
  330      16     13246     13230   160.334       120   0.0750039    0.397781
  331      16     13252     13236   159.922        24     2.59722    0.398091
  332      16     13305     13289   160.079       212   0.0846953    0.398721
  333      16     13347     13331   160.103       168    0.116368    0.398734
  334      16     13384     13368   160.066       148    0.116726    0.398807
  335      16     13429     13413   160.126       180    0.179768    0.398707
  336      16     13473     13457   160.173       176    0.366427    0.398531
  337      16     13498     13482   159.994       100    0.242708    0.398625
  338      16     13524     13508   159.829       104    0.262901    0.399015
  339      16     13562     13546   159.806       152      1.4873    0.399297
2020-02-14 23:03:00.370616 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399439
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16     13599     13583   159.771       148    0.243578    0.399439
  341      16     13628     13612   159.642       116    0.122422    0.399548
  342      16     13664     13648   159.597       144    0.203605    0.399703
  343      16     13702     13686   159.574       152   0.0766511    0.400014
  344      16     13747     13731   159.634       180     1.52715    0.399873
  345      16     13795     13779   159.727       192    0.366039    0.399474
  346      16     13841     13825   159.797       184    0.260386    0.399637
  347      16     13889     13873    159.89       192    0.183705    0.399119
  348      16     13924     13908   159.833       140    0.150892    0.399158
  349      16     13961     13945   159.799       148   0.0381405      0.3995
  350      16     13999     13983   159.777       152    0.157954    0.399424
  351      16     14048     14032    159.88       196     0.10399    0.399199
  352      16     14091     14075   159.914       172    0.726027    0.398989
  353      16     14120     14104    159.79       116   0.0818657    0.399228
  354      16     14150     14134   159.677       120    0.110338    0.399421
  355      16     14197     14181   159.757       188     1.76671    0.399498
  356      16     14223     14207     159.6       104    0.477281    0.399529
  357      16     14257     14241   159.534       136    0.129476    0.399781
  358      16     14288     14272   159.435       124    0.180709     0.40029
  359      16     14329     14313   159.448       164   0.0875322    0.400258
2020-02-14 23:03:20.373028 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399981
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     14384     14368   159.616       220    0.146278    0.399981
  361      16     14430     14414   159.683       184    0.229665    0.399846
  362      16     14464     14448   159.618       136     0.11968    0.400016
  363      16     14506     14490   159.641       168    0.391185    0.399769
  364      16     14539     14523   159.565       132    0.357392    0.399884
  365      16     14581     14565   159.588       168    0.115715    0.399998
  366      16     14620     14604   159.578       156    0.441937    0.399781
  367      16     14644     14628   159.405        96    0.426861    0.400008
  368      16     14682     14666   159.385       152    0.088657    0.400141
  369      16     14727     14711    159.44       180     0.56245    0.400236
  370      16     14769     14753   159.464       168    0.419094    0.400239
  371      16     14808     14792   159.454       156    0.114699    0.400303
  372      16     14846     14830   159.434       152    0.397308    0.400321
  373      16     14874     14858   159.307       112    0.247276    0.400412
  374      16     14911     14895   159.277       148    0.195536    0.400649
  375      16     14951     14935   159.278       160    0.121796    0.400688
  376      16     14991     14975    159.28       160    0.073619    0.400694
  377      16     15028     15012    159.25       148   0.0845578    0.400822
  378      16     15065     15049   159.221       148   0.0779144     0.40086
  379      16     15109     15093   159.265       176     0.16969    0.400802
2020-02-14 23:03:40.375367 min lat: 0.037177 max lat: 3.7977 avg lat: 0.400692
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     15157     15141   159.351       192     1.84038    0.400692
  381      16     15198     15182   159.363       164     1.47763    0.400556
  382      16     15236     15220   159.344       152    0.106555    0.400497
  383      16     15275     15259   159.335       156    0.414832    0.400508
  384      16     15318     15302   159.368       172   0.0423731    0.400559
  385      16     15371     15355   159.504       212    0.181143    0.400256
  386      16     15411     15395   159.506       160    0.258882    0.400208
  387      16     15449     15433   159.486       152    0.118214    0.399985
  388      16     15500     15484   159.601       204    0.176247    0.400139
  389      16     15544     15528   159.643       176    0.602476     0.40003
  390      16     15589     15573   159.695       180     1.69999    0.399822
  391      16     15630     15614   159.706       164     0.20906    0.399637
  392      16     15690     15674   159.911       240    0.138637    0.399445
  393      16     15736     15720   159.972       184     0.12967    0.399093
  394      16     15777     15761   159.982       164    0.975968    0.399107
  395      16     15820     15804   160.013       172   0.0863053    0.398987
  396      16     15855     15839   159.962       140     1.73136    0.399017
  397      16     15884     15868   159.851       116   0.0723575    0.399227
  398      16     15923     15907   159.841       156    0.170595    0.399218
  399      16     15965     15949   159.862       168    0.157549     0.39924
2020-02-14 23:04:00.378050 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399188
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     15998     15982   159.792       132    0.510901    0.399188
  401      16     16040     16024   159.813       168   0.0437401    0.399302
  402      16     16083     16067   159.843       172    0.168033    0.399238
  403      16     16120     16104   159.813       148    0.345317    0.399344
  404      16     16159     16143   159.804       156    0.183047    0.399492
  405      16     16198     16182   159.795       156   0.0394316    0.399513
  406      16     16255     16239   159.962       228    0.219768    0.399318
  407      16     16301     16285   160.021       184    0.161034    0.399045
  408      16     16340     16324   160.012       156    0.170848    0.398944
  409      16     16369     16353   159.904       116      1.5619    0.399074
  410      16     16410     16394   159.914       164    0.222773    0.399063
  411      16     16443     16427   159.846       132     2.20133     0.39934
  412      16     16483     16467   159.846       160    0.125939    0.399433
  413      16     16520     16504   159.818       148    0.120945    0.399436
  414      16     16552     16536   159.741       128     0.12296    0.399584
  415      16     16589     16573   159.712       148    0.584697     0.39955
  416      16     16636     16620    159.78       188    0.124498    0.399635
  417      16     16683     16667   159.848       188   0.0618612    0.399389
  418      16     16717     16701   159.791       136   0.0873612    0.399501
  419      16     16765     16749   159.868       192    0.260076    0.399289
2020-02-14 23:04:20.380228 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399332
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     16806     16790   159.877       164     1.90153    0.399332
  421      16     16853     16837   159.944       188       1.263    0.399369
  422      16     16899     16883   160.001       184    0.148121     0.39918
  423      16     16939     16923   160.001       160    0.396006    0.399018
  424      16     16967     16951   159.888       112    0.115557    0.399174
  425      16     17005     16989   159.869       152    0.127425    0.399208
  426      16     17034     17018   159.766       116    0.157153    0.399253
  427      16     17079     17063   159.814       180    0.100375    0.399326
  428      16     17129     17113   159.907       200    0.329485    0.399365
  429      16     17173     17157   159.945       176    0.199182    0.399197
  430      16     17220     17204    160.01       188   0.0822289    0.399088
  431      16     17258     17242   159.991       152     0.32056    0.398906
  432      16     17286     17270    159.88       112    0.517494    0.399051
  433      16     17323     17307   159.852       148    0.248073     0.39906
  434      16     17361     17345   159.834       152   0.0802191    0.399336
  435      16     17394     17378    159.77       132     1.75602      0.3995
  436      16     17430     17414   159.734       144    0.121855    0.399523
  437      16     17458     17442   159.625       112    0.400538    0.399591
  438      16     17500     17484   159.644       168     1.80967    0.399974
  439      16     17546     17530   159.699       184    0.272027    0.399943
2020-02-14 23:04:40.383551 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399764
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     17590     17574   159.736       176     1.24767    0.399764
  441      16     17628     17612   159.719       152    0.649968    0.399615
  442      16     17666     17650   159.701       152    0.112506    0.399624
  443      16     17708     17692    159.72       168     1.63763    0.399684
  444      16     17732     17716   159.576        96    0.120362    0.399878
  445      16     17767     17751   159.532       140    0.150185    0.400019
  446      16     17813     17797   159.587       184   0.0470421    0.399966
  447      16     17848     17832   159.543       140    0.257707    0.399965
  448      16     17884     17868   159.509       144    0.173857    0.400361
  449      16     17932     17916   159.581       192    0.193253    0.400293
  450      16     17977     17961   159.626       180    0.202328    0.400058
  451      16     18016     18000   159.618       156    0.138909    0.400007
  452      16     18067     18051   159.716       204    0.106073    0.399858
  453      16     18125     18109   159.876       232      0.1988    0.399573
  454      16     18177     18161   159.982       208   0.0382571    0.399189
  455      16     18215     18199   159.964       152     1.60289    0.399198
  456      16     18246     18230   159.885       124      1.1903     0.39943
  457      16     18297     18281   159.982       204    0.111288    0.399199
  458      16     18334     18318   159.955       148   0.0711721      0.3991
  459      16     18379     18363   159.999       180    0.461058    0.399224
2020-02-14 23:05:00.386041 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399022
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     18422     18406   160.025       172      1.1225    0.399022
  461      16     18457     18441   159.982       140   0.0806834    0.398942
  462      16     18500     18484   160.007       172    0.117183    0.399126
  463      16     18545     18529    160.05       180    0.630182    0.398932
  464      16     18581     18565   160.016       144     1.54936    0.398989
  465      16     18611     18595    159.93       120    0.126494    0.399008
  466      16     18651     18635    159.93       160    0.187751    0.399056
  467      16     18688     18672   159.904       148    0.449994    0.399037
  468      16     18725     18709   159.879       148    0.195858    0.399093
  469      16     18770     18754   159.921       180    0.157345    0.399193
  470      16     18803     18787   159.862       132    0.190713    0.399338
  471      16     18845     18829   159.879       168     1.75981    0.399363
  472      16     18875     18859   159.795       120      0.1132     0.39942
  473      16     18914     18898   159.787       156    0.249205    0.399319
  474      16     18960     18944   159.838       184    0.191665    0.399535
  475      16     19012     18996   159.939       208    0.173457    0.399357
  476      16     19033     19017   159.779        84     1.84464    0.399301
  477      16     19060     19044   159.671       108    0.964735    0.399741
  478      16     19094     19078   159.621       136    0.181253    0.400068
  479      16     19136     19120   159.638       168    0.334372    0.399976
2020-02-14 23:05:20.391051 min lat: 0.037177 max lat: 3.7977 avg lat: 0.399879
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     19190     19174   159.756       216    0.125889    0.399879
  481      16     19230     19214   159.756       160    0.650473      0.3997
  482      16     19267     19251   159.732       148    0.480142    0.399799
  483      16     19319     19303   159.832       208    0.148037    0.399706
  484      16     19376     19360   159.972       228   0.0523986    0.399397
  485      16     19425     19409   160.047       196   0.0760418    0.399184
  486      16     19485     19469   160.211       240    0.155393    0.398693
  487      16     19534     19518   160.285       196    0.345075    0.398532
  488      16     19569     19553   160.243       140    0.164762    0.398346
  489      16     19604     19588   160.202       140   0.0783712    0.398602
  490      16     19648     19632   160.234       176   0.0378586    0.398487
  491      16     19678     19662   160.152       120    0.233977    0.398649
  492      16     19726     19710   160.216       192   0.0997582    0.398701
  493      16     19751     19735   160.094       100     0.60529    0.398624
  494      16     19787     19771   160.061       144    0.236845    0.398846
  495      16     19821     19805   160.013       136    0.119041    0.399088
  496      16     19870     19854   160.085       196     1.86207    0.398926
  497      16     19903     19887   160.029       132     0.43073    0.398934
  498      16     19948     19932   160.069       180    0.250958    0.398887
  499      16     19992     19976   160.101       176    0.236857    0.398962
2020-02-14 23:05:40.394422 min lat: 0.037177 max lat: 3.7977 avg lat: 0.398994
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     20014     19998   159.956        88    0.819045    0.398994
  501      16     20055     20039   159.964       164    0.370929    0.399065
  502      16     20101     20085   160.012       184     1.82925    0.399049
  503      16     20131     20115   159.933       120     1.76174    0.399085
  504      16     20167     20151   159.901       144    0.221506    0.399129
  505      16     20208     20192   159.909       164    0.303542    0.399263
  506      16     20245     20229   159.885       148   0.0892217    0.399339
  507      16     20273     20257   159.791       112    0.368673    0.399401
  508      16     20305     20289   159.728       128    0.133467    0.399454
  509      16     20347     20331   159.744       168    0.151511    0.399908
  510      16     20404     20388   159.878       228    0.226145    0.399611
  511      16     20460     20444   160.004       224   0.0998638    0.399261
  512      16     20517     20501   160.136       228    0.728087     0.39902
  513      16     20568     20552   160.222       204    0.654557    0.398659
  514      16     20615     20599   160.276       188    0.119261    0.398601
  515      16     20659     20643   160.306       176    0.324779    0.398453
  516      16     20709     20693   160.383       200   0.0373624    0.398206
  517      16     20750     20734    160.39       164    0.530034    0.398125
  518      16     20791     20775   160.397       164    0.881272     0.39807
  519      16     20837     20821   160.443       184     2.24914    0.397978
2020-02-14 23:06:00.397664 min lat: 0.037177 max lat: 3.7977 avg lat: 0.39792
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     20883     20867   160.488       184    0.308555     0.39792
  521      16     20932     20916   160.556       196    0.075896    0.397743
  522      16     20963     20947   160.486       124    0.122013    0.397673
  523      16     20996     20980   160.431       132    0.155341    0.397848
  524      16     21043     21027   160.484       188     1.87746    0.398072
  525      16     21089     21073   160.529       184    0.112876    0.397831
  526      16     21134     21118   160.565       180    0.176155    0.397811
  527      16     21174     21158   160.564       160    0.194761    0.397689
  528      16     21213     21197   160.556       156    0.111733    0.397644
  529      16     21251     21235    160.54       152    0.156289    0.397799
  530      16     21276     21260   160.425       100   0.0980361    0.397866
  531      16     21300     21284   160.304        96    0.291603    0.397951
  532      16     21334     21318   160.258       136    0.476705    0.398449
  533      16     21360     21344   160.153       104     2.18596    0.398563
  534      16     21391     21375   160.085       124     2.37675    0.398714
  535      16     21439     21423   160.144       192    0.122922    0.398903
  536      16     21484     21468   160.181       180    0.189145    0.398757
  537      16     21523     21507   160.174       156      1.1346    0.398735
  538      16     21560     21544   160.151       148     1.74236    0.398757
  539      16     21615     21599   160.262       220    0.209218    0.398616
2020-02-14 23:06:20.401240 min lat: 0.037177 max lat: 3.7977 avg lat: 0.398422
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     21662     21646   160.313       188    0.168682    0.398422
  541      16     21692     21676   160.239       120   0.0384589    0.398558
  542      16     21730     21714   160.223       152   0.0747618    0.398493
  543      16     21757     21741   160.127       108    0.117671    0.398652
  544      16     21795     21779   160.112       152    0.206234    0.398827
  545      16     21840     21824   160.149       180   0.0733131    0.398636
  546      16     21894     21878   160.251       216    0.118838    0.398582
  547      16     21924     21908   160.177       120     1.66878    0.398576
  548      16     21984     21968   160.323       240    0.125454    0.398488
  549      16     22036     22020    160.41       208    0.645959    0.398198
  550      16     22070     22054   160.365       136    0.171635    0.398285
  551      16     22122     22106   160.452       208    0.121334    0.398098
  552      16     22166     22150    160.48       176     0.17879    0.397975
  553      16     22209     22193     160.5       172   0.0401149    0.397901
  554      16     22250     22234   160.507       164   0.0938098    0.397835
  555      16     22302     22286   160.592       208    0.123682    0.397779
  556      16     22337     22321   160.555       140   0.0870244    0.397675
  557      16     22370     22354   160.504       132    0.177097    0.397892
  558      16     22418     22402    160.56       192    0.288839    0.397874
  559      16     22467     22451   160.624       196    0.357258    0.397636
2020-02-14 23:06:40.404157 min lat: 0.037177 max lat: 3.7977 avg lat: 0.397569
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     22505     22489   160.608       152    0.184911    0.397569
  561      16     22537     22521    160.55       128     2.02169    0.397682
  562      16     22566     22550   160.471       116     1.05594    0.397722
  563      16     22600     22584   160.427       136     1.48791    0.397947
  564      16     22657     22641   160.547       228    0.221685    0.398042
  565      16     22702     22686   160.581       180    0.130405    0.397764
  566      16     22754     22738   160.665       208    0.193956    0.397543
  567      16     22796     22780   160.678       168    0.158087     0.39733
  568      16     22834     22818   160.663       152    0.109477    0.397427
  569      16     22863     22847   160.584       116    0.126454    0.397471
  570      16     22899     22883   160.555       144    0.247076    0.397482
  571      16     22945     22929   160.596       184    0.166743    0.397661
  572      16     22982     22966   160.574       148     1.47256    0.397634
  573      16     23012     22996   160.503       120    0.136346    0.397875
  574      16     23052     23036   160.502       160   0.0375928    0.397865
  575      16     23108     23092   160.613       224    0.107713    0.397704
  576      16     23161     23145   160.702       212    0.150169    0.397591
  577      16     23209     23193   160.756       192    0.644658    0.397368
  578      16     23242     23226   160.706       132     1.61862    0.397304
  579      16     23277     23261    160.67       140    0.248773    0.397355
2020-02-14 23:07:00.406933 min lat: 0.037177 max lat: 3.7977 avg lat: 0.397411
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     23306     23290   160.593       116   0.0796421    0.397411
  581      16     23345     23329   160.585       156    0.326844    0.397633
  582      16     23389     23373   160.612       176   0.0858893    0.397783
  583      16     23436     23420   160.659       188    0.111512    0.397561
  584      16     23479     23463   160.678       172   0.0374978    0.397498
  585      16     23513     23497   160.636       136    0.144081    0.397596
  586      16     23551     23535   160.621       152   0.0732863    0.397519
  587      16     23594     23578   160.641       172    0.140598    0.397588
  588      16     23646     23630   160.721       208    0.112979    0.397386
  589      16     23694     23678   160.774       192    0.173778    0.397287
  590      16     23739     23723   160.806       180    0.132361    0.397262
  591      16     23794     23778   160.907       220    0.190519    0.397091
  592      16     23851     23835    161.02       228    0.183477    0.396797
  593      16     23886     23870   160.984       140   0.0813829    0.396705
  594      16     23917     23901   160.922       124    0.121268    0.396711
  595      16     23946     23930   160.847       116     0.12366    0.396998
  596      16     23995     23979   160.905       196     0.46652    0.397142
  597      16     24058     24042   161.058       252    0.104585    0.396743
  598      16     24097     24081   161.049       156    0.839443    0.396593
  599      16     24119     24103   160.927        88    0.175991    0.396742
2020-02-14 23:07:20.410917 min lat: 0.037177 max lat: 3.7977 avg lat: 0.396839
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      16     24147     24131   160.846       112    0.561309    0.396839
Total time run:       600.975391
Total reads made:     24147
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   160.719
Average IOPS:         40
Stddev IOPS:          7
Max IOPS:             64
Min IOPS:             6
Average Latency(s):   0.397626
Max latency(s):       3.7977
Min latency(s):       0.037177

Addendum 2020-02-15

Like before, a 500GB block device used in a CentOS Stream VM.

Sequential Read, 4k Blocksize

  • READ: bw=81.6MiB/s
  • iops: avg=20881.32

fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4k

[root@centos-stream ~]# fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4k
readtest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
readtest: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=92.8MiB/s,w=0KiB/s][r=23.7k,w=0 IOPS][eta 00m:01s]
readtest: (groupid=0, jobs=1): err= 0: pid=22260: Sat Feb 15 14:35:24 2020
   read: IOPS=20.9k, BW=81.6MiB/s (85.5MB/s)(160GiB/2008932msec)
    clat (nsec): min=1030, max=664744k, avg=46222.65, stdev=1647229.86
     lat (nsec): min=1360, max=664745k, avg=46586.50, stdev=1647230.37
    clat percentiles (nsec):
     |  1.00th=[    1096],  5.00th=[    1144], 10.00th=[    1192],
     | 20.00th=[    1272], 30.00th=[    1784], 40.00th=[    3760],
     | 50.00th=[    3856], 60.00th=[    3984], 70.00th=[    4080],
     | 80.00th=[    4320], 90.00th=[    4640], 95.00th=[    5088],
     | 99.00th=[   13376], 99.50th=[   19072], 99.90th=[ 7700480],
     | 99.95th=[35389440], 99.99th=[80216064]
   bw (  KiB/s): min= 8192, max=143360, per=100.00%, avg=83525.31, stdev=17156.90, samples=4016
   iops        : min= 2048, max=35840, avg=20881.32, stdev=4289.23, samples=4016
  lat (usec)   : 2=30.67%, 4=31.84%, 10=35.97%, 20=1.08%, 50=0.30%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.02%, 50=0.04%
  lat (msec)   : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=1.85%, sys=10.16%, ctx=51916, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=41943040,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=81.6MiB/s (85.5MB/s), 81.6MiB/s-81.6MiB/s (85.5MB/s-85.5MB/s), io=160GiB (172GB), run=2008932-2008932msec

Disk stats (read/write):
  vdc: ios=166858/7, merge=0/0, ticks=19324638/895, in_queue=19242019, util=9.64%

Sequential Read, 4m Blocksize

  • READ: bw=87.2MiB/s
  • iops: avg=21.80

fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4m

[root@centos-stream ~]# fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4m
readtest: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=100MiB/s,w=0KiB/s][r=25,w=0 IOPS][eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=22595: Sat Feb 15 15:08:00 2020
   read: IOPS=21, BW=87.2MiB/s (91.4MB/s)(160GiB/1879130msec)
    clat (usec): min=410, max=642644, avg=45828.27, stdev=55998.10
     lat (usec): min=410, max=642645, avg=45829.46, stdev=55998.01
    clat percentiles (usec):
     |  1.00th=[   502],  5.00th=[   652], 10.00th=[   947], 20.00th=[  1795],
     | 30.00th=[  2573], 40.00th=[  3752], 50.00th=[ 17433], 60.00th=[ 40109],
     | 70.00th=[ 66323], 80.00th=[ 96994], 90.00th=[131597], 95.00th=[156238],
     | 99.00th=[206570], 99.50th=[227541], 99.90th=[295699], 99.95th=[346031],
     | 99.99th=[450888]
   bw (  KiB/s): min= 8192, max=147456, per=100.00%, avg=89370.05, stdev=18267.50, samples=3754
   iops        : min=    2, max=   36, avg=21.80, stdev= 4.46, samples=3754
  lat (usec)   : 500=1.00%, 750=7.24%, 1000=2.23%
  lat (msec)   : 2=12.46%, 4=18.10%, 10=5.36%, 20=4.66%, 50=13.10%
  lat (msec)   : 100=16.89%, 250=18.70%, 500=0.25%, 750=0.01%
  cpu          : usr=0.03%, sys=4.81%, ctx=39885, majf=0, minf=525
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40960,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=87.2MiB/s (91.4MB/s), 87.2MiB/s-87.2MiB/s (91.4MB/s-91.4MB/s), io=160GiB (172GB), run=1879130-1879130msec

Disk stats (read/write):
  vdc: ios=166878/0, merge=17/0, ticks=17788526/0, in_queue=17705166, util=10.28%

Sequential Write, 4k Blocksize

  • READ: bw=36.3MiB/s
  • iops: avg=9293.92

fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k

[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
writetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=32.3MiB/s][r=0,w=8261 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=23010: Sat Feb 15 16:27:47 2020
  write: IOPS=9294, BW=36.3MiB/s (38.1MB/s)(160GiB/4512682msec)
    clat (usec): min=2, max=56749, avg=105.88, stdev=940.46
     lat (usec): min=2, max=56750, avg=106.26, stdev=940.46
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    6], 80.00th=[    6], 90.00th=[    7], 95.00th=[    8],
     | 99.00th=[ 6718], 99.50th=[ 8586], 99.90th=[11469], 99.95th=[11469],
     | 99.99th=[11600]
   bw (  KiB/s): min=15960, max=948840, per=100.00%, avg=37175.72, stdev=16605.80, samples=9025
   iops        : min= 3990, max=237210, avg=9293.92, stdev=4151.45, samples=9025
  lat (usec)   : 4=66.41%, 10=30.55%, 20=1.59%, 50=0.23%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.96%, 20=0.25%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.72%, sys=4.66%, ctx=508702, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,41943040,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=36.3MiB/s (38.1MB/s), 36.3MiB/s-36.3MiB/s (38.1MB/s-38.1MB/s), io=160GiB (172GB), run=4512682-4512682msec

Disk stats (read/write):
  vdc: ios=0/166857, merge=0/3, ticks=0/863796689, in_queue=863713179, util=0.57%

Sequential Write, 4m Blocksize

  • READ: bw=56.5MiB/s
  • iops: avg=14.34

fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m

[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
writetest: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1                                                                                                                      
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=60.1MiB/s][r=0,w=15 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=23663: Sat Feb 15 17:16:05 2020
  write: IOPS=14, BW=56.5MiB/s (59.3MB/s)(160GiB/2897626msec)
    clat (usec): min=1369, max=146898, avg=69494.24, stdev=12401.95
     lat (usec): min=1465, max=147023, avg=69620.43, stdev=12400.00
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   50], 10.00th=[   60], 20.00th=[   61],
     | 30.00th=[   64], 40.00th=[   70], 50.00th=[   70], 60.00th=[   72],
     | 70.00th=[   77], 80.00th=[   81], 90.00th=[   82], 95.00th=[   90],
     | 99.00th=[   94], 99.50th=[  101], 99.90th=[  110], 99.95th=[  116],
     | 99.99th=[  132]
   bw (  KiB/s): min=32768, max=1785856, per=100.00%, avg=58818.35, stdev=28286.29, samples=5704
   iops        : min=    8, max=  436, avg=14.34, stdev= 6.91, samples=5704
  lat (msec)   : 2=0.49%, 4=0.38%, 10=0.04%, 20=0.05%, 50=5.20%
  lat (msec)   : 100=93.64%, 250=0.21%
  cpu          : usr=0.13%, sys=3.42%, ctx=344222, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,40960,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=56.5MiB/s (59.3MB/s), 56.5MiB/s-56.5MiB/s (59.3MB/s-59.3MB/s), io=160GiB (172GB), run=2897626-2897626msec

Disk stats (read/write):
  vdc: ios=0/166761, merge=0/37, ticks=0/549460466, in_queue=549377072, util=0.90%

Sequential, 80% Read, 20% Write, 4k Blocksize

  • READ: bw=58.6MiB/s
  • iops: avg=15219.35
  • WRITE: bw=14.6MiB/s
  • iops: avg=3804.37

fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k

[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k
read80write20test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
read80write20test: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [M(1)][100.0%][r=76.9MiB/s,w=18.0MiB/s][r=19.7k,w=4856 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=24828: Sat Feb 15 19:11:50 2020
   read: IOPS=14.0k, BW=58.6MiB/s (61.4MB/s)(128GiB/2238448msec)
    clat (nsec): min=1040, max=6732.9M, avg=63398.09, stdev=3332745.35
     lat (nsec): min=1370, max=6732.9M, avg=63756.83, stdev=3332745.67
    clat percentiles (nsec):
     |  1.00th=[     1112],  5.00th=[     1144], 10.00th=[     1176],
     | 20.00th=[     1208], 30.00th=[     1256], 40.00th=[     1320],
     | 50.00th=[     1496], 60.00th=[     3376], 70.00th=[     3856],
     | 80.00th=[     4048], 90.00th=[     4384], 95.00th=[     4832],
     | 99.00th=[     9792], 99.50th=[    16512], 99.90th=[  7634944],
     | 99.95th=[ 41680896], 99.99th=[116916224]
   bw (  KiB/s): min= 4087, max=131072, per=100.00%, avg=60877.40, stdev=27969.38, samples=4408
   iops        : min= 1021, max=32768, avg=15219.35, stdev=6992.35, samples=4408
  write: IOPS=3747, BW=14.6MiB/s (15.3MB/s)(31.0GiB/2238448msec)
    clat (nsec): min=1800, max=9485.3k, avg=4791.46, stdev=7221.28
     lat (usec): min=2, max=9485, avg= 5.19, stdev= 7.27
    clat percentiles (nsec):
     |  1.00th=[ 2024],  5.00th=[ 2352], 10.00th=[ 2448], 20.00th=[ 2512],
     | 30.00th=[ 2672], 40.00th=[ 2896], 50.00th=[ 3536], 60.00th=[ 5280],
     | 70.00th=[ 5792], 80.00th=[ 6240], 90.00th=[ 7648], 95.00th=[ 9408],
     | 99.00th=[19328], 99.50th=[24704], 99.90th=[38656], 99.95th=[45824],
     | 99.99th=[68096]
   bw (  KiB/s): min=  832, max=32752, per=100.00%, avg=15217.50, stdev=6995.75, samples=4408
   iops        : min=  208, max= 8188, avg=3804.37, stdev=1748.94, samples=4408
  lat (usec)   : 2=47.21%, 4=25.51%, 10=25.67%, 20=1.17%, 50=0.33%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
  lat (msec)   : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=1.63%, sys=8.15%, ctx=40585, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=33555281,8387759,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=58.6MiB/s (61.4MB/s), 58.6MiB/s-58.6MiB/s (61.4MB/s-61.4MB/s), io=128GiB (137GB), run=2238448-2238448msec
  WRITE: bw=14.6MiB/s (15.3MB/s), 14.6MiB/s-14.6MiB/s (15.3MB/s-15.3MB/s), io=31.0GiB (34.4GB), run=2238448-2238448msec                                                                                                                      

Disk stats (read/write):
  vdc: ios=133172/33814, merge=0/9, ticks=19969679/24963286, in_queue=44849237, util=7.38%

Sequential, 80% Read, 20% Write, 4m Blocksize

  • READ: bw=65.2MiB/s
  • iops: avg=16.71
  • WRITE: bw=16.5MiB/s
  • iops: avg=5.31

fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m

[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
read80write20test: (g=0): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1                                                                                                                 
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=32.0MiB/s,w=4100KiB/s][r=8,w=1 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=25164: Sat Feb 15 19:45:16 2020
   read: IOPS=16, BW=65.2MiB/s (68.4MB/s)(128GiB/2005834msec)
    clat (usec): min=417, max=1571.8k, avg=60611.73, stdev=93993.32
     lat (usec): min=417, max=1571.8k, avg=60612.91, stdev=93993.26
    clat percentiles (usec):
     |  1.00th=[    494],  5.00th=[    619], 10.00th=[   1188],
     | 20.00th=[   1926], 30.00th=[   2737], 40.00th=[   7242],
     | 50.00th=[  30016], 60.00th=[  49546], 70.00th=[  72877],
     | 80.00th=[ 103285], 90.00th=[ 149947], 95.00th=[ 202376],
     | 99.00th=[ 467665], 99.50th=[ 616563], 99.90th=[ 851444],
     | 99.95th=[ 943719], 99.99th=[1266680]
   bw (  KiB/s): min= 8143, max=147456, per=100.00%, avg=68526.64, stdev=29831.76, samples=3907
   iops        : min=    1, max=   36, avg=16.71, stdev= 7.28, samples=3907
  write: IOPS=4, BW=16.5MiB/s (17.3MB/s)(32.3GiB/2005834msec)
    clat (usec): min=851, max=13613, avg=2636.18, stdev=1151.90
     lat (usec): min=890, max=13834, avg=2727.32, stdev=1157.80
    clat percentiles (usec):
     |  1.00th=[ 1012],  5.00th=[ 1319], 10.00th=[ 1385], 20.00th=[ 1500],
     | 30.00th=[ 1762], 40.00th=[ 2089], 50.00th=[ 2442], 60.00th=[ 2835],
     | 70.00th=[ 3228], 80.00th=[ 3687], 90.00th=[ 4228], 95.00th=[ 4621],
     | 99.00th=[ 5342], 99.50th=[ 6194], 99.90th=[ 8979], 99.95th=[10683],
     | 99.99th=[13566]
   bw (  KiB/s): min= 8110, max=106496, per=100.00%, avg=21843.13, stdev=14054.37, samples=3102
   iops        : min=    1, max=   26, avg= 5.31, stdev= 3.43, samples=3102
  lat (usec)   : 500=0.96%, 750=4.22%, 1000=1.59%
  lat (msec)   : 2=17.72%, 4=22.73%, 10=5.93%, 20=3.37%, 50=11.73%
  lat (msec)   : 100=15.00%, 250=13.97%, 500=2.10%, 750=0.50%, 1000=0.15%
  cpu          : usr=0.06%, sys=4.60%, ctx=31635, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32688,8272,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=65.2MiB/s (68.4MB/s), 65.2MiB/s-65.2MiB/s (68.4MB/s-68.4MB/s), io=128GiB (137GB), run=2005834-2005834msec
  WRITE: bw=16.5MiB/s (17.3MB/s), 16.5MiB/s-16.5MiB/s (17.3MB/s-17.3MB/s), io=32.3GiB (34.7GB), run=2005834-2005834msec                                                                                                                      

Disk stats (read/write):
  vdc: ios=132985/33922, merge=14/0, ticks=17974508/26677750, in_queue=44568551, util=8.07%

Addendum 2020-02-16

I removed the 2 small HDDs from the chassis.

command log of removing osd.9 and osd.10

[root@ceph-mon-01 ~]# ceph osd out  osd.10
[root@ceph-mon-01 ~]# ceph osd out  osd.9
[root@ceph-mon-01 ~]# ceph -w # until cluster is healthy again.
[root@ceph-mon-01 ~]# ceph osd tree
[root@f5-422-01 ~]# systemctl stop ceph-osd@9.service
[root@f5-422-01 ~]# systemctl stop ceph-osd@10.service
[root@f5-422-01 ~]# cat /etc/ceph/ceph.conf 
[root@ceph-mon-01 ~]# ceph osd purge osd.9 --yes-i-really-mean-it
[root@ceph-mon-01 ~]# ceph osd purge osd.10 --yes-i-really-mean-it

Remove old /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm… symlinks

Addendum 2020-02-17

cleaned remaining HDDs and deleted /dev/sda4

└─sda4                                         8:4    0 399.2G  0 part  
  ├─VG_Ceph_SSD_01-LV_Ceph_SSD_OSD_01        253:5    0    80G  0 lvm   
  │ └─q8tPpn-1wSg-ais7-7cac-k5eB-HLd1-UItqkV 253:17   0    80G  0 crypt 
  ├─VG_Ceph_SSD_01-LV_Ceph_SSD_OSD_02        253:6    0     8G  0 lvm   
  │ └─EmDgfU-hhKz-1Pe5-ua8x-Hsl3-F0TH-ythz10 253:15   0     8G  0 crypt 
  ├─VG_Ceph_SSD_01-LV_Ceph_SSD_OSD_03        253:7    0     8G  0 lvm   
  │ └─jtXDYp-nG6y-bKsp-okzW-5Aj3-zgJQ-5cDWz1 253:16   0     8G  0 crypt 
  └─VG_Ceph_SSD_01-LV_Ceph_SSD_OSD_00        253:8    0    80G  0 lvm   
    └─YGRkoK-BgOK-CqU7-GUih-Jc6m-gvjH-qw593H 253:20   0    80G  0 crypt 

then ran playbook terramaster-prep-disks.yml which simply created VGs and LVs for use with Ceph. I only made a Playbook since I will have to prepare multiple F5-422. Creating them by hand is also fine, but they must be pre-created for osd_scenario: lvm if you use ceph-ansible.

I created 2 OSDs, each with one full HDD and a 80GiB LV on the SSD.

[root@f5-422-01 ~]# ceph-volume lvm create \
                    --data VG_HDD_sl2_Ceph/LV_HDD_sl2_OSD_data \
                    --block.db VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_01 \
                    --dmcrypt \
                    --bluestore
[root@f5-422-01 ~]# ceph-volume lvm create \
                    --data VG_HDD_sl3_Ceph/LV_HDD_sl3_OSD_data \
                    --block.db VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_02 \
                    --dmcrypt \
                    --bluestore

lsblk

[root@f5-422-01 ~]# lsblk
NAME                                         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                            8:0    0 465.8G  0 disk  
├─sda1                                         8:1    0   600M  0 part  /boot/efi
├─sda2                                         8:2    0     1G  0 part  /boot
├─sda3                                         8:3    0    65G  0 part  
│ ├─VG_f5--422--01-LV_root                   253:0    0     8G  0 lvm   /
│ ├─VG_f5--422--01-LV_swap                   253:1    0     4G  0 lvm   [SWAP]
│ ├─VG_f5--422--01-LV_home                   253:5    0     4G  0 lvm   /home
│ ├─VG_f5--422--01-LV_var                    253:6    0     8G  0 lvm   /var
│ └─VG_f5--422--01-LV_var_log                253:7    0     2G  0 lvm   /var/log
└─sda4                                         8:4    0 399.2G  0 part  
  ├─VG_SSD_sl1_Ceph-LV_SSD_sl1_OSD_db_01     253:2    0    80G  0 lvm   
  │ └─DMKQ8z-zn6W-08os-acCj-mi1v-xD6c-tXMa6F 253:10   0    80G  0 crypt 
  └─VG_SSD_sl1_Ceph-LV_SSD_sl1_OSD_db_02     253:4    0    80G  0 lvm   
    └─yJV8Ej-IJNV-Qr8i-pJ5K-ytay-SFXc-brEK8F 253:12   0    80G  0 crypt 
sdb                                            8:16   0 931.5G  0 disk  
└─sdb1                                         8:17   0 931.5G  0 part  
  └─VG_HDD_sl2_Ceph-LV_HDD_sl2_OSD_data      253:8    0 931.5G  0 lvm   
    └─XWXbNF-SZ2B-0jRG-0jNY-GUjr-131X-05v16p 253:9    0 931.5G  0 crypt 
sdc                                            8:32   0   1.8T  0 disk  
└─sdc1                                         8:33   0   1.8T  0 part  
  └─VG_HDD_sl3_Ceph-LV_HDD_sl3_OSD_data      253:3    0   1.8T  0 lvm   
    └─S7R1hz-KfGQ-dcfe-RhKg-xl0N-EYQj-82orQE 253:11   0   1.8T  0 crypt

ceph osd tree

[root@ceph-mon-01 ~]# ceph osd tree
ID  CLASS WEIGHT   TYPE NAME              STATUS REWEIGHT PRI-AFF 
 -1       12.73514 root default                                   
-17        2.72899     host f5-422-01                             
  7   hdd  0.90970         osd.7              up  1.00000 1.00000 
  8   hdd  1.81929         osd.8              up  1.00000 1.00000 
-13        0.90959     host odroid-hc2-00                         
  6   hdd  0.90959         osd.6              up  1.00000 1.00000 
-11        0.90970     host odroid-hc2-01                         
  0   hdd  0.90970         osd.0              up  1.00000 1.00000 
 -9        0.90970     host odroid-hc2-02                         
  4   hdd  0.90970         osd.4              up  1.00000 1.00000 
-15        1.81929     host odroid-hc2-03                         
  5   hdd  1.81929         osd.5              up  1.00000 1.00000 
 -3        1.81929     host odroid-hc2-04                         
  1   hdd  1.81929         osd.1              up  1.00000 1.00000 
 -5        1.81929     host odroid-hc2-05                         
  2   hdd  1.81929         osd.2              up  1.00000 1.00000 
 -7        1.81929     host odroid-hc2-06                         
  3   hdd  1.81929         osd.3              up  1.00000 1.00000

ceph-volume lvm list

[root@f5-422-01 ~]# ceph-volume lvm list


====== osd.8 =======

  [block]    /dev/VG_HDD_sl3_Ceph/LV_HDD_sl3_OSD_data

      type                      block
      osd id                    8
      cluster fsid              154b75c2-1767-45c5-b7f6-ba7e3a36fc12
      cluster name              ceph
      osd fsid                  d2c7d993-a204-43b7-9c87-4bc6331feb8f
      db device                 /dev/VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_02
      encrypted                 1
      db uuid                   yJV8Ej-IJNV-Qr8i-pJ5K-ytay-SFXc-brEK8F
      cephx lockbox secret      [REDACTED]
      block uuid                S7R1hz-KfGQ-dcfe-RhKg-xl0N-EYQj-82orQE
      block device              /dev/VG_HDD_sl3_Ceph/LV_HDD_sl3_OSD_data
      vdo                       0
      crush device class        None
      devices                   /dev/sdc1

  [  db]    /dev/VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_02

      type                      db
      osd id                    8
      cluster fsid              154b75c2-1767-45c5-b7f6-ba7e3a36fc12
      cluster name              ceph
      osd fsid                  d2c7d993-a204-43b7-9c87-4bc6331feb8f
      db device                 /dev/VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_02
      encrypted                 1
      db uuid                   yJV8Ej-IJNV-Qr8i-pJ5K-ytay-SFXc-brEK8F
      cephx lockbox secret      [REDACTED]
      block uuid                S7R1hz-KfGQ-dcfe-RhKg-xl0N-EYQj-82orQE
      block device              /dev/VG_HDD_sl3_Ceph/LV_HDD_sl3_OSD_data
      vdo                       0
      crush device class        None
      devices                   /dev/sda4

====== osd.7 =======

  [block]    /dev/VG_HDD_sl2_Ceph/LV_HDD_sl2_OSD_data

      type                      block
      osd id                    7
      cluster fsid              154b75c2-1767-45c5-b7f6-ba7e3a36fc12
      cluster name              ceph
      osd fsid                  d3417340-754e-4e99-ade2-1ebe93a274cf
      db device                 /dev/VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_01
      encrypted                 1
      db uuid                   DMKQ8z-zn6W-08os-acCj-mi1v-xD6c-tXMa6F
      cephx lockbox secret      [REDACTED]
      block uuid                XWXbNF-SZ2B-0jRG-0jNY-GUjr-131X-05v16p
      block device              /dev/VG_HDD_sl2_Ceph/LV_HDD_sl2_OSD_data
      vdo                       0
      crush device class        None
      devices                   /dev/sdb1

  [  db]    /dev/VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_01

      type                      db
      osd id                    7
      cluster fsid              154b75c2-1767-45c5-b7f6-ba7e3a36fc12
      cluster name              ceph
      osd fsid                  d3417340-754e-4e99-ade2-1ebe93a274cf
      db device                 /dev/VG_SSD_sl1_Ceph/LV_SSD_sl1_OSD_db_01
      encrypted                 1
      db uuid                   DMKQ8z-zn6W-08os-acCj-mi1v-xD6c-tXMa6F
      cephx lockbox secret      [REDACTED]
      block uuid                XWXbNF-SZ2B-0jRG-0jNY-GUjr-131X-05v16p
      block device              /dev/VG_HDD_sl2_Ceph/LV_HDD_sl2_OSD_data
      vdo                       0
      crush device class        None
      devices                   /dev/sda4

Volume Groups

[root@f5-422-01 ~]# vgs
  VG              #PV #LV #SN Attr   VSize    VFree   
  VG_HDD_sl2_Ceph   1   1   0 wz--n- <931.51g       0 
  VG_HDD_sl3_Ceph   1   1   0 wz--n-   <1.82t       0 
  VG_SSD_sl1_Ceph   1   2   0 wz--n- <399.17g <239.17g
  VG_f5-422-01      1   5   0 wz--n-   65.00g   39.00g

Logical Volumes

[root@f5-422-01 ~]# lvs
  LV                   VG              Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  LV_HDD_sl2_OSD_data  VG_HDD_sl2_Ceph -wi-ao---- <931.51g                                                    
  LV_HDD_sl3_OSD_data  VG_HDD_sl3_Ceph -wi-ao----   <1.82t                                                    
  LV_SSD_sl1_OSD_db_01 VG_SSD_sl1_Ceph -wi-ao----   80.00g                                                    
  LV_SSD_sl1_OSD_db_02 VG_SSD_sl1_Ceph -wi-ao----   80.00g                                                    
  LV_home              VG_f5-422-01    -wi-ao----    4.00g                                                    
  LV_root              VG_f5-422-01    -wi-ao----    8.00g                                                    
  LV_swap              VG_f5-422-01    -wi-ao----    4.00g                                                    
  LV_var               VG_f5-422-01    -wi-ao----    8.00g                                                    
  LV_var_log           VG_f5-422-01    -wi-ao----    2.00g

Physical Volumes

[root@f5-422-01 ~]# pvs
  PV         VG              Fmt  Attr PSize    PFree   
  /dev/sda3  VG_f5-422-01    lvm2 a--    65.00g   39.00g
  /dev/sda4  VG_SSD_sl1_Ceph lvm2 a--  <399.17g <239.17g
  /dev/sdb1  VG_HDD_sl2_Ceph lvm2 a--  <931.51g       0 
  /dev/sdc1  VG_HDD_sl3_Ceph lvm2 a--    <1.82t       0