Four TerraMaster F5-422 with Ceph
Table of Contents
After installing RHEL 8, I started building a Ceph Nautilus cluster with my four TerraMaster F5-422 nodes.
I was made aware of these cheap, yet equipped with one 10G PCIe NIC and 5 SATA bays, machines in in c’t 2 / 2020 (caveat, paywall)
While my ODROID-HC2 Ceph cluster served well for ceph-ansible training,
I wanted something more beefy for a while now.
I first bought one and ran some tests
I’ll try to avoid having x86 VMs running on my hypervisor that are needed for cluster operation. That way I will be able take the hypervisor down without affecting my Ceph storage. But I’ll have to see how the small NAS boxes perform with colocation.
Hardware Bought
Since Ceph needs a minimum of three nodes to be even remotely useful, I bought four TerraMaster F5-422 nodes. As they come with 10G copper interfaces, I recently added a 10G capable UniFi Switch 16 XG to my homelab (and promptly had to modify the fans).
This cluster will replace my ODROID-HC2 based cluster. I have enough old disks that I can recycle when tearing down the ODROID-HC2 cluster.
Shopping List
In case you plan a similar setup;
- 1 UniFi Switch 16 XG
- 2 Noctua NF-A4x20 FLX
- 4 TerraMaster F5-422
- 4 SO-DIMMs, 8GB
- 4 SATA SSD, 500GB
- 2 DAC cables
- 8 Cat 5e cables
- 4 Cat 6 cables
- 1 4x HDMI, USB keyboard & mouse switch
HDD are old ones that were previously used in the now decommissioned cluster.
Hardware Notes
Since I paid only some 560€ per F5-422 (plus some for RAM and storage), I fully expect some downsides. But not quite as many as I had with my ODROID-HC2 boards.
So far I have not had any unexpected major ones.
Downsides
General Downsides
With only 6 PCIe lanes on the CPU and seemingly the Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
only being attached with 2 lanes I only got between 5.4 Gbits/sec and 5.8 Gbits/sec in initial testing.
With 3 drives up and running, only 2 of the drive LEDs in the chassis light up.
Ceph Relevant Downsides
While I get more than 1 Gbit/s, it’s by no means full 10 Gbit/s.\
Install RHEL8 x86_64
See here for my installation notes. Especially the Firmware settings, the rest is a standard install with all non standard steps listed again in this post.
Log In Via ssh
My kickstart dumps an ssh pubkey for the root user, so on the Ansible control node, as user, I
- verify the ssh key I plan to use for ansible is loaded in the
ssh-agent - verify I can log in to a node as root
ssh-add -l
[...]
ssh root@f5-422-01
Check Ansible Inventory
Since I have both group_vars and host_vars that apply to these hosts, I check of all values is in order.
ansible-inventory -i ~/work/git/HouseNet/ansible/inventories/ceph-F5-422-cluster.ini --list all
ansible-inventory -i ~/work/git/HouseNet/ansible/inventories/ceph-F5-422-cluster.ini --graph all
Verify Ansible Connectivity
On the Ansible control node, as user (-e ansible_user=root is only needed until terramaster-initial-setup-centos8.yml created the ansible user)
ansible -i ../inventories/ceph-F5-422-cluster.ini all -m ping -e ansible_user=root
Initially Configure the F5-422
Create an ansible user with permission to do password-less sudo.
I do this via Playbook.
Running My Initial Setup Playbook
terramaster-initial-setup-rhel8.yml
takes care of
- creating an ansible user
- basic-security-setup
- local settings (timezone etc)
- ensuring EPEL is disabled
- disabling console blanking (normally there is no screen connected, but if I do connect one I want the display not blanked)
As user, on my F31 x96_64 workstation.
ansible-playbook terramaster-initial-setup-rhel8.yml
full playbook terramaster-initial-setup-rhel8.yml
---
# sets up a RHEL 8 minimal install to be ready for ceph-ansible
#
# this is for my home setup, not for production!
- hosts:
- TerraMaster_boxes
become: false
roles:
- pcfe.user_owner
- pcfe.basic_security_setup
- pcfe.housenet
#- pcfe.checkmk
# note though that check-mk-agent is in EPEL but RHCS 4 docs say EPEL needs to be off
# for now I manually installed the rpm with
# dnf install http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/check-mk-agent-1.6.0p5-1.noarch.rpm
# no need for double indirect if you are OK with checking in ak details into git
# this is OK to do if you use an in-house Satellite server and your security policies allow it
# this is not a good idea if you register your systems directly to redhat.com and cannot guarantee that your git remains private
vars_files:
- "vars/subscription-manager-autoattach-ak-secrets.yml"
vars:
ansible_user: root
user_owner: ansible
common_timezone: Europe/Berlin
rhsm_activationkey: "{{ vaulted_rhsm_activationkey }}"
rhsm_org_id: "{{ vaulted_rhsm_org_id }}"
rhsm_pool_ids: "{{ vaulted_rhsm_pool_ids }}"
pre_tasks:
# work around identical UUID on each host as per https://theforeman.org/plugins/katello/nightly/troubleshooting/content_hosts.html
# The TerraMaster F5-422 boxes all have the same system-uuid :-(
# RHSM does not like that, so override to avoid both
# "HTTP error (409 - Conflict): Request failed due to concurrent modification, please re-try.\n"
# and all 4 boxes overriding each other in insights inventory
- name: "RHSM | ensure uuid override is derived from fqdn, F5-422 all have idential uuid in DMI"
copy:
dest: "/etc/rhsm/facts/uuid_override.facts"
owner: "root"
group: "root"
mode: 0644
content: |
{"dmi.system.uuid": "{{ ansible_fqdn | to_uuid }}"}
# https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html-single/installation_guide/index#enabling-the-red-hat-ceph-storage-repositories-install
- name: "RHSM on the RHEL8 boxes"
block:
- name: "RHSM | ensure system is registered with my activation key"
redhat_subscription:
activationkey: "{{ rhsm_activationkey }}"
org_id: "{{ rhsm_org_id }}"
syspurpose:
usage: "Development/Test"
role: ""
service_level_agreement: "Self-Support"
- name: "RHSM | ensure RHEL8 BaseOS repos needed for Ceph are enabled"
rhsm_repository:
name:
- rhel-8-for-x86_64-baseos-rpms
- rhel-8-for-x86_64-appstream-rpms
state: enabled
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
tags: do_subsmgr_register
- name: "RHSM on the RHEL7 boxes"
block:
- name: "RHSM | ensure system is registered with my activation key"
redhat_subscription:
activationkey: "{{ rhsm_activationkey }}"
org_id: "{{ rhsm_org_id }}"
- name: "RHSM | disable all repositories, next task will enable needed repos"
rhsm_repository:
name: '*'
state: disabled
- name: "RHSM | ensure RHEL7 BaseOS repos needed for Ceph are enabled"
rhsm_repository:
name:
- rhel-7-server-rpms
- rhel-7-server-extras-rpms
state: enabled
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "7"
tags: do_subsmgr_register
tasks:
# !!!
#
# if I ever enable EPEL, then I MUST exclude
# - ansible
# - ceph
# in the EPEL repo files to ensure no newer versions of those packages are pulled in from EPEL
#
# exclude = *ceph* nfs-ganesha-rgw rbd-mirror *ansible*
#
# !!!
#
# https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html-single/installation_guide/index#enabling-the-red-hat-ceph-storage-repositories-install
- name: "REPOS | ensure EPEL is disabled"
yum_repository:
name: epel
state: absent
# start by enabling time sync, RHSM operations will fail on too large time delta
# note that this uses chronyd, not ntpd.
- name: "CHRONYD | ensure chrony is installed"
package:
name: chrony
state: present
- name: "CHRONYD | ensure chrony-wait is enabled"
service:
name: chrony-wait
enabled: true
- name: "CHRONYD | ensure chronyd is enabled and running"
service:
name: chronyd
enabled: true
state: started
# enable persistent journal
- name: "JOURNAL | ensure persistent logging for the systemd journal is possible"
file:
path: /var/log/journal
state: directory
owner: root
group: systemd-journal
mode: u=rwx,g=rs,o=rx
# 2.10. Enabling Password-less SSH for Ansible
- name: "SUDO | enable passwordless sudo for ansible user"
copy:
dest: /etc/sudoers.d/ansible
content: |
ansible ALL=NOPASSWD: ALL
owner: root
group: root
mode: 0440
# I admit, the regexp is a search engine hit
- name: "GRUB | ensure console blanking is disabled in defaults file"
lineinfile:
state: present
dest: /etc/default/grub
backrefs: yes
regexp: '^(GRUB_CMDLINE_LINUX=(?!.* consoleblank)\"[^\"]+)(\".*)'
line: '\1 consoleblank=0\2'
(click the triangle to expand)
Perform Hardware Specific Setup
Running My Hardware Specific Playbook
terramaster-general-setup.yml
Besides running some of my roles, it takes care of
- ensuring Wake on LAN is set up for the 1Gig PCIe network interface
- set up watchdog, rngd, tuned
- ensuring cockbit is installed but disabled
- ensuring libvirtd is not running
- ensuring kdump is on
- ensuring I can monitor the host with my Check MK instance
- ensuring
consoleblank=0is set in my default boot options - ensuring I can kickstart the node again via a target in grub
- ensuring the system has the latest updates applied
As user, on my F31 x96_64 workstation.
ansible-playbook terramaster-general-setup.yml
full playbook terramaster-general-setup-rhel8.yml
---
- name: General setup of my Terramaster F5-422 Ceph nodes
hosts:
- TerraMaster_boxes
become: true
roles:
- fedora.linux_system_roles.network
- pcfe.user_owner
- pcfe.basic_security_setup
- pcfe.housenet
- pcfe.comfort
# - pcfe.checkmk
handlers:
- name: Handle running grub2-mkconfig for dist RedHat major vers 9
ansible.builtin.command: grub2-mkconfig -o /boot/grub2/grub.cfg
changed_when: true
- name: Handle running grub2-mkconfig for dist RedHat major vers 8
ansible.builtin.command: grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
changed_when: true
tasks:
# work around identical UUID on each host as per https://theforeman.org/plugins/katello/nightly/troubleshooting/content_hosts.html
# The TerraMaster F5-422 boxes all have the same system-uuid :-(
# RHSM does not like that, so override to avoid both
# "HTTP error (409 - Conflict): Request failed due to concurrent modification, please re-try.\n"
# and all 4 boxes overriding each other in insights inventory
- name: "RHSM | ensure uuid override is derived from fqdn, F5-422 all have idential uuid in DMI"
ansible.builtin.copy:
dest: /etc/rhsm/facts/uuid_override.facts
owner: root
group: root
mode: u=rw,g=r,o=r
content: |
{"dmi.system.uuid": "{{ ansible_fqdn | to_uuid }}"}
# Ensure the packages that the RHEL8 only preflight play from cephadm-ansible would install
# also ensure ceph-selinux is installed, docs say that's in the default set
- name: Ensure needed packages for RHCS5 on RHEL9 are installed
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "9"
ansible.builtin.package:
name:
- chrony
- cephadm
- podman
- ceph-common
- ceph-selinux
state: present
update_cache: false
# Install some tools
- name: "PACKAGE | tool installation, el8 and el9"
ansible.builtin.package:
name:
- pciutils
- usbutils
- fio
- powertop
- tuned
- tuned-utils
- numactl
- iperf3
- tcpdump
- hwloc-gui
- fwupd
state: present
update_cache: false
- name: "PACKAGE | tool installation el9"
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "9"
ansible.builtin.package:
name:
- s-nail
state: present
update_cache: false
- name: "PACKAGE | tool installation el8"
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
ansible.builtin.package:
name:
- teamd
- NetworkManager-team
- mailx
state: present
update_cache: false
# linux-system-roles.network sets static network config (from host_vars)
# but I want the static hostname nailed down too
#
# FIXME; ansible_fqdn is a fact, so that does not help when it's a freshly kickstarted localhost.localdomain
- name: Ensure static hostname is set to FQDN
ansible.builtin.hostname:
name: "{{ ansible_fqdn }}"
use: systemd
# enable WOL manually until https://github.com/linux-system-roles/network/issues/150 is fixed
- name: Ensure Wake On LAN is enable for the upper network card
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
ansible.builtin.lineinfile:
path: /etc/sysconfig/network-scripts/ifcfg-1G_PCIe
create: false
regexp: '^ETHTOOL_OPTS= '
insertafter: '^TYPE=Ethernet'
line: 'ETHTOOL_OPTS="wol g"'
# enable watchdog
# it's a Jun 22 13:12:09 localhost kernel: sbsa-gwdt e0bb0000.gwdt: Initialized with 10s timeout @ 250000000 Hz, action=0.
- name: "WATCHDOG | ensure kernel module wdat_wdt has correct options configured"
ansible.builtin.lineinfile:
path: /etc/modprobe.d/wdat_wdt.conf
create: true
regexp: '^options '
insertafter: '^#options'
line: 'options wdat_wdt nowayout=0'
mode: u=rw,g=r,o=r
# configure both watchdog.service and systemd watchdog, but only use the latter
- name: "PACKAGE | ensure watchdog package is installed"
ansible.builtin.package:
name: watchdog
state: present
update_cache: false
- name: "WATCHDOG | ensure correct watchdog-device is used by watchdog.service"
ansible.builtin.lineinfile:
path: /etc/watchdog.conf
regexp: '^watchdog-device'
insertafter: '^#watchdog-device'
line: 'watchdog-device = /dev/watchdog'
- name: "WATCHDOG | ensure timeout is set to 30 seconds for watchdog.service"
ansible.builtin.lineinfile:
path: /etc/watchdog.conf
regexp: '^watchdog-timeout'
insertafter: '^#watchdog-timeout'
line: 'watchdog-timeout = 30'
# Using systemd watchdog rather than watchdog.service
- name: "WATCHDOG | ensure watchdog.service is disabled"
ansible.builtin.systemd:
name: watchdog.service
state: stopped
enabled: false
# configure systemd watchdog
# c.f. http://0pointer.de/blog/projects/watchdog.html
- name: "SYSTEMD | ensure systemd watchdog is enabled"
ansible.builtin.lineinfile:
path: /etc/systemd/system.conf
regexp: '^RuntimeWatchdogSec'
insertafter: 'EOF'
line: 'RuntimeWatchdogSec=30'
- name: "SYSTEMD | ensure systemd shutdown watchdog is enabled"
ansible.builtin.lineinfile:
path: /etc/systemd/system.conf
regexp: '^ShutdownWatchdogSec'
insertafter: 'EOF'
line: 'ShutdownWatchdogSec=30'
# install and enable rngd
- name: "PACKAGE | ensure rng-tools package is installed"
ansible.builtin.package:
name: rng-tools
state: present
update_cache: false
- name: "RNGD | ensure rngd.service is enabled and started"
ansible.builtin.systemd:
name: rngd.service
state: started
enabled: true
# ensure tuned is set up as I wish
- name: "TUNED | ensure tuned.service is enabled and running"
ansible.builtin.systemd:
name: tuned.service
state: started
enabled: true
- name: "TUNED | check which tuned profile is active"
ansible.builtin.command: tuned-adm active
register: tuned_active_profile
ignore_errors: true
changed_when: false
- name: "TUNED | activate tuned profile {{ tuned_profile }}"
ansible.builtin.command: "tuned-adm profile {{ tuned_profile }}"
when: not tuned_active_profile.stdout is search('Current active profile:' ~ ' ' ~ tuned_profile)
changed_when: true
# install cockpit, but disabled for now
- name: "COCKPIT | ensure packages for https://cockpit-project.org/ are installed"
ansible.builtin.package:
name:
- cockpit
- cockpit-selinux
- cockpit-kdump
- cockpit-storaged
- cockpit-system
state: present
update_cache: false
- name: "COCKPIT | ensure cockpit.socket is stopped and disabled"
ansible.builtin.systemd:
name: cockpit.socket
state: stopped
enabled: false
- name: "COCKPIT | ensure firewalld forbids service cockpit in zone public"
ansible.posix.firewalld:
service: cockpit
zone: public
permanent: true
state: disabled
immediate: true
# disable libvirtd
- name: "Ensure libvirtd.service is disabled and stopped"
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
ansible.builtin.systemd:
name: libvirtd.service
state: stopped
enabled: false
# enable kdump.service since kickstart now creates a sufficiently large /var/crash
# alternatively, you could set up netdump
- name: "Ensure kdump.service is enabled and started"
ansible.builtin.systemd:
name: kdump.service
state: started
enabled: true
# podman
- name: "PACKAGE | ensure podman is installed"
ansible.builtin.package:
name:
- podman
- podman-docker
state: present
# setroubleshoot, see also https://danwalsh.livejournal.com/20931.html
- name: "PACKAGE | ensure setroubleshoot for headless server is installed"
ansible.builtin.package:
name:
- setroubleshoot-server
- setroubleshoot-plugins
state: present
- name: "MONITORING | ensure packages for monitoring are installed"
ansible.builtin.package:
name:
- smartmontools
- hdparm
- check-mk-agent
- lm_sensors
state: present
- name: "MONITORING | ensure firewalld permits 6556 in zone public for check-mk-agent"
ansible.posix.firewalld:
port: 6556/tcp
permanent: true
state: enabled
immediate: true
zone: public
- name: "MONITORING | ensure tarsnap cache is in fileinfo"
ansible.builtin.lineinfile:
path: /etc/check_mk/fileinfo.cfg
line: "/usr/local/tarsnap-cache/cache"
create: true
mode: u=rw,g=r,o=r
- name: "MONITORING | ensure entropy_avail plugin for Check_MK is present"
ansible.builtin.template:
src: templates/check-mk-agent-plugin-entropy_avail.j2
dest: /usr/lib/check_mk_agent/plugins/entropy_avail
mode: u=rwx,g=rx,o=rx
group: root
owner: root
- name: "MONITORING | plugins from running CEE instance"
ansible.builtin.get_url:
url: "http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/plugins/{{ item }}"
dest: "/usr/lib/check_mk_agent/plugins/{{ item }}"
mode: u=rwx,g=rx,o=rx
loop:
- smart
- lvm
- name: "MONITORING | ensure check_mk.socket is started and enabled"
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
ansible.builtin.systemd:
name: check_mk.socket
state: started
enabled: true
- name: "MONITORING | ensure check-mk-agent.socket is started and enabled"
when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "9"
ansible.builtin.systemd:
name: check-mk-agent.socket
state: started
enabled: true
# I admit, the regexp is a search engine hit
# maybe using grubby(8) would be more readable
# yamllint disable rule:line-length
# - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/configuring-kernel-command-line-parameters_managing-monitoring-and-updating-the-kernel#what-is-grubby_configuring-kernel-command-line-parameters
# - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sec-Making_Persistent_Changes_to_a_GRUB_2_Menu_Using_the_grubby_Tool
# yamllint enable rule:line-length
- name: "GRUB | ensure console blanking is disabled in defaults file plus handler"
ansible.builtin.lineinfile:
state: present
dest: /etc/default/grub
backrefs: true
regexp: '^(GRUB_CMDLINE_LINUX=(?!.* consoleblank)\"[^\"]+)(\".*)'
line: '\1 consoleblank=0\2'
notify: "Handle running grub2-mkconfig for dist {{ ansible_distribution }} major vers {{ ansible_distribution_major_version }}"
# Since I do not manage to get these F5-422 to PXE boot, add an entry into grub
# so that I can kickstart the box after this without fiddling with a USB stick
- name: "GRUB | ensure initrd for RHEL 8.9 kickstart is present"
ansible.builtin.get_url:
url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.9/Server/x86_64/os/images/pxeboot/initrd.img"
dest: "/boot/initrd-kickstart-rhel89.img"
mode: u=rw
- name: "GRUB | ensure kernel for RHEL 8.9 kickstart is present"
ansible.builtin.get_url:
url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.9/Server/x86_64/os/images/pxeboot/vmlinuz"
dest: "/boot/vmlinuz-kickstart-rhel89"
mode: u=rwx,g=rx,o=rx
# yamllint disable rule:line-length
- name: "GRUB | ensure kickstarting RHEL 8.9 entry is present"
ansible.builtin.copy:
dest: "/etc/grub.d/11_RHEL8x_kickstart"
owner: "root"
group: "root"
mode: u=rwx,g=rx,o=rx
content: |
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
menuentry "WARNING Kickstart this box with RHEL 8.9 as a F5-422 ceph node WARNING" {
linuxefi /vmlinuz-kickstart-rhel89 ip=dhcp inst.repo=ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.9/Server/x86_64/os inst.ks=ftp://fileserver.internal.pcfe.net/pub/kickstart/RHEL89-TerraMaster-F5-422-ks.cfg
initrdefi /initrd-kickstart-rhel89.img
}
notify: "Handle running grub2-mkconfig for dist {{ ansible_distribution }} major vers {{ ansible_distribution_major_version }}"
# yamllint enable rule:line-length
- name: "GRUB | ensure initrd for RHEL 9.3 kickstart is present"
ansible.builtin.get_url:
url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-9.3/Server/x86_64/os/images/pxeboot/initrd.img"
dest: "/boot/initrd-kickstart-rhel93.img"
mode: u=rw
- name: "GRUB | ensure kernel for RHEL 9.3 kickstart is present"
ansible.builtin.get_url:
url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-9.3/Server/x86_64/os/images/pxeboot/vmlinuz"
dest: "/boot/vmlinuz-kickstart-rhel93"
mode: u=rwx,g=rx,o=rx
# yamllint disable rule:line-length
- name: "GRUB | ensure kickstarting RHEL 9.3 entry is present"
ansible.builtin.copy:
dest: "/etc/grub.d/12_RHEL9x_kickstart"
owner: "root"
group: "root"
mode: u=rwx,g=rx,o=rx
content: |
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
menuentry "WARNING Kickstart this box with RHEL 9.3 as a F5-422 ceph node WARNING" {
linuxefi /vmlinuz-kickstart-rhel93 ip=dhcp inst.repo=ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-9.3/Server/x86_64/os inst.ks=ftp://fileserver.internal.pcfe.net/pub/kickstart/RHEL93-TerraMaster-F5-422-ks.cfg
initrdefi /initrd-kickstart-rhel93.img
}
notify: "Handle running grub2-mkconfig for dist {{ ansible_distribution }} major vers {{ ansible_distribution_major_version }}"
# yamllint enable rule:line-length
# upgrade the box
- name: Package | ensure all updates are applied
ansible.builtin.package:
update_cache: true
name: '*'
state: latest # noqa package-latest
tags: apply_errata
(click the triangle to expand)
Test Watchdog by Forcing Crash
Verify that the watchdog works as expected.
As root, on a F5-422:
echo '1' > /proc/sys/kernel/sysrq
echo 'c' > /proc/sysrq-trigger
As expected, I get an oops output on the console acter the c and,
as previously tested, the box reboots itself after 30 seconds.
Planned Ceph Rollout
I had planned to roll out Red Hat Ceph Storage 4 containerised on theses RHEL 8.1 nodes after I ran into this bug on CentOS Stream.
My Ceph-Specific Playbook
Simply follows the steps from 2. Requirements for Installing Red Hat Ceph Storage in the Red Hat Ceph Storage 4 (aka Nautilus) docs.
As user, on my F31 x96_64 workstation.
ansible-playbook -i ../inventories/ceph-F5-422-cluster.ini ceph-nautilus-F4-422-cluster-prepare-x86.yml
full playbook ceph-nautilus-F4-422-cluster-prepare-x86.yml
This file was removed from my git repo because it's been replaced by another. FIXME: update blog post
old Prepare SSD and HDDs
I used to share one 500 GB SSD between the OS (60GB) and as multiple db for the HDDs with
lvm_volumes:
- data: LV_HDD_bay2_OSD_data
data_vg: VG_HDD_bay2_Ceph
db: LV_SSD_bay1_OSD_db_01
db_vg: VG_SSD_bay1_Ceph
- data: LV_HDD_bay3_OSD_data
data_vg: VG_HDD_bay3_Ceph
db: LV_SSD_bay1_OSD_db_02
db_vg: VG_SSD_bay1_Ceph
For that one has to pre create the VGs and LVs oneself.
As user, on my F31 x96_64 workstation I would run.
ansible-playbook -i ../inventories/ceph-F5-422-cluster.ini terramaster-prep-disks.yml
# sets partitions on my TerraMaster F5-422 boxes
# OS is installed in VG_SSD_bay1_OS (of 65 GBiB) on the SSD in bay 1
# HDDs should be unused until this is run
- hosts:
- TerraMaster_boxes
become: yes
# to avoid using half arsed names like sda, sdb, etc
vars:
storage_bay1: /dev/disk/by-path/pci-0000:00:12.0-ata-1
storage_bay2: /dev/disk/by-path/pci-0000:00:12.0-ata-2
storage_bay3: /dev/disk/by-path/pci-0000:03:00.0-ata-1
storage_bay4: /dev/disk/by-path/pci-0000:03:00.0-ata-2
storage_bay5: /dev/disk/by-path/pci-0000:04:00.0-ata-1
# inspired by https://www.epilis.gr/en/blog/2017/08/09/extending-root-fs-whole-farm/
tasks:
- name: "PARTITIONS | get partition information of OS device in bay 1"
parted:
device: "{{ storage_bay1 }}"
register: bay1_info
- debug: var=bay1_info
# evtl set fact fuer die Partition?
- block:
- name: "PARTITIONS | if more than 160 MiB space left on the SSD, then create a new partition after {{bay1_info.partitions[-1].end}}KiB"
parted:
part_start: "{{ bay1_info.partitions[-1].end }}KiB"
device: "{{ storage_bay1 }}"
number: "{{ bay1_info.partitions[-1].num + 1 }}"
flags: [ lvm ]
label: gpt
state: present
- name: "PARTITIONS | partprobe after change to OS device in bay 1"
command: partprobe
- name: "LVM | create VG_SSD_bay1_Ceph using PV {{ storage_bay1 }}-part{{ bay1_info.partitions[-1].num + 1 }}"
lvg:
vg: VG_SSD_bay1_Ceph
pvs: "{{ storage_bay1 }}-part{{ bay1_info.partitions[-1].num + 1 }}"
when: (bay1_info.partitions[-1].end + 163840) < bay1_info.disk.size
- name: "PARTITIONS | get partition information of the device in bay 2"
parted:
device: "{{ storage_bay2 }}"
register: bay2_info
- debug: var=bay2_info.partitions
- block:
- name: "PARTITIONS | if no partitions on the device in bay 2, then create one covering whole disk"
parted:
part_start: "0%"
part_end: "100%"
device: "{{ storage_bay2 }}"
number: 1
flags: [ lvm ]
label: gpt
state: present
- name: "PARTITIONS | partprobe after change to device in bay 2"
command: partprobe
- name: "LVM | create VG_Ceph_HDD_bay2 using PV {{ storage_bay2 }}-part1"
lvg:
vg: VG_HDD_bay2_Ceph
pvs: "{{ storage_bay2 }}-part1"
- name: "LVM | create LV_HDD_bay2_OSD_data in VG_HDD_bay2_Ceph"
lvol:
vg: VG_HDD_bay2_Ceph
lv: LV_HDD_bay2_OSD_data
size: 100%FREE
- name: "LVM | create LV_SSD_bay1_OSD_db_01 in VG_SSD_bay1_Ceph"
lvol:
vg: VG_SSD_bay1_Ceph
lv: LV_SSD_bay1_OSD_db_01
size: 100G
# VG_SSD_bay1_Ceph is currently 414.26g
# I have 4 slots left for HDD, so just make it 100G
# 80G is roughly 4% of 2TB
# 40G is roughly 4% of 1TB
- name: "PARTITIONS | get partition information of the HDD in bay 2 again"
parted:
device: "{{ storage_bay2 }}"
register: bay2_info
when: not bay2_info.partitions|length
- debug: var=bay2_info.partitions
- name: "PARTITIONS | get partition information of the HDD in bay 3"
parted:
device: "{{ storage_bay3 }}"
register: bay3_info
- debug: var=bay3_info.partitions
- block:
- name: "PARTITIONS | if no partitions on the device in bay 3, then create one covering whole disk"
parted:
part_start: "0%"
part_end: "100%"
device: "{{ storage_bay3 }}"
number: 1
flags: [ lvm ]
label: gpt
state: present
- name: "PARTITIONS | partprobe after change to device in bay 3"
command: partprobe
- name: "LVM | create VG_HDD_bay3_Ceph using PV {{ storage_bay3 }}-part1"
lvg:
vg: VG_HDD_bay3_Ceph
pvs: "{{ storage_bay3 }}-part1"
- name: "LVM | create LV_HDD_bay3_OSD_data in VG_HDD_bay3_Ceph"
lvol:
vg: VG_HDD_bay3_Ceph
lv: LV_HDD_bay3_OSD_data
size: 100%FREE
- name: "LVM | create LV_SSD_bay1_OSD_db_02 in VG_SSD_bay1_Ceph"
lvol:
vg: VG_SSD_bay1_Ceph
lv: LV_SSD_bay1_OSD_db_02
size: 100G
- name: "PARTITIONS | get partition information of the device in bay 3 again"
parted:
device: "{{ storage_bay3 }}"
register: bay3_info
when: not bay3_info.partitions|length
- debug: var=bay3_info.partitions
new dedicated 120 GB SSD for OS
In early June 2020 I wanted to try a deployment with devices, so I did not run …-prep-disks.yml (see https://docs.ceph.com/ceph-ansible/master/osds/scenarios.html for details) and have in my host_vars the following
# avoid using half arsed names like sda, sdb, etc
# mapping is:
# bay 1 /dev/disk/by-path/pci-0000:00:12.0-ata-1
# bay 2 /dev/disk/by-path/pci-0000:00:12.0-ata-2
# bay 3 /dev/disk/by-path/pci-0000:03:00.0-ata-1
# bay 4 /dev/disk/by-path/pci-0000:03:00.0-ata-2
# bay 5 /dev/disk/by-path/pci-0000:04:00.0-ata-1
dmcrypt: True
osd_objectstore: bluestore
osd_scenario: lvm
devices:
- /dev/disk/by-path/pci-0000:00:12.0-ata-2
- /dev/disk/by-path/pci-0000:03:00.0-ata-1
- /dev/disk/by-path/pci-0000:03:00.0-ata-2
- /dev/disk/by-path/pci-0000:04:00.0-ata-1
re-install nodes while Ceph actively in use
My aim was to do invasive maintenance (OS needs to be on another SSD than it currently is on) one each node in turn, and using Ceph’s self-healing to avoid making my CephFS and RBD pools unavailable while doing so.
Since it was quicker for me to kickstart the nodes again to get the OS onto the new 120 GB SSDs,
I simply followed the remove OSDs
procedure for one node, installed it anew, used sgdisk --zap-all … on the 500 GB SSD and the HDDs,
and for the new nodes ran infrastructure-playbooks/add-osd.yml after following the instruction in the playbook
as a first test with node 04 (besides OSDs it only ran grafana).
After this went fine (although the moving back and forth of data while Ceph heals itself of course takes time)
I plan to do
the other nodes with site-container.yml.
At this moment node 03 is being emptied of it’s data and will be re-installed tonight or tomorrow.
Run ceph-ansible
After the …cluster-prepare-x86.yml Playbook dumped inventory, host_vars and group_vars onto the Ansible control node.
Alternatively, create those files manually as per docs. Remember to back them up if you use the maual approach.
As user ansible, on the RHEL8 VM ceph-ansible.internal.pcfe.net
ansible-playbook site-container.yml
Read the Docs WRT Containerized Ceph
Be sure to read https://github.com/ceph/ceph-container/blob/master/src/daemon/README.md
ceph -s with containers
[root@f5-422-01 ~]# systemctl status ceph-mon@f5-422-01.service
● ceph-mon@f5-422-01.service - Ceph Monitor
Loaded: loaded (/etc/systemd/system/ceph-mon@.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2020-03-08 03:00:21 CET; 28min ago
Main PID: 28143 (podman)
Tasks: 12 (limit: 26213)
Memory: 42.9M
CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@f5-422-01.service
└─28143 /usr/bin/podman run --rm --name ceph-mon-f5-422-01 --memory=11694m --cpus=1 -v /var/lib/ceph:/var/>
Mar 08 03:28:45 f5-422-01.internal.pcfe.net podman[28143]: cluster 2020-03-08 03:28:42.049124 mgr.f5-422-02 (mgr.4696>
Mar 08 03:28:45 f5-422-01.internal.pcfe.net podman[28143]: cluster 2020-03-08 03:28:44.050143 mgr.f5-422-02 (mgr.4696>
Mar 08 03:28:50 f5-422-01.internal.pcfe.net podman[28143]: debug 2020-03-08 03:28:50.264 7fbc2c34b700 1 mon.f5-422-0>
Mar 08 03:28:55 f5-422-01.internal.pcfe.net podman[28143]: debug 2020-03-08 03:28:55.265 7fbc2c34b700 1 mon.f5-422-0>
[root@f5-422-01 ~]# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4f44a896ab6d quay.io/rhceph-dev/rhceph:latest-4.3 26 minutes ago Up 26 minutes ago ceph-osd-4
a8a08bac84a5 quay.io/rhceph-dev/rhceph:latest-4.3 26 minutes ago Up 26 minutes ago ceph-osd-0
c2c546eb2e48 quay.io/rhceph-dev/rhceph:latest-4.3 29 minutes ago Up 29 minutes ago ceph-mgr-f5-422-01
414e1f62efa8 quay.io/rhceph-dev/rhceph:latest-4.3 31 minutes ago Up 31 minutes ago ceph-mon-f5-422-01
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph -s
cluster:
id: 385fff79-9c77-4e09-9280-07993eb33988
health: HEALTH_WARN
too few PGs per OSD (9 < min 30)
services:
mon: 3 daemons, quorum f5-422-01,f5-422-02,f5-422-03 (age 29m)
mgr: f5-422-02(active, since 18m), standbys: f5-422-01, f5-422-03
osd: 8 osds: 8 up (since 24m), 8 in (since 24m)
data:
pools: 3 pools, 24 pgs
objects: 0 objects, 0 B
usage: 648 GiB used, 11 TiB / 12 TiB avail
pgs: 24 active+clean
ceph -w with containers
[root@f5-422-01 ~]# podman exec --interactive --tty ceph-mon-f5-422-01 ceph -w
Not sure yet, but this has given my trouble with the rolling upgrade twice now and IIRC I only tried rolling_upgrade.yml two times on this cluster.
see also https://github.com/containers/libpod/issues/3226
try this instead
[root@f5-422-01 ~]# journalctl -f --full -u ceph-mon@f5-422-01.service
Workaround for RHBZ #1793564
Until Bug 1793564 is fixed,
work around as follows after ansible-playbook infrastructure-playbooks/rolling_update.yml finished;
[root@f5-422-01 ~]# podman exec --interactive --tty ceph-mon-f5-422-01 ceph -s
cluster:
id: 385fff79-9c77-4e09-9280-07993eb33988
health: HEALTH_WARN
norebalance flag(s) set
[...]
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph osd unset norebalance
norebalance is unset
Performance Baseline, pool with 8 PGs
I gathered a performance baseline based on the instructions at 6.2. Performance baseline of the RHCS 4 Administration Guide.
Performance Measurements with rados bench, pool with 8 PGs
All rados bench tests were run on f5-422-01.
Sequential Write Test, pool with 8 PGs
podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 write --no-cleanup | tee --append $(date +%F)-rados-test-write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_f5-422-01.internal.pcfe.net_1206
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 36 20 79.9845 80 0.723583 0.576026
2 16 61 45 89.9843 100 0.584442 0.606205
3 16 85 69 91.9845 96 0.531685 0.619067
4 16 112 96 95.9664 108 0.574702 0.633199
5 16 130 114 91.1704 72 0.594669 0.633379
6 16 155 139 92.6393 100 0.66 0.666516
7 16 180 164 93.6886 100 0.155607 0.652325
8 16 199 183 91.4749 76 0.544948 0.662251
9 16 221 205 91.0849 88 0.559885 0.67024
10 16 242 226 90.3741 84 0.132288 0.67139
11 16 270 254 92.3382 112 0.92944 0.675088
12 16 290 274 91.3083 80 0.118037 0.673343
13 16 314 298 91.6678 96 0.316335 0.675676
14 16 345 329 93.9754 124 0.527899 0.667295
15 16 369 353 94.1091 96 0.0945891 0.665796
16 16 394 378 94.4758 100 0.616406 0.668414
17 16 415 399 93.8588 84 0.484051 0.668813
18 16 442 426 94.6434 108 0.305182 0.661079
19 16 470 454 95.556 112 0.215421 0.657006
2020-03-08 03:40:56.073907 min lat: 0.0813909 max lat: 1.54837 avg lat: 0.662172
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 490 474 94.7773 80 0.684891 0.662172
21 16 512 496 94.4475 88 0.562806 0.664983
22 16 535 519 94.3261 92 0.457757 0.668154
23 16 563 547 95.0915 112 0.561656 0.663839
24 16 586 570 94.9612 92 0.347066 0.661026
25 16 613 597 95.482 108 0.727905 0.661553
26 16 643 627 96.4242 120 0.347148 0.656599
27 16 666 650 96.2598 92 0.326133 0.656999
28 16 689 673 96.107 92 0.317507 0.655335
29 16 716 700 96.5156 108 0.429798 0.654834
30 16 740 724 96.4979 96 0.589708 0.657404
31 16 760 744 95.9647 80 0.586795 0.656781
32 16 787 771 96.34 108 0.715718 0.659763
33 16 808 792 95.9655 84 0.688204 0.657657
34 16 832 816 95.9658 96 0.533666 0.660013
35 16 855 839 95.8521 92 0.777156 0.659789
36 16 881 865 96.0779 104 0.603781 0.65963
37 16 909 893 96.5074 112 0.273226 0.658766
38 16 930 914 96.1777 84 0.566452 0.656826
39 16 958 942 96.5829 112 0.543521 0.657998
2020-03-08 03:41:16.082412 min lat: 0.0813909 max lat: 1.54837 avg lat: 0.660309
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 978 962 96.168 80 0.983711 0.660309
41 16 1003 987 96.2603 100 0.837931 0.660551
42 16 1025 1009 96.0633 88 0.32828 0.662518
43 16 1049 1033 96.0615 96 1.02893 0.66174
44 16 1069 1053 95.6961 80 0.248816 0.663268
45 16 1087 1071 95.1692 72 1.0122 0.664249
46 16 1110 1094 95.0999 92 0.199519 0.66612
47 16 1141 1125 95.7142 124 0.10855 0.665186
48 16 1163 1147 95.5527 88 0.338549 0.664097
49 16 1194 1178 96.1326 124 0.436331 0.661901
50 16 1214 1198 95.8096 80 1.05029 0.661884
51 16 1237 1221 95.7344 92 0.772366 0.664174
52 16 1258 1242 95.5084 84 0.641119 0.664483
53 16 1283 1267 95.5927 100 0.931158 0.665329
54 16 1309 1293 95.748 104 0.770038 0.665611
55 16 1335 1319 95.8976 104 0.968497 0.664508
56 16 1353 1337 95.4707 72 0.87869 0.66501
57 16 1382 1366 95.8301 116 0.84359 0.663624
58 16 1406 1390 95.8327 96 0.930628 0.663563
59 16 1426 1410 95.564 80 1.01355 0.664224
2020-03-08 03:41:36.087460 min lat: 0.0813909 max lat: 1.54837 avg lat: 0.663119
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 1454 1438 95.8374 112 0.855286 0.663119
61 16 1479 1463 95.9052 100 0.660708 0.662736
62 16 1502 1486 95.8419 92 0.651422 0.663765
63 16 1525 1509 95.7808 92 0.65479 0.663624
64 16 1548 1532 95.7214 92 0.108128 0.664383
65 16 1570 1554 95.6015 88 0.129301 0.664194
66 16 1596 1580 95.7283 104 0.833698 0.66505
67 16 1616 1600 95.4931 80 0.484905 0.665961
68 16 1649 1633 96.0295 132 0.279223 0.66307
69 16 1676 1660 96.2024 108 0.573273 0.661686
70 16 1700 1684 96.199 96 0.17875 0.661252
71 16 1727 1711 96.3648 108 0.677561 0.660304
72 16 1748 1732 96.1926 84 0.585211 0.661393
73 16 1777 1761 96.4636 116 0.570952 0.659998
74 16 1799 1783 96.3491 88 0.424756 0.661915
75 16 1824 1808 96.3975 100 0.594112 0.661561
76 16 1845 1829 96.2339 84 0.434104 0.661296
77 16 1869 1853 96.2304 96 0.188077 0.660654
78 16 1895 1879 96.3297 104 0.600443 0.661122
79 16 1923 1907 96.5279 112 0.6466 0.660552
2020-03-08 03:41:56.093084 min lat: 0.07524 max lat: 1.54837 avg lat: 0.659295
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 1946 1930 96.4711 92 0.194369 0.659295
81 16 1973 1957 96.6131 108 0.477408 0.659159
82 16 2007 1991 97.093 136 0.700025 0.657076
83 16 2030 2014 97.0315 92 0.115752 0.6568
84 16 2054 2038 97.0188 96 0.492626 0.656201
85 16 2081 2065 97.1476 108 0.260029 0.656981
86 16 2107 2091 97.2271 104 0.196699 0.655703
87 16 2133 2117 97.3047 104 0.683626 0.655943
88 16 2156 2140 97.2443 92 0.505637 0.655697
89 16 2185 2169 97.4544 116 0.509705 0.65502
90 16 2205 2189 97.2601 80 0.101186 0.655156
91 16 2234 2218 97.4657 116 0.581524 0.654454
92 16 2254 2238 97.2755 80 0.964339 0.653804
93 16 2282 2266 97.4336 112 0.674037 0.654363
94 16 2309 2293 97.5458 108 0.663199 0.654833
95 16 2333 2317 97.5292 96 0.446743 0.652527
96 16 2361 2345 97.6796 112 0.546457 0.652748
97 16 2383 2367 97.5795 88 0.969492 0.652821
98 16 2406 2390 97.5223 92 0.771261 0.653418
99 16 2430 2414 97.5065 96 0.656395 0.653307
2020-03-08 03:42:16.098625 min lat: 0.07524 max lat: 1.54837 avg lat: 0.65362
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 2457 2441 97.6112 108 0.782992 0.65362
101 16 2485 2469 97.7534 112 0.623987 0.65313
102 16 2509 2493 97.736 96 0.504933 0.652843
103 16 2531 2515 97.6413 88 0.0907685 0.652572
104 16 2556 2540 97.6639 100 0.571569 0.652897
105 16 2580 2564 97.6479 96 0.969512 0.653634
106 16 2599 2583 97.4434 76 0.622518 0.654013
107 16 2620 2604 97.3175 84 0.721271 0.654591
108 16 2643 2627 97.2681 92 0.900107 0.655839
109 16 2663 2647 97.1095 80 0.570751 0.656755
110 16 2685 2669 97.0264 88 0.389845 0.656463
111 16 2713 2697 97.1611 112 0.83569 0.657452
112 16 2735 2719 97.0789 88 0.0963093 0.657552
113 16 2760 2744 97.1044 100 0.629961 0.65806
114 16 2786 2770 97.1644 104 0.321872 0.657039
115 16 2804 2788 96.9453 72 0.476548 0.657546
116 16 2831 2815 97.0369 108 0.194845 0.65761
117 16 2852 2836 96.9235 84 0.622427 0.657697
118 16 2875 2859 96.8813 92 1.15613 0.658838
119 16 2898 2882 96.8398 92 0.133981 0.658353
2020-03-08 03:42:36.111236 min lat: 0.0735553 max lat: 1.54837 avg lat: 0.658728
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 2925 2909 96.9326 108 0.609364 0.658728
121 16 2945 2929 96.7926 80 0.381071 0.658807
122 16 2970 2954 96.8187 100 0.123735 0.658164
123 16 2996 2980 96.877 104 0.851339 0.658832
124 16 3029 3013 97.16 132 0.441264 0.657168
125 16 3051 3035 97.0866 88 0.826064 0.657295
126 16 3078 3062 97.1731 108 0.499654 0.65676
127 16 3104 3088 97.2262 104 0.437078 0.657114
128 16 3125 3109 97.1228 84 0.218222 0.657184
129 16 3147 3131 97.0518 88 0.686379 0.657279
130 16 3178 3162 97.2589 124 0.507535 0.656603
131 16 3204 3188 97.3101 104 0.539586 0.655695
132 16 3232 3216 97.4209 112 0.873489 0.655031
133 16 3255 3239 97.3798 92 0.340408 0.655164
134 16 3289 3273 97.6678 136 0.345793 0.654292
135 16 3314 3298 97.6821 100 0.529305 0.654292
136 16 3341 3325 97.7576 108 0.48735 0.653234
137 16 3370 3354 97.8906 116 0.263476 0.652577
138 16 3389 3373 97.732 76 0.941394 0.653171
139 16 3411 3395 97.6618 88 0.214443 0.653669
2020-03-08 03:42:56.120600 min lat: 0.0735553 max lat: 1.54837 avg lat: 0.653734
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 3433 3417 97.5927 88 0.674246 0.653734
141 16 3454 3438 97.4962 84 0.617448 0.654569
142 16 3479 3463 97.5137 100 0.218338 0.654403
143 16 3500 3484 97.4179 84 0.853201 0.65507
144 16 3525 3509 97.4356 100 0.680571 0.655173
145 16 3548 3532 97.3981 92 0.575641 0.655856
146 16 3574 3558 97.4432 104 0.121961 0.655444
147 16 3599 3583 97.4604 100 0.553883 0.654605
148 16 3623 3607 97.4502 96 0.517809 0.654811
149 16 3652 3636 97.5745 116 0.518947 0.654115
150 16 3672 3656 97.4572 80 0.663762 0.654778
151 16 3697 3681 97.4738 100 0.75867 0.655086
152 16 3721 3705 97.4639 96 0.511764 0.655137
153 16 3743 3727 97.4017 88 0.512141 0.654982
154 16 3768 3752 97.4184 100 0.705314 0.655127
155 16 3799 3783 97.5896 124 0.271455 0.654865
156 16 3826 3810 97.6561 108 0.584037 0.654706
157 16 3848 3832 97.5945 88 0.332718 0.654546
158 16 3875 3859 97.6601 108 0.744455 0.654254
159 16 3897 3881 97.5984 88 0.45706 0.653767
2020-03-08 03:43:16.129325 min lat: 0.0731889 max lat: 1.54837 avg lat: 0.653318
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 3926 3910 97.7132 116 0.120934 0.653318
161 16 3953 3937 97.7768 108 0.909301 0.653319
162 16 3975 3959 97.7163 88 0.522766 0.653119
163 16 4002 3986 97.7792 108 0.413138 0.653256
164 16 4026 4010 97.7681 96 0.74238 0.653492
165 16 4049 4033 97.7331 92 0.424301 0.653297
166 16 4076 4060 97.7948 108 0.558388 0.653234
167 16 4100 4084 97.784 96 0.161245 0.653062
168 16 4124 4108 97.7731 96 0.1536 0.652695
169 16 4152 4136 97.8572 112 0.677584 0.652788
170 16 4170 4154 97.7049 72 0.565134 0.653094
171 16 4191 4175 97.6247 84 0.749154 0.653765
172 16 4217 4201 97.6617 104 0.60786 0.653897
173 16 4242 4226 97.6743 100 0.404547 0.653995
174 16 4265 4249 97.6416 92 0.68237 0.654205
175 16 4288 4272 97.6092 92 0.682304 0.654389
176 16 4310 4294 97.5546 88 1.3512 0.654532
177 16 4327 4311 97.3874 68 1.04726 0.655169
178 16 4352 4336 97.4018 100 0.367301 0.655149
179 16 4383 4367 97.5502 124 0.838149 0.654715
2020-03-08 03:43:36.136047 min lat: 0.0731889 max lat: 1.81909 avg lat: 0.654916
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 4406 4390 97.5193 92 0.503992 0.654916
181 16 4434 4418 97.5991 112 0.675084 0.654778
182 16 4457 4441 97.5683 92 0.389599 0.654577
183 16 4481 4465 97.5593 96 0.867131 0.654497
184 16 4505 4489 97.55 96 0.526497 0.65482
185 16 4526 4510 97.4767 84 0.724663 0.655057
186 16 4550 4534 97.4687 96 0.763484 0.655434
187 16 4574 4558 97.4607 96 0.627487 0.655562
188 16 4596 4580 97.4103 88 0.884156 0.656023
189 16 4622 4606 97.4451 104 0.426692 0.655285
190 16 4649 4633 97.5004 108 0.197179 0.654889
191 16 4674 4658 97.5134 100 0.597208 0.654921
192 16 4698 4682 97.5054 96 0.269022 0.655068
193 16 4723 4707 97.5181 100 0.635283 0.654982
194 16 4744 4728 97.4483 84 0.0821723 0.6548
195 16 4772 4756 97.5229 112 0.294763 0.654944
196 16 4795 4779 97.4946 92 0.757824 0.655168
197 16 4819 4803 97.4868 96 0.710587 0.655305
198 16 4835 4819 97.3177 64 0.274916 0.6554
199 16 4861 4845 97.3512 104 0.844698 0.655809
2020-03-08 03:43:56.142296 min lat: 0.0731889 max lat: 1.81909 avg lat: 0.655642
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 4891 4875 97.4643 120 0.686626 0.655642
201 16 4923 4907 97.6161 128 0.429688 0.655042
202 16 4941 4925 97.4891 72 0.130057 0.654871
203 16 4967 4951 97.521 104 0.81226 0.655342
204 16 4989 4973 97.4743 88 0.561826 0.655461
205 16 5012 4996 97.4474 92 0.0867183 0.655194
206 16 5037 5021 97.4596 100 0.650913 0.655479
207 16 5060 5044 97.4332 92 1.0034 0.655956
208 16 5083 5067 97.407 92 0.199391 0.655856
209 16 5110 5094 97.4575 108 0.403252 0.655701
210 16 5132 5116 97.4124 88 0.882838 0.655858
211 16 5158 5142 97.4435 104 0.818807 0.65572
212 16 5187 5171 97.531 116 0.463549 0.655693
213 16 5207 5191 97.4486 80 0.0813537 0.655316
214 16 5237 5221 97.5539 120 0.407225 0.655021
215 16 5260 5244 97.5279 92 0.680947 0.654703
216 16 5283 5267 97.5022 92 0.394172 0.65523
217 16 5302 5286 97.4029 76 0.499632 0.655293
218 16 5329 5313 97.4514 108 0.49312 0.655517
219 16 5348 5332 97.3534 76 0.665668 0.655652
2020-03-08 03:44:16.147475 min lat: 0.0722055 max lat: 1.81909 avg lat: 0.656167
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 5370 5354 97.3108 88 0.882733 0.656167
221 16 5397 5381 97.359 108 0.553846 0.656445
222 16 5423 5407 97.3885 104 0.492728 0.656124
223 16 5452 5436 97.4718 116 0.604319 0.655787
224 16 5476 5460 97.4651 96 0.442083 0.655512
225 16 5502 5486 97.4931 104 0.257033 0.655599
226 16 5525 5509 97.4687 92 0.145584 0.655854
227 16 5546 5530 97.4093 84 0.464115 0.655692
228 16 5574 5558 97.4732 112 0.583735 0.655675
229 16 5600 5584 97.5016 104 0.657425 0.655223
230 16 5623 5607 97.4776 92 0.85098 0.655599
231 16 5644 5628 97.4192 84 1.03594 0.655795
232 16 5672 5656 97.482 112 0.857936 0.655887
233 16 5695 5679 97.4584 92 0.520848 0.655773
234 16 5722 5706 97.5033 108 0.798259 0.655544
235 16 5748 5732 97.5309 104 0.421147 0.65538
236 16 5770 5754 97.4904 88 0.89876 0.655434
237 16 5798 5782 97.5501 112 0.607513 0.655208
238 16 5828 5812 97.6443 120 0.61895 0.654877
239 16 5850 5834 97.6039 88 0.678715 0.65488
2020-03-08 03:44:36.157961 min lat: 0.0722055 max lat: 1.81909 avg lat: 0.654761
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 5875 5859 97.6139 100 0.680681 0.654761
241 16 5898 5882 97.5904 92 0.683184 0.65472
242 16 5930 5914 97.7159 128 0.181657 0.654245
243 16 5953 5937 97.6923 92 0.616281 0.654187
244 16 5979 5963 97.7181 104 0.423063 0.653989
245 16 6006 5990 97.76 108 0.537171 0.653939
246 16 6028 6012 97.7203 88 0.398697 0.653941
247 16 6053 6037 97.7294 100 0.564957 0.653728
248 16 6080 6064 97.7707 108 0.890527 0.653698
249 16 6098 6082 97.6671 72 0.60422 0.654047
250 16 6121 6105 97.6444 92 0.596185 0.654216
251 16 6143 6127 97.6058 88 0.185879 0.654253
252 16 6171 6155 97.6629 112 0.576198 0.654299
253 16 6194 6178 97.6404 92 0.433882 0.654087
254 16 6226 6210 97.7598 128 0.834579 0.653943
255 16 6249 6233 97.7371 92 0.702789 0.653851
256 16 6277 6261 97.7927 112 0.370002 0.653654
257 16 6300 6284 97.7701 92 0.0794214 0.653368
258 16 6324 6308 97.7631 96 0.816806 0.653491
259 16 6350 6334 97.7871 104 0.75631 0.653768
2020-03-08 03:44:56.162892 min lat: 0.0722055 max lat: 1.81909 avg lat: 0.653396
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 6376 6360 97.8109 104 0.521866 0.653396
261 16 6402 6386 97.8345 104 0.607561 0.653683
262 16 6420 6404 97.7359 72 0.975247 0.653987
263 16 6438 6422 97.638 72 0.738197 0.653971
264 16 6464 6448 97.662 104 0.674055 0.653959
265 16 6491 6475 97.7004 108 0.232167 0.654134
266 16 6524 6508 97.8292 132 0.411683 0.653385
267 16 6550 6534 97.852 104 0.53891 0.653122
268 16 6576 6560 97.8748 104 0.639489 0.652857
269 16 6599 6583 97.8529 92 0.452174 0.652917
270 16 6616 6600 97.7422 68 0.799574 0.653141
271 16 6645 6629 97.8094 116 0.538319 0.653322
272 16 6670 6654 97.8173 100 1.16929 0.653417
273 16 6693 6677 97.7958 92 1.12137 0.653402
274 16 6717 6701 97.7891 96 0.531808 0.653514
275 16 6744 6728 97.8261 108 0.602214 0.653414
276 16 6765 6749 97.7759 84 0.381065 0.653348
277 16 6791 6775 97.7982 104 0.823354 0.653629
278 16 6812 6796 97.7486 84 0.165915 0.653343
279 16 6841 6825 97.8139 116 0.227242 0.653518
2020-03-08 03:45:16.171457 min lat: 0.0722055 max lat: 1.81909 avg lat: 0.653339
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 6870 6854 97.8785 116 0.396636 0.653339
281 16 6889 6873 97.8005 76 0.439017 0.653331
282 16 6914 6898 97.8083 100 0.537555 0.653592
283 16 6940 6924 97.8301 104 0.852184 0.653501
284 16 6964 6948 97.823 96 1.14132 0.653507
285 16 6986 6970 97.7884 88 0.916644 0.653348
286 16 7014 6998 97.8376 112 0.506791 0.653226
287 16 7040 7024 97.859 104 0.572337 0.653072
288 16 7060 7044 97.7969 80 0.522784 0.653251
289 16 7089 7073 97.8598 116 0.353064 0.653152
290 16 7120 7104 97.9499 124 0.185093 0.652811
291 16 7152 7136 98.053 128 0.110128 0.652291
292 16 7173 7157 98.0049 84 0.683066 0.652086
293 16 7196 7180 97.9843 92 0.836498 0.652206
294 16 7222 7206 98.0047 104 0.417568 0.652181
295 16 7251 7235 98.0655 116 0.423006 0.652047
296 16 7273 7257 98.0315 88 0.555123 0.652055
297 16 7299 7283 98.0515 104 0.571609 0.652229
298 16 7318 7302 97.9775 76 0.687783 0.652374
299 16 7344 7328 97.9967 104 0.654878 0.6524
2020-03-08 03:45:36.181436 min lat: 0.0722055 max lat: 1.81909 avg lat: 0.652362
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 7370 7354 98.0166 104 0.172118 0.652362
301 16 7390 7374 97.9567 80 0.637129 0.652306
302 16 7416 7400 97.9767 104 0.174233 0.652427
303 16 7436 7420 97.9173 80 0.863182 0.652697
304 16 7457 7441 97.8715 84 0.730474 0.652911
305 16 7486 7470 97.9308 116 0.36906 0.652781
306 16 7506 7490 97.8721 80 0.166193 0.652849
307 16 7530 7514 97.8659 96 0.666696 0.65314
308 16 7551 7535 97.8209 84 1.08226 0.653418
309 16 7575 7559 97.8149 96 0.647883 0.653323
310 16 7599 7583 97.809 96 0.680075 0.65348
311 16 7623 7607 97.8031 96 0.392244 0.653554
312 16 7649 7633 97.8229 104 0.441637 0.653621
313 16 7678 7662 97.8809 116 0.283462 0.653335
314 16 7697 7681 97.8112 76 0.721894 0.653253
315 16 7725 7709 97.8561 112 0.345154 0.653423
316 16 7754 7738 97.9134 116 0.543706 0.652996
317 16 7773 7757 97.8443 76 0.394007 0.653071
318 16 7801 7785 97.8888 112 0.873398 0.65298
319 16 7832 7816 97.9703 124 0.492376 0.652724
2020-03-08 03:45:56.186311 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.652228
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 7862 7846 98.0391 120 0.306675 0.652228
321 16 7888 7872 98.0576 104 0.628735 0.652143
322 16 7913 7897 98.0636 100 0.364634 0.651911
323 16 7945 7929 98.1562 128 0.401277 0.651503
324 16 7971 7955 98.174 104 0.58735 0.65123
325 16 7998 7982 98.2041 108 0.610951 0.651142
326 16 8024 8008 98.2217 104 0.12831 0.6509
327 16 8046 8030 98.1904 88 0.13002 0.650769
328 16 8069 8053 98.1714 92 0.758953 0.650784
329 16 8089 8073 98.1161 80 0.651354 0.651252
330 16 8117 8101 98.1581 112 0.292907 0.651184
331 16 8141 8125 98.1515 96 0.151734 0.651369
332 16 8167 8151 98.1691 104 0.651185 0.651512
333 16 8192 8176 98.1745 100 0.642366 0.651203
334 16 8218 8202 98.1919 104 0.62424 0.651161
335 16 8234 8218 98.0898 64 0.629394 0.651505
336 16 8261 8245 98.1193 108 0.641029 0.65164
337 16 8285 8269 98.1128 96 0.578223 0.651536
338 16 8313 8297 98.1537 112 0.505778 0.651474
339 16 8338 8322 98.159 100 0.553791 0.651275
2020-03-08 03:46:16.192894 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.651397
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 8360 8344 98.129 88 0.120949 0.651397
341 16 8384 8368 98.1225 96 0.154636 0.651536
342 16 8407 8391 98.1045 92 0.092277 0.651419
343 16 8434 8418 98.1333 108 0.758751 0.65145
344 16 8455 8439 98.0921 84 0.361099 0.651556
345 16 8484 8468 98.1439 116 0.612094 0.651483
346 16 8507 8491 98.126 92 0.0878817 0.651489
347 16 8533 8517 98.1428 104 0.11184 0.65146
348 16 8554 8538 98.1021 84 0.671261 0.651549
349 16 8578 8562 98.096 96 0.225289 0.651457
350 16 8606 8590 98.1356 112 0.642619 0.651639
351 16 8629 8613 98.1181 92 0.500608 0.651663
352 16 8652 8636 98.1003 92 0.877735 0.651581
353 16 8679 8663 98.1283 108 0.69056 0.651748
354 16 8706 8690 98.156 108 0.638846 0.651503
355 16 8733 8717 98.1834 108 0.499441 0.651573
356 16 8756 8740 98.166 92 0.52004 0.651275
357 16 8778 8762 98.1374 88 0.716946 0.651083
358 16 8814 8798 98.2654 144 0.284235 0.6509
359 16 8832 8816 98.1922 72 0.173222 0.651092
2020-03-08 03:46:36.201780 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.651138
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 8855 8839 98.1749 92 0.0930602 0.651138
361 16 8883 8867 98.2131 112 0.621431 0.65103
362 16 8904 8888 98.1738 84 0.692532 0.650976
363 16 8931 8915 98.2009 108 0.634838 0.650998
364 16 8959 8943 98.2387 112 0.642821 0.650817
365 16 8990 8974 98.3091 124 0.471871 0.650382
366 16 9016 9000 98.3245 104 0.0863513 0.650132
367 16 9042 9026 98.3399 104 0.86341 0.650207
368 16 9066 9050 98.3335 96 0.240845 0.650217
369 16 9092 9076 98.3487 104 0.155596 0.650023
370 16 9114 9098 98.3206 88 0.270602 0.649957
371 16 9144 9128 98.379 120 0.320459 0.649874
372 16 9166 9150 98.3511 88 0.673788 0.650072
373 16 9191 9175 98.3554 100 0.0950065 0.650089
374 16 9213 9197 98.3265 88 0.730065 0.650268
375 16 9242 9226 98.3735 116 0.708608 0.649841
376 16 9264 9248 98.3459 88 0.738357 0.649942
377 16 9297 9281 98.4351 132 0.451988 0.649633
378 16 9325 9309 98.4709 112 0.751524 0.649429
379 16 9347 9331 98.4433 88 0.231051 0.64933
2020-03-08 03:46:56.211448 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.649382
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 9374 9358 98.4684 108 0.894548 0.649382
381 16 9393 9377 98.4094 76 0.258674 0.649351
382 16 9417 9401 98.403 96 0.633855 0.649682
383 16 9446 9430 98.4488 116 0.603489 0.649474
384 16 9472 9456 98.4632 104 0.721873 0.649214
385 16 9493 9477 98.4255 84 0.134118 0.649484
386 16 9519 9503 98.4399 104 0.838446 0.649382
387 16 9546 9530 98.4645 108 0.613837 0.649567
388 16 9568 9552 98.4374 88 0.108603 0.649265
389 16 9594 9578 98.4516 104 0.768979 0.649239
390 16 9624 9608 98.5067 120 0.342783 0.649138
391 16 9647 9631 98.4901 92 0.0767883 0.649052
392 16 9677 9661 98.5449 120 0.8313 0.648956
393 16 9699 9683 98.518 88 0.569558 0.648924
394 16 9719 9703 98.4709 80 1.16451 0.649148
395 16 9739 9723 98.424 80 1.00984 0.64933
396 16 9772 9756 98.5088 132 0.601244 0.649229
397 16 9795 9779 98.4923 92 0.755582 0.649343
398 16 9824 9808 98.5363 116 0.381295 0.649113
399 16 9846 9830 98.5098 88 0.986095 0.648989
2020-03-08 03:47:16.217239 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.649237
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 9871 9855 98.5135 100 0.811227 0.649237
401 16 9890 9874 98.4573 76 0.880914 0.649308
402 16 9910 9894 98.4114 80 0.727828 0.649551
403 16 9936 9920 98.4252 104 0.499338 0.64959
404 16 9961 9945 98.4291 100 0.45856 0.649678
405 16 9977 9961 98.344 64 0.700774 0.649913
406 16 10006 9990 98.3874 116 0.472278 0.650156
407 16 10027 10011 98.3521 84 0.0919184 0.650152
408 16 10049 10033 98.3265 88 0.646558 0.650333
409 16 10080 10064 98.3891 124 0.730495 0.649984
410 16 10101 10085 98.354 84 0.814081 0.650132
411 16 10125 10109 98.3482 96 0.913662 0.650053
412 16 10148 10132 98.3327 92 0.565592 0.650188
413 16 10171 10155 98.3174 92 0.55227 0.650311
414 16 10195 10179 98.3117 96 0.547988 0.650383
415 16 10221 10205 98.3254 104 0.695495 0.650298
416 16 10244 10228 98.3102 92 0.298014 0.65042
417 16 10270 10254 98.3237 104 0.70301 0.650549
418 16 10294 10278 98.3181 96 0.533756 0.650403
419 16 10316 10300 98.2935 88 0.469423 0.650496
2020-03-08 03:47:36.222336 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.650763
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 10335 10319 98.2404 76 0.741445 0.650763
421 16 10359 10343 98.235 96 0.657943 0.650894
422 16 10385 10369 98.2486 104 0.56101 0.650909
423 16 10413 10397 98.281 112 0.240182 0.650805
424 16 10431 10415 98.2188 72 0.585013 0.650777
425 16 10448 10432 98.1477 68 1.33352 0.651027
426 16 10471 10455 98.1331 92 0.550709 0.651319
427 16 10497 10481 98.1468 104 0.587922 0.651463
428 16 10521 10505 98.1417 96 0.574542 0.651528
429 16 10540 10524 98.0901 76 0.500882 0.651556
430 16 10570 10554 98.1409 120 0.244854 0.651698
431 16 10600 10584 98.1915 120 0.559218 0.651374
432 16 10626 10610 98.2049 104 0.406348 0.651151
433 16 10653 10637 98.2272 108 0.206702 0.65111
434 16 10676 10660 98.2128 92 0.672195 0.651088
435 16 10698 10682 98.1892 88 0.782215 0.651299
436 16 10715 10699 98.1199 68 0.924002 0.651413
437 16 10744 10728 98.1608 116 0.651134 0.65159
438 16 10765 10749 98.1283 84 0.504326 0.651649
439 16 10789 10773 98.1234 96 0.659381 0.651608
2020-03-08 03:47:56.230544 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.651727
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 10813 10797 98.1185 96 0.643826 0.651727
441 16 10834 10818 98.0864 84 0.472631 0.65197
442 16 10859 10843 98.0906 100 0.481773 0.652183
443 16 10886 10870 98.1129 108 0.876912 0.651854
444 16 10906 10890 98.072 80 0.112952 0.651808
445 16 10930 10914 98.0672 96 0.588135 0.652
446 16 10949 10933 98.0176 76 0.413525 0.651948
447 16 10981 10965 98.0846 128 0.178786 0.652009
448 16 11007 10991 98.0978 104 0.684228 0.65186
449 16 11031 11015 98.0931 96 0.405251 0.651899
450 16 11060 11044 98.1328 116 0.574425 0.651817
451 16 11083 11067 98.1192 92 0.228945 0.651503
452 16 11108 11092 98.1233 100 0.487363 0.651504
453 16 11138 11122 98.1715 120 0.573114 0.651313
454 16 11168 11152 98.2196 120 0.113332 0.651239
455 16 11190 11174 98.1971 88 0.602596 0.651189
456 16 11220 11204 98.2448 120 0.494584 0.651004
457 16 11243 11227 98.2311 92 0.551401 0.650912
458 16 11265 11249 98.2087 88 0.299751 0.650913
459 16 11290 11274 98.2126 100 0.758831 0.651052
2020-03-08 03:48:16.236538 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.651116
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 11311 11295 98.1817 84 0.591291 0.651116
461 16 11335 11319 98.1768 96 0.727311 0.65137
462 16 11363 11347 98.2067 112 0.652792 0.651309
463 16 11390 11374 98.2278 108 0.560692 0.65122
464 16 11416 11400 98.2402 104 0.829137 0.651021
465 16 11444 11428 98.2698 112 0.163392 0.650791
466 16 11470 11454 98.282 104 0.454778 0.650629
467 16 11494 11478 98.2771 96 0.316587 0.650594
468 16 11521 11505 98.2978 108 0.353619 0.650679
469 16 11543 11527 98.2758 88 0.569877 0.650651
470 16 11566 11550 98.2624 92 0.878457 0.6508
471 16 11591 11575 98.2661 100 0.407657 0.650821
472 16 11611 11595 98.2273 80 0.77719 0.650922
473 16 11637 11621 98.2394 104 0.520483 0.650859
474 16 11666 11650 98.2767 116 0.594113 0.650738
475 16 11685 11669 98.2297 76 0.479718 0.650856
476 16 11713 11697 98.2586 112 0.369176 0.650822
477 16 11743 11727 98.3039 120 0.809787 0.650624
478 16 11766 11750 98.2907 92 0.663079 0.650689
479 16 11783 11767 98.2274 68 0.805232 0.650822
2020-03-08 03:48:36.243342 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.65089
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 11808 11792 98.231 100 0.697284 0.65089
481 16 11837 11821 98.2678 116 0.611617 0.65101
482 16 11855 11839 98.2129 72 1.20229 0.651155
483 16 11879 11863 98.2082 96 0.731059 0.65101
484 16 11911 11895 98.2696 128 0.510453 0.650826
485 16 11940 11924 98.3062 116 0.659076 0.650693
486 16 11962 11946 98.2849 88 0.655587 0.650758
487 16 11986 11970 98.2802 96 0.190026 0.650729
488 16 12015 11999 98.3165 116 0.691374 0.650607
489 16 12038 12022 98.3035 92 0.650487 0.650561
490 16 12065 12049 98.3232 108 0.371408 0.650362
491 16 12087 12071 98.302 88 0.471904 0.650409
492 16 12112 12096 98.3054 100 0.713182 0.650466
493 16 12136 12120 98.3007 96 0.691278 0.65061
494 16 12162 12146 98.3121 104 0.120674 0.650506
495 16 12185 12169 98.2993 92 0.740252 0.65063
496 16 12208 12192 98.2865 92 0.695458 0.650776
497 16 12228 12212 98.2497 80 0.531881 0.650684
498 16 12252 12236 98.2451 96 0.675745 0.650922
499 16 12277 12261 98.2485 100 0.549756 0.650907
2020-03-08 03:48:56.252444 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.650717
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 12304 12288 98.268 108 0.38789 0.650717
501 16 12327 12311 98.2554 92 0.522524 0.650629
502 16 12355 12339 98.2819 112 0.704965 0.650842
503 16 12387 12371 98.3409 128 0.265571 0.650511
504 16 12407 12391 98.3045 80 1.04143 0.650469
505 16 12435 12419 98.3315 112 0.651006 0.650557
506 16 12456 12440 98.3031 84 0.84314 0.650636
507 16 12475 12459 98.2587 76 0.547352 0.650639
508 16 12504 12488 98.2935 116 0.155038 0.650645
509 16 12531 12515 98.3125 108 0.475052 0.650498
510 16 12559 12543 98.3392 112 0.547172 0.650479
511 16 12586 12570 98.358 108 0.265545 0.65027
512 16 12610 12594 98.353 96 0.678192 0.650154
513 16 12635 12619 98.3561 100 0.756847 0.650182
514 16 12662 12646 98.3747 108 0.68959 0.650072
515 16 12690 12674 98.4007 112 0.585789 0.650125
516 16 12709 12693 98.3572 76 0.489568 0.650178
517 16 12737 12721 98.3836 112 0.608358 0.650071
518 16 12767 12751 98.4252 120 0.469216 0.649885
519 16 12788 12772 98.3973 84 0.834859 0.64992
2020-03-08 03:49:16.270759 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.649745
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 12814 12798 98.408 104 0.158556 0.649745
521 16 12843 12827 98.4417 116 0.681851 0.649672
522 16 12869 12853 98.4522 104 0.668717 0.649652
523 16 12893 12877 98.4475 96 0.560833 0.649561
524 16 12914 12898 98.4198 84 0.802026 0.649686
525 16 12938 12922 98.4151 96 0.656702 0.64972
526 16 12962 12946 98.4105 96 0.439403 0.649928
527 16 12980 12964 98.3603 72 0.525097 0.650017
528 16 13011 12995 98.4088 124 0.456089 0.650071
529 16 13040 13024 98.4419 116 0.592353 0.649778
530 16 13064 13048 98.4372 96 0.078142 0.64971
531 16 13086 13070 98.4176 88 0.107893 0.649798
532 16 13112 13096 98.428 104 0.91691 0.649908
533 16 13135 13119 98.4159 92 0.597152 0.649901
534 16 13155 13139 98.3814 80 0.563284 0.649952
535 16 13176 13160 98.3545 84 0.612143 0.650101
536 16 13199 13183 98.3426 92 0.832742 0.650123
537 16 13228 13212 98.3754 116 0.823199 0.650273
538 16 13247 13231 98.3338 76 0.0853203 0.650155
539 16 13280 13264 98.3962 132 0.315826 0.65008
2020-03-08 03:49:36.277297 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.650103
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 13298 13282 98.3473 72 0.953939 0.650103
541 16 13328 13312 98.3872 120 0.713192 0.650181
542 16 13356 13340 98.4123 112 0.505564 0.650028
543 16 13381 13365 98.4152 100 0.552712 0.649885
544 16 13400 13384 98.374 76 0.492521 0.649989
545 16 13426 13410 98.3842 104 0.564447 0.65003
546 16 13451 13435 98.3872 100 1.07467 0.650045
547 16 13474 13458 98.3755 92 0.790483 0.650031
548 16 13500 13484 98.3857 104 0.74705 0.650151
549 16 13522 13506 98.3667 88 0.498086 0.650122
550 16 13551 13535 98.3988 116 0.625565 0.650102
551 16 13571 13555 98.3653 80 0.923903 0.65013
552 16 13595 13579 98.361 96 0.535927 0.650129
553 16 13625 13609 98.4001 120 0.685549 0.65004
554 16 13646 13630 98.3741 84 0.931208 0.650087
555 16 13670 13654 98.3698 96 0.561547 0.650125
556 16 13691 13675 98.3439 84 1.04318 0.650216
557 16 13719 13703 98.3684 112 0.677335 0.650262
558 16 13742 13726 98.3569 92 0.134026 0.650171
559 16 13766 13750 98.3527 96 0.268108 0.650323
2020-03-08 03:49:56.281318 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.650447
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 13787 13771 98.327 84 0.538947 0.650447
561 16 13817 13801 98.3656 120 0.262342 0.650401
562 15 13837 13822 98.3398 84 0.470362 0.650362
563 16 13863 13847 98.3427 100 0.435433 0.650324
564 16 13887 13871 98.3385 96 1.14796 0.65044
565 16 13910 13894 98.3273 92 0.746258 0.650412
566 16 13935 13919 98.3301 100 0.492521 0.650374
567 16 13960 13944 98.333 100 0.3715 0.650398
568 16 13985 13969 98.3359 100 0.579969 0.650587
569 16 14007 13991 98.3177 88 0.555807 0.65055
570 16 14028 14012 98.2925 84 0.623199 0.650649
571 16 14048 14032 98.2604 80 0.442698 0.650731
572 16 14073 14057 98.2634 100 1.16068 0.650952
573 16 14097 14081 98.2594 96 0.760908 0.650853
574 16 14125 14109 98.2833 112 0.137254 0.650796
575 16 14147 14131 98.2654 88 0.781419 0.65081
576 16 14175 14159 98.2892 112 0.420487 0.65086
577 16 14203 14187 98.3129 112 0.437793 0.650675
578 16 14224 14208 98.2881 84 0.411578 0.650664
579 16 14251 14235 98.3049 108 0.244459 0.650607
2020-03-08 03:50:16.287823 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.65054
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 14279 14263 98.3284 112 0.574155 0.65054
581 16 14303 14287 98.3244 96 0.398587 0.650546
582 16 14331 14315 98.3479 112 0.629273 0.650455
583 16 14351 14335 98.3164 80 1.02209 0.650517
584 16 14379 14363 98.3398 112 0.465879 0.65041
585 16 14406 14390 98.3562 108 0.91134 0.650376
586 16 14432 14416 98.3658 104 0.603578 0.650222
587 16 14456 14440 98.3618 96 0.433777 0.650184
588 16 14478 14462 98.3441 88 0.243715 0.650223
589 16 14507 14491 98.374 116 0.150017 0.650202
590 16 14528 14512 98.3497 84 1.1645 0.650245
591 16 14553 14537 98.3523 100 0.68081 0.650382
592 16 14572 14556 98.3145 76 0.582767 0.650425
593 16 14600 14584 98.3375 112 0.284335 0.650249
594 16 14629 14613 98.3672 116 0.521806 0.650191
595 16 14655 14639 98.3766 104 0.780194 0.650188
596 16 14678 14662 98.3658 92 0.642921 0.650252
597 16 14699 14683 98.3417 84 0.496099 0.650398
598 16 14717 14701 98.2976 72 0.749483 0.650579
599 16 14743 14727 98.3071 104 0.748699 0.650638
2020-03-08 03:50:36.295938 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.650637
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 16 14761 14745 98.2629 72 0.55221 0.650637
Total time run: 600.488
Total writes made: 14761
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 98.3266
Stddev Bandwidth: 14.0561
Max bandwidth (MB/sec): 144
Min bandwidth (MB/sec): 64
Average IOPS: 24
Stddev IOPS: 3.51401
Max IOPS: 36
Min IOPS: 16
Average Latency(s): 0.65082
Stddev Latency(s): 0.244896
Max latency(s): 1.81909
Min latency(s): 0.0680884
Total time run: 600.488
Total writes made: 14761
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 98.3266
Stddev Bandwidth: 14.0561
Max bandwidth (MB/sec): 144
Min bandwidth (MB/sec): 64
Average IOPS: 24
Stddev IOPS: 3.51401
Max IOPS: 36
Min IOPS: 16
Average Latency(s): 0.65082
Stddev Latency(s): 0.244896
Max latency(s): 1.81909
Min latency(s): 0.0680884
Sequential Read Test, pool with 8 PGs
podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 seq | tee --append $(date +%F)-rados-test-read-seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 50 34 135.929 136 0.975141 0.284386
2 16 81 65 129.927 124 0.0218552 0.390906
3 15 110 95 126.553 120 0.835151 0.445451
4 16 144 128 127.892 132 0.876989 0.450691
5 16 182 166 132.628 152 0.777139 0.454517
6 16 216 200 133.185 136 0.830861 0.44884
7 16 248 232 132.443 128 0.963055 0.449191
8 16 286 270 134.883 152 0.829292 0.448969
9 16 319 303 134.558 132 0.139848 0.451466
10 16 353 337 134.65 136 0.891417 0.453271
11 16 386 370 134.407 132 0.17349 0.454992
12 16 414 398 132.54 112 0.199415 0.461636
13 16 456 440 135.262 168 0.0210695 0.456778
14 16 492 476 135.879 144 0.0634224 0.45462
15 16 525 509 135.612 132 0.147348 0.456454
16 16 559 543 135.632 136 0.845269 0.456958
17 16 592 576 135.412 132 0.65893 0.460863
18 16 625 609 135.216 132 0.828546 0.460951
19 16 659 643 135.252 136 0.711763 0.462084
2020-03-08 03:56:54.842799 min lat: 0.0203516 max lat: 1.07755 avg lat: 0.46355
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 690 674 134.686 124 0.846478 0.46355
21 16 723 707 134.551 132 0.255897 0.463483
22 16 756 740 134.434 132 0.758799 0.46475
23 15 786 771 133.975 124 0.915678 0.467316
24 16 818 802 133.559 124 0.899369 0.468738
25 16 850 834 133.333 128 0.160319 0.468427
26 16 881 865 132.97 124 0.765729 0.47168
27 16 914 898 132.93 132 0.797315 0.471295
28 16 951 935 133.464 148 0.637504 0.471333
29 16 982 966 133.137 124 0.769804 0.471666
30 15 1020 1005 133.892 156 0.126077 0.46897
31 16 1055 1039 133.959 136 0.148952 0.46799
32 16 1088 1072 133.897 132 0.787927 0.469686
33 16 1121 1105 133.839 132 0.965245 0.469038
34 16 1152 1136 133.547 124 0.908163 0.470386
35 16 1185 1169 133.5 132 0.688508 0.471094
36 16 1215 1199 133.124 120 1.03773 0.471998
37 16 1249 1233 133.199 136 0.867507 0.472192
38 16 1287 1271 133.691 152 0.0462029 0.470585
39 16 1323 1307 133.952 144 0.66804 0.470431
2020-03-08 03:57:14.855548 min lat: 0.0203516 max lat: 1.16678 avg lat: 0.470094
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1358 1342 134.101 140 0.684519 0.470094
41 15 1393 1378 134.331 144 0.191476 0.469144
42 16 1426 1410 134.18 128 0.841362 0.469472
43 16 1462 1446 134.408 144 0.161133 0.468827
44 15 1495 1480 134.438 136 0.0507931 0.468935
45 16 1529 1513 134.383 132 0.72111 0.469962
46 16 1566 1550 134.677 148 0.901066 0.467784
47 15 1600 1585 134.781 140 0.213974 0.468174
48 16 1635 1619 134.805 136 0.0225542 0.467605
49 16 1669 1653 134.827 136 0.23991 0.467815
50 16 1705 1689 135.008 144 0.128391 0.467435
51 16 1736 1720 134.79 124 0.068189 0.468261
52 16 1768 1752 134.658 128 0.695251 0.468518
53 16 1803 1787 134.757 140 0.838286 0.468082
54 16 1840 1824 135 148 0.82736 0.468017
55 16 1873 1857 134.944 132 0.188068 0.467539
56 16 1910 1894 135.175 148 1.03513 0.467152
57 16 1940 1924 134.908 120 0.0673579 0.467887
58 16 1974 1958 134.924 136 0.12003 0.467981
59 16 2007 1991 134.874 132 0.862308 0.468366
2020-03-08 03:57:34.874359 min lat: 0.0203516 max lat: 1.16678 avg lat: 0.466844
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 2045 2029 135.158 152 0.0689746 0.466844
61 16 2077 2061 135.038 128 0.804247 0.467989
62 16 2110 2094 134.989 132 0.189748 0.46756
63 16 2143 2127 134.936 132 0.0489882 0.468168
64 16 2178 2162 135.01 140 0.155104 0.467166
65 16 2213 2197 135.086 140 0.20208 0.468235
66 16 2251 2235 135.34 152 0.939328 0.466502
67 16 2283 2267 135.231 128 0.929973 0.466944
68 16 2315 2299 135.124 128 0.868231 0.46777
69 16 2350 2334 135.194 140 1.03562 0.467395
70 16 2380 2364 134.977 120 0.850697 0.468152
71 16 2418 2402 135.217 152 0.180088 0.467554
72 16 2452 2436 135.227 136 0.0480928 0.467477
73 16 2486 2470 135.237 136 0.997825 0.467046
74 16 2518 2502 135.138 128 0.070269 0.467827
75 16 2551 2535 135.096 132 0.152167 0.467803
76 16 2583 2567 135.001 128 0.794 0.468238
77 15 2615 2600 134.96 132 0.719849 0.469051
78 16 2647 2631 134.818 124 0.71598 0.469157
79 16 2681 2665 134.833 136 0.0365208 0.469306
2020-03-08 03:57:54.886943 min lat: 0.0203516 max lat: 1.22369 avg lat: 0.469596
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 2712 2696 134.697 124 0.837074 0.469596
81 16 2743 2727 134.565 124 0.845693 0.47011
82 16 2777 2761 134.581 136 0.118733 0.470016
83 16 2806 2790 134.356 116 0.756143 0.470865
84 15 2842 2827 134.509 148 0.870993 0.470244
85 16 2873 2857 134.338 120 0.774067 0.470871
86 16 2910 2894 134.496 148 0.0542694 0.470643
87 16 2942 2926 134.421 128 0.927292 0.470743
88 16 2974 2958 134.348 128 0.865166 0.470971
89 16 3012 2996 134.545 152 0.728277 0.470583
90 16 3043 3027 134.428 124 0.20032 0.470135
91 16 3078 3062 134.489 140 0.818342 0.470469
92 16 3109 3093 134.374 124 0.234331 0.470648
93 16 3144 3128 134.435 140 0.811083 0.470936
94 16 3179 3163 134.494 140 0.0224531 0.470709
95 15 3212 3197 134.509 136 0.960089 0.47034
96 15 3244 3229 134.44 128 0.175016 0.470492
97 16 3281 3265 134.539 144 1.06565 0.47031
98 16 3311 3295 134.39 120 0.639112 0.471379
99 16 3344 3328 134.366 132 0.928796 0.470734
2020-03-08 03:58:14.898922 min lat: 0.0203516 max lat: 1.22369 avg lat: 0.471156
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3377 3361 134.342 132 1.00684 0.471156
101 16 3406 3390 134.16 116 0.798939 0.471895
102 16 3442 3426 134.256 144 0.801796 0.47159
103 16 3477 3461 134.312 140 0.736081 0.471432
104 16 3509 3493 134.251 128 0.707635 0.471639
105 16 3541 3525 134.191 128 0.783155 0.47186
106 16 3579 3563 134.343 152 0.0701487 0.471216
107 16 3614 3598 134.396 140 0.757796 0.471319
108 16 3647 3631 134.373 132 0.167515 0.471397
109 16 3679 3663 134.314 128 0.170124 0.471318
110 16 3711 3695 134.257 128 0.0715369 0.471835
111 16 3745 3729 134.272 136 0.0415188 0.471837
112 16 3776 3760 134.18 124 0.148915 0.472048
113 16 3811 3795 134.231 140 0.752372 0.471844
114 16 3846 3830 134.28 140 0.959556 0.471629
115 16 3880 3864 134.295 136 0.0428712 0.471715
116 16 3915 3899 134.344 140 0.0452116 0.47172
117 16 3951 3935 134.423 144 0.176952 0.471427
118 16 3981 3965 134.3 120 0.683724 0.471923
119 15 4014 3999 134.312 136 0.0220373 0.471609
2020-03-08 03:58:34.922421 min lat: 0.0203516 max lat: 1.22369 avg lat: 0.471021
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 4056 4040 134.558 164 0.0383263 0.471021
121 16 4093 4077 134.668 148 0.877943 0.47034
122 16 4129 4113 134.744 144 0.334867 0.470029
123 16 4158 4142 134.59 116 0.779281 0.470851
124 16 4190 4174 134.537 128 1.13204 0.470701
125 16 4224 4208 134.549 136 0.696269 0.470985
126 16 4259 4243 134.592 140 0.692259 0.4708
127 15 4293 4278 134.632 140 0.71671 0.47083
128 16 4326 4310 134.58 128 0.678277 0.471015
129 16 4362 4346 134.652 144 0.136745 0.47043
130 16 4396 4380 134.662 136 0.796352 0.470561
131 16 4429 4413 134.641 132 0.734982 0.470756
132 16 4462 4446 134.621 132 0.414072 0.470607
133 16 4496 4480 134.631 136 0.220275 0.470567
134 16 4528 4512 134.581 128 0.651016 0.471096
135 16 4560 4544 134.532 128 0.321922 0.470934
136 16 4598 4582 134.66 152 0.0383355 0.470739
137 16 4635 4619 134.757 148 0.050658 0.470188
138 16 4671 4655 134.82 144 0.8999 0.470147
139 16 4708 4692 134.915 148 0.678899 0.469833
2020-03-08 03:58:54.936877 min lat: 0.0203516 max lat: 1.22369 avg lat: 0.469671
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 4744 4728 134.979 144 0.216934 0.469671
141 16 4778 4762 134.985 136 0.100927 0.46967
142 16 4810 4794 134.936 128 0.222963 0.469965
143 16 4842 4826 134.887 128 0.786333 0.469959
144 16 4875 4859 134.867 132 0.735611 0.469992
145 16 4908 4892 134.847 132 0.0210776 0.469935
146 16 4942 4926 134.855 136 0.131441 0.469846
147 16 4972 4956 134.754 120 0.959912 0.470109
148 16 5008 4992 134.816 144 0.155662 0.470122
149 16 5042 5026 134.824 136 1.01963 0.470261
150 16 5079 5063 134.911 148 0.0207624 0.469949
151 16 5111 5095 134.865 128 1.03945 0.47001
152 16 5145 5129 134.872 136 0.839832 0.470117
153 16 5174 5158 134.748 116 0.0561273 0.470465
154 16 5204 5188 134.652 120 0.0266074 0.470978
155 16 5236 5220 134.605 128 0.825376 0.470939
156 15 5265 5250 134.51 120 0.622478 0.471615
157 15 5298 5283 134.491 132 0.697012 0.471603
158 16 5331 5315 134.45 128 1.04815 0.471255
159 16 5363 5347 134.409 128 1.04153 0.471713
2020-03-08 03:59:14.952133 min lat: 0.0203516 max lat: 1.22369 avg lat: 0.471873
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 5395 5379 134.369 128 0.930668 0.471873
161 16 5429 5413 134.379 136 0.900192 0.471789
162 16 5465 5449 134.438 144 0.0210947 0.471793
163 16 5495 5479 134.349 120 0.787277 0.472051
164 16 5526 5510 134.283 124 0.924526 0.472028
165 16 5559 5543 134.269 132 0.0551954 0.472396
166 16 5599 5583 134.424 160 0.0781094 0.47185
167 16 5631 5615 134.385 128 0.870821 0.471827
168 16 5664 5648 134.371 132 0.982932 0.471739
169 16 5698 5682 134.38 136 0.722235 0.47203
170 16 5730 5714 134.343 128 0.175746 0.471645
171 16 5763 5747 134.329 132 0.133635 0.472029
172 16 5797 5781 134.338 136 0.783323 0.472232
173 16 5835 5819 134.44 152 0.614553 0.471872
174 15 5869 5854 134.472 140 0.0213479 0.471601
175 16 5908 5892 134.572 152 1.05504 0.47132
176 16 5942 5926 134.574 136 0.0292772 0.471109
177 16 5977 5961 134.604 140 0.596964 0.471326
178 16 6010 5994 134.589 132 0.0807799 0.471159
179 16 6042 6026 134.552 128 0.894737 0.471273
2020-03-08 03:59:34.970362 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.47137
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 6077 6061 134.581 140 0.819267 0.47137
181 16 6109 6093 134.544 128 0.807277 0.471377
182 16 6139 6123 134.464 120 0.794841 0.471707
183 16 6176 6160 134.538 148 0.0670039 0.47139
184 16 6211 6195 134.568 140 0.676913 0.471449
185 16 6245 6229 134.575 136 0.862948 0.471514
186 16 6284 6268 134.689 156 0.187745 0.470781
187 16 6315 6299 134.632 124 0.929776 0.470944
188 16 6350 6334 134.659 140 0.739409 0.471048
189 16 6385 6369 134.687 140 0.786382 0.470928
190 16 6418 6402 134.673 132 0.883835 0.470936
191 16 6450 6434 134.638 128 0.246533 0.47107
192 16 6486 6470 134.684 144 0.0231771 0.471047
193 16 6530 6514 134.898 176 0.843133 0.470284
194 16 6565 6549 134.924 140 0.792298 0.470143
195 16 6598 6582 134.909 132 0.883757 0.470015
196 16 6634 6618 134.955 144 0.995747 0.469922
197 16 6668 6652 134.961 136 0.0534438 0.470077
198 16 6703 6687 134.986 140 0.778019 0.46993
199 15 6737 6722 135.01 140 0.0486843 0.469695
2020-03-08 03:59:54.981825 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.470245
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 6768 6752 134.935 120 0.670818 0.470245
201 16 6798 6782 134.86 120 0.27175 0.470312
202 16 6832 6816 134.866 136 0.871731 0.470364
203 16 6867 6851 134.891 140 0.763679 0.470376
204 16 6897 6881 134.815 120 0.965714 0.470474
205 16 6932 6916 134.841 140 0.114796 0.470554
206 16 6971 6955 134.943 156 0.0479505 0.470198
207 16 7006 6990 134.967 140 0.155196 0.469929
208 16 7034 7018 134.857 112 0.71625 0.470463
209 16 7069 7053 134.881 140 0.145525 0.470375
210 16 7105 7089 134.924 144 0.0696467 0.470195
211 16 7142 7126 134.985 148 0.842074 0.47004
212 16 7175 7159 134.971 132 0.0683049 0.469995
213 16 7205 7189 134.9 120 0.82763 0.470352
214 16 7242 7226 134.961 148 0.769618 0.47012
215 16 7277 7261 134.984 140 0.130375 0.470114
216 16 7305 7289 134.878 112 0.855446 0.470359
217 16 7340 7324 134.901 140 0.0218787 0.470229
218 16 7373 7357 134.888 132 0.301954 0.470317
219 16 7406 7390 134.874 132 0.174339 0.470329
2020-03-08 04:00:14.994503 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.470706
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 7437 7421 134.824 124 0.721956 0.470706
221 16 7472 7456 134.847 140 0.215404 0.470533
222 16 7506 7490 134.851 136 0.478741 0.470636
223 16 7542 7526 134.882 144 0.0498284 0.47043
224 16 7575 7559 134.869 132 0.926292 0.470323
225 16 7607 7591 134.839 128 0.062399 0.470517
226 16 7640 7624 134.826 132 0.974291 0.470412
227 16 7675 7659 134.849 140 0.0410486 0.470404
228 16 7706 7690 134.801 124 0.249063 0.470585
229 16 7741 7725 134.824 140 0.241634 0.470435
230 16 7779 7763 134.898 152 0.934999 0.47046
231 16 7818 7802 134.989 156 0.0222671 0.470106
232 16 7854 7838 135.028 144 0.720556 0.469978
233 16 7887 7871 135.015 132 0.77449 0.470023
234 16 7920 7904 135.001 132 0.787365 0.47
235 16 7956 7940 135.039 144 0.954715 0.469977
236 16 7988 7972 135.008 128 1.09556 0.469822
237 16 8025 8009 135.063 148 0.187537 0.469881
238 16 8057 8041 135.033 128 0.208595 0.469911
239 16 8089 8073 135.004 128 0.820872 0.470072
2020-03-08 04:00:35.019331 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.46997
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 8123 8107 135.008 136 0.153667 0.46997
241 16 8153 8137 134.946 120 0.879839 0.470223
242 16 8185 8169 134.917 128 0.434752 0.470248
243 16 8218 8202 134.905 132 0.652999 0.470429
244 16 8249 8233 134.86 124 0.657258 0.470621
245 16 8280 8264 134.815 124 0.726724 0.470887
246 16 8322 8306 134.95 168 0.918289 0.470133
247 16 8355 8339 134.938 132 0.672391 0.470415
248 16 8386 8370 134.893 124 0.793471 0.47043
249 16 8419 8403 134.882 132 0.941931 0.470427
250 15 8449 8434 134.838 124 0.776849 0.470756
251 16 8481 8465 134.795 124 1.01873 0.470612
252 16 8513 8497 134.768 128 0.961651 0.470841
253 16 8545 8529 134.741 128 0.122275 0.470873
254 16 8580 8564 134.762 140 0.0233841 0.470849
255 16 8611 8595 134.719 124 0.743347 0.471203
256 16 8642 8626 134.677 124 0.726062 0.47128
257 16 8671 8655 134.605 116 0.856562 0.471398
258 16 8703 8687 134.579 128 0.156371 0.471497
259 16 8739 8723 134.615 144 0.765914 0.471528
2020-03-08 04:00:55.026016 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.471454
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 8773 8757 134.619 136 0.0213606 0.471454
261 16 8807 8791 134.625 136 0.162843 0.471297
262 16 8839 8823 134.599 128 0.0211701 0.471523
263 16 8871 8855 134.574 128 0.174515 0.471689
264 16 8905 8889 134.579 136 0.734734 0.471681
265 16 8936 8920 134.539 124 0.746554 0.471834
266 16 8972 8956 134.575 144 0.844828 0.4716
267 15 9007 8992 134.608 144 0.784119 0.471513
268 16 9046 9030 134.672 152 0.0732528 0.471265
269 16 9077 9061 134.632 124 0.868041 0.471379
270 16 9111 9095 134.636 136 0.161968 0.471325
271 16 9144 9128 134.627 132 0.699678 0.471489
272 16 9176 9160 134.602 128 0.071531 0.471503
273 16 9209 9193 134.592 132 0.84155 0.471558
274 16 9246 9230 134.641 148 0.911372 0.471328
275 16 9283 9267 134.689 148 0.0226142 0.471188
276 16 9319 9303 134.722 144 0.0575931 0.471141
277 16 9354 9338 134.741 140 0.131408 0.47107
278 16 9392 9376 134.803 152 0.89028 0.470875
279 16 9425 9409 134.793 132 0.265949 0.470783
2020-03-08 04:01:15.039602 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.470947
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 9458 9442 134.783 132 0.985732 0.470947
281 16 9487 9471 134.716 116 0.68781 0.471327
282 16 9522 9506 134.735 140 1.0122 0.471053
283 16 9558 9542 134.767 144 0.0465632 0.470978
284 16 9590 9574 134.743 128 0.0428525 0.470901
285 16 9623 9607 134.733 132 0.136339 0.471115
286 16 9657 9641 134.737 136 0.959743 0.471061
287 16 9689 9673 134.714 128 0.88085 0.47123
288 16 9719 9703 134.662 120 0.113394 0.471322
289 16 9759 9743 134.75 160 0.0669207 0.471123
290 16 9790 9774 134.712 124 0.825954 0.471225
291 16 9821 9805 134.675 124 0.252587 0.471242
292 16 9857 9841 134.707 144 0.0298389 0.471182
293 16 9885 9869 134.629 112 0.0525508 0.471451
294 15 9916 9901 134.606 128 0.0495012 0.471646
295 16 9949 9933 134.583 128 0.130198 0.471651
296 16 9980 9964 134.547 124 0.108019 0.471882
297 16 10011 9995 134.512 124 0.190732 0.471896
298 16 10041 10025 134.463 120 0.888149 0.471997
299 16 10078 10062 134.508 148 0.700575 0.47202
2020-03-08 04:01:35.049816 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.471865
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 10115 10099 134.553 148 0.781747 0.471865
301 16 10149 10133 134.558 136 0.786359 0.471858
302 16 10183 10167 134.562 136 0.714866 0.471858
303 16 10217 10201 134.567 136 0.352637 0.471757
304 16 10252 10236 134.585 140 0.812156 0.471677
305 16 10285 10269 134.576 132 0.0479317 0.471674
306 16 10319 10303 134.581 136 0.850186 0.471684
307 16 10350 10334 134.546 124 0.776645 0.471896
308 16 10381 10365 134.512 124 0.984267 0.471865
309 16 10415 10399 134.516 136 0.798314 0.471959
310 16 10442 10426 134.431 108 0.798814 0.472244
311 16 10473 10457 134.397 124 0.795969 0.472355
312 15 10508 10493 134.427 144 0.795033 0.472324
313 16 10541 10525 134.406 128 1.01292 0.472366
314 16 10580 10564 134.475 156 0.279062 0.472099
315 16 10617 10601 134.518 148 0.230418 0.471825
316 16 10651 10635 134.522 136 0.146848 0.471878
317 16 10686 10670 134.539 140 0.802798 0.471898
318 16 10720 10704 134.544 136 1.02918 0.471912
319 16 10755 10739 134.561 140 0.791742 0.471798
2020-03-08 04:01:55.057587 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.471884
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 10789 10773 134.565 136 0.0536146 0.471884
321 16 10822 10806 134.557 132 0.161004 0.471857
322 16 10851 10835 134.499 116 0.846112 0.471972
323 15 10888 10873 134.55 152 0.884879 0.471905
324 16 10922 10906 134.542 132 0.251293 0.471865
325 16 10955 10939 134.534 132 0.804962 0.471909
326 16 10993 10977 134.588 152 0.715206 0.471846
327 16 11025 11009 134.567 128 0.58962 0.471899
328 15 11059 11044 134.583 140 0.829034 0.471791
329 16 11093 11077 134.575 132 0.157537 0.4718
330 16 11129 11113 134.603 144 0.300787 0.471535
331 16 11159 11143 134.559 120 0.865221 0.471816
332 16 11194 11178 134.575 140 0.0391026 0.471751
333 16 11233 11217 134.639 156 0.812567 0.471432
334 16 11267 11251 134.643 136 0.700221 0.471611
335 16 11294 11278 134.563 108 0.0464601 0.471761
336 16 11331 11315 134.603 148 0.677002 0.471794
337 16 11369 11353 134.655 152 0.14504 0.471403
338 16 11399 11383 134.611 120 0.877266 0.471665
339 16 11437 11421 134.662 152 0.026671 0.471438
2020-03-08 04:02:15.074288 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.47131
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 11475 11459 134.713 152 0.0435939 0.47131
341 16 11505 11489 134.67 120 0.183064 0.471383
342 16 11536 11520 134.638 124 0.166195 0.471566
343 16 11569 11553 134.631 132 0.0225562 0.471598
344 16 11607 11591 134.681 152 0.0411435 0.471535
345 16 11643 11627 134.708 144 0.831116 0.471329
346 16 11674 11658 134.677 124 0.0240121 0.471392
347 15 11706 11691 134.668 132 0.991573 0.471454
348 16 11744 11728 134.706 148 0.424977 0.471241
349 16 11778 11762 134.71 136 0.868019 0.471374
350 16 11815 11799 134.748 148 0.192133 0.471113
351 16 11848 11832 134.74 132 0.0205455 0.471171
352 16 11885 11869 134.777 148 0.73958 0.471088
353 16 11919 11903 134.78 136 0.855659 0.471055
354 16 11953 11937 134.784 136 1.01404 0.470989
355 16 11988 11972 134.798 140 0.14677 0.471023
356 16 12025 12009 134.835 148 0.16013 0.470892
357 16 12064 12048 134.894 156 0.0759443 0.470737
358 16 12099 12083 134.909 140 0.789609 0.470635
359 16 12133 12117 134.911 136 0.145218 0.470623
2020-03-08 04:02:35.083813 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.470506
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 12168 12152 134.926 140 0.939901 0.470506
361 16 12197 12181 134.873 116 0.691135 0.470821
362 16 12227 12211 134.832 120 0.0236928 0.470827
363 15 12263 12248 134.866 148 0.0758222 0.470817
364 16 12301 12285 134.901 148 0.424673 0.470587
365 16 12332 12316 134.87 124 0.938377 0.470733
366 16 12373 12357 134.95 164 0.0962637 0.470436
367 16 12406 12390 134.942 132 0.86539 0.470478
368 16 12440 12424 134.944 136 1.13545 0.470388
369 16 12475 12459 134.958 140 0.845053 0.470473
370 15 12509 12494 134.965 140 0.888389 0.470394
371 16 12538 12522 134.903 112 0.96583 0.470503
372 16 12573 12557 134.916 140 0.994498 0.470652
373 16 12606 12590 134.908 132 0.334376 0.470671
374 16 12640 12624 134.91 136 0.230143 0.470627
375 16 12674 12658 134.913 136 0.718391 0.470628
376 16 12711 12695 134.948 148 1.00987 0.47044
377 16 12745 12729 134.95 136 0.0438632 0.470521
378 16 12776 12760 134.921 124 0.638258 0.47067
379 16 12815 12799 134.976 156 0.980965 0.470386
2020-03-08 04:02:55.122400 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.470335
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 12851 12835 135 144 0.0365161 0.470335
381 16 12883 12867 134.981 128 0.687467 0.47044
382 15 12914 12899 134.962 128 0.148723 0.470393
383 16 12947 12931 134.943 128 0.147092 0.470552
384 16 12974 12958 134.872 108 0.195706 0.470709
385 16 13010 12994 134.896 144 0.0379402 0.470712
386 16 13042 13026 134.878 128 0.85072 0.470721
387 16 13075 13059 134.87 132 0.780163 0.470841
388 16 13110 13094 134.883 140 0.143982 0.4708
389 16 13145 13129 134.896 140 0.792834 0.470797
390 16 13177 13161 134.879 128 0.668 0.470847
391 16 13212 13196 134.892 140 0.770741 0.470756
392 16 13246 13230 134.894 136 0.0461209 0.470636
393 16 13278 13262 134.877 128 0.689751 0.470764
394 16 13312 13296 134.879 136 0.592166 0.470869
395 16 13345 13329 134.872 132 0.768612 0.47079
396 16 13381 13365 134.895 144 0.886712 0.470734
397 16 13410 13394 134.847 116 0.316061 0.47087
398 16 13440 13424 134.809 120 0.954841 0.471008
399 16 13474 13458 134.812 136 0.889035 0.471037
2020-03-08 04:03:15.138738 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.4713
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 13502 13486 134.755 112 0.694989 0.4713
401 16 13537 13521 134.767 140 0.685174 0.471272
402 16 13571 13555 134.77 136 0.814546 0.471202
403 16 13606 13590 134.783 140 0.178205 0.471167
404 16 13636 13620 134.745 120 0.910459 0.471202
405 16 13668 13652 134.728 128 0.807004 0.471354
406 16 13699 13683 134.702 124 0.163664 0.471398
407 16 13735 13719 134.724 144 1.08022 0.471238
408 16 13767 13751 134.707 128 0.132035 0.471331
409 16 13798 13782 134.681 124 0.807371 0.471481
410 16 13829 13813 134.655 124 0.0629575 0.471532
411 16 13864 13848 134.668 140 0.0542624 0.47147
412 15 13897 13882 134.667 136 0.954487 0.471456
413 15 13938 13923 134.738 164 0.744029 0.471368
414 15 13971 13956 134.731 132 0.0819889 0.471289
415 16 14006 13990 134.734 136 0.750822 0.471307
416 15 14038 14023 134.727 132 0.160715 0.471283
417 16 14070 14054 134.701 124 0.1169 0.471388
418 16 14101 14085 134.675 124 0.0210572 0.471431
419 16 14138 14122 134.707 148 0.677568 0.471434
2020-03-08 04:03:35.166219 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.4712
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 14174 14158 134.729 144 1.14233 0.4712
421 16 14215 14199 134.798 164 0.790375 0.471117
422 16 14250 14234 134.81 140 0.0226378 0.471038
423 16 14281 14265 134.784 124 0.794616 0.47116
424 16 14312 14296 134.758 124 1.08469 0.47117
425 16 14344 14328 134.742 128 0.681334 0.471321
426 16 14380 14364 134.764 144 0.949349 0.471186
427 16 14415 14399 134.776 140 0.817058 0.471177
428 16 14444 14428 134.732 116 0.785967 0.471275
429 16 14477 14461 134.726 132 0.78142 0.471356
430 16 14517 14501 134.785 160 0.0237873 0.471104
431 15 14550 14535 134.787 136 1.05292 0.471088
432 16 14588 14572 134.817 148 1.08526 0.470971
433 16 14626 14610 134.857 152 0.771281 0.470927
434 16 14657 14641 134.832 124 0.7577 0.471031
435 16 14689 14673 134.816 128 0.779094 0.471056
436 16 14716 14700 134.755 108 0.131726 0.471154
437 16 14752 14736 134.776 144 0.293693 0.471108
Total time run: 438.252
Total reads made: 14761
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 134.726
Average IOPS: 33
Stddev IOPS: 2.73859
Max IOPS: 44
Min IOPS: 27
Average Latency(s): 0.471676
Max latency(s): 1.55417
Min latency(s): 0.0203516
Total time run: 438.252
Total reads made: 14761
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 134.726
Average IOPS: 33
Stddev IOPS: 2.73859
Max IOPS: 44
Min IOPS: 27
Average Latency(s): 0.471676
Max latency(s): 1.55417
Min latency(s): 0.0203516
Random Read Test, pool with 8 PGs
podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 rand | tee --append $(date +%F)-rados-test-read-rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 15 52 37 147.942 148 0.158301 0.301189
2 16 87 71 141.952 136 0.832574 0.362475
3 16 118 102 135.759 124 0.654045 0.420607
4 16 152 136 135.704 136 0.204245 0.428888
5 16 186 170 135.74 136 0.729829 0.437042
6 16 218 202 134.436 128 1.05547 0.438213
7 16 255 239 136.361 148 0.0709774 0.4441
8 16 286 270 134.811 124 0.869281 0.448637
9 16 316 300 133.164 120 0.685196 0.457519
10 16 351 335 133.838 140 0.62829 0.459062
11 16 383 367 133.302 128 0.982444 0.459559
12 15 416 401 133.524 136 0.128743 0.460705
13 16 452 436 134.015 140 0.610945 0.462623
14 16 482 466 133.009 120 0.911242 0.463799
15 16 515 499 132.904 132 0.31573 0.465893
16 16 556 540 134.785 164 0.853867 0.459184
17 16 589 573 134.62 132 0.689177 0.46168
18 16 627 611 135.583 152 0.0113234 0.458222
19 16 659 643 135.183 128 0.799105 0.46169
2020-03-08 04:06:56.313003 min lat: 0.0113234 max lat: 1.09869 avg lat: 0.458643
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 694 678 135.422 140 0.310124 0.458643
21 16 727 711 135.258 132 0.650914 0.463073
22 16 757 741 134.565 120 0.885841 0.465206
23 16 792 776 134.799 140 0.786337 0.462671
24 16 827 811 135.015 140 0.936009 0.462674
25 16 863 847 135.373 144 0.842838 0.462295
26 16 896 880 135.238 132 0.19784 0.462491
27 15 929 914 135.264 136 0.116197 0.462619
28 15 961 946 134.998 128 0.0664747 0.464232
29 16 994 978 134.756 128 0.740329 0.46574
30 16 1028 1012 134.797 136 0.805194 0.465616
31 16 1061 1045 134.706 132 0.195299 0.465224
32 16 1093 1077 134.496 128 0.633873 0.467363
33 16 1128 1112 134.662 140 0.855817 0.466699
34 15 1158 1143 134.348 124 0.868035 0.467697
35 16 1192 1176 134.28 132 0.0482176 0.468328
36 16 1226 1210 134.328 136 0.933862 0.467658
37 16 1258 1242 134.156 128 0.967674 0.467292
38 16 1292 1276 134.204 136 0.0855933 0.468221
39 16 1325 1309 134.144 132 0.130788 0.469733
2020-03-08 04:07:16.320111 min lat: 0.00938755 max lat: 1.19962 avg lat: 0.469549
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1356 1340 133.888 124 1.04387 0.469549
41 16 1390 1374 133.936 136 1.03238 0.469286
42 16 1420 1404 133.603 120 0.184339 0.471017
43 16 1456 1440 133.842 144 0.788066 0.470925
44 16 1492 1476 134.069 144 0.987314 0.469536
45 16 1521 1505 133.665 116 0.178718 0.471201
46 16 1555 1539 133.715 136 0.705637 0.471781
47 16 1586 1570 133.508 124 0.212288 0.471136
48 16 1617 1601 133.308 124 0.841423 0.472915
49 16 1651 1635 133.361 136 0.769515 0.473012
50 16 1683 1667 133.253 128 0.905305 0.472691
51 16 1716 1700 133.228 132 0.881412 0.473543
52 16 1755 1739 133.665 156 0.150636 0.471752
53 16 1786 1770 133.481 124 0.955843 0.472042
54 16 1823 1807 133.748 148 0.799171 0.472218
55 16 1856 1840 133.716 132 0.668243 0.472101
56 16 1890 1874 133.756 136 0.196989 0.472061
57 16 1924 1908 133.795 136 0.820321 0.472028
58 16 1953 1937 133.488 116 0.234328 0.472514
59 15 1987 1972 133.597 140 0.793436 0.472705
2020-03-08 04:07:36.330639 min lat: 0.00938755 max lat: 1.19962 avg lat: 0.472024
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 2024 2008 133.769 144 0.0448707 0.472024
61 16 2061 2045 134.002 148 0.72615 0.471522
62 16 2093 2077 133.903 128 0.046765 0.471707
63 16 2125 2109 133.808 128 0.707081 0.472379
64 16 2155 2139 133.592 120 0.724463 0.472926
65 16 2188 2172 133.567 132 0.892171 0.472853
66 16 2219 2203 133.422 124 0.160451 0.473304
67 15 2247 2232 133.158 116 0.836406 0.474656
68 16 2279 2263 133.023 124 0.136684 0.474207
69 15 2309 2294 132.89 124 0.160315 0.475248
70 16 2342 2326 132.818 128 0.0546066 0.475489
71 16 2373 2357 132.693 124 0.738566 0.476682
72 16 2407 2391 132.738 136 0.0981024 0.476519
73 16 2438 2422 132.618 124 0.763401 0.477024
74 16 2470 2454 132.554 128 0.155362 0.476518
75 16 2505 2489 132.652 140 0.252369 0.476325
76 16 2539 2523 132.695 136 0.19797 0.476156
77 16 2579 2563 133.048 160 0.927667 0.475364
78 16 2610 2594 132.932 124 0.165136 0.475321
79 16 2646 2630 133.071 144 0.0831507 0.475251
2020-03-08 04:07:56.343072 min lat: 0.00837324 max lat: 1.241 avg lat: 0.474378
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 2681 2665 133.156 140 0.0659497 0.474378
81 16 2714 2698 133.14 132 1.10988 0.474809
82 16 2747 2731 133.125 132 0.621805 0.475296
83 16 2779 2763 133.063 128 0.0752005 0.475567
84 16 2813 2797 133.097 136 0.0468582 0.4753
85 15 2848 2833 133.225 144 0.126747 0.475067
86 15 2883 2868 133.304 140 0.923802 0.474424
87 16 2917 2901 133.288 132 0.923176 0.474476
88 16 2948 2932 133.183 124 0.70343 0.475175
89 16 2980 2964 133.124 128 0.904401 0.475157
90 16 3013 2997 133.111 132 0.759653 0.475733
91 16 3041 3025 132.879 112 0.15295 0.47627
92 16 3073 3057 132.826 128 0.57827 0.476727
93 16 3107 3091 132.86 136 0.0740786 0.476464
94 16 3143 3127 132.978 144 0.150103 0.475982
95 16 3177 3161 133.009 136 0.263318 0.475629
96 16 3213 3197 133.123 144 0.15989 0.475244
97 16 3250 3234 133.276 148 0.0543193 0.474866
98 16 3287 3271 133.426 148 0.858228 0.474385
99 16 3322 3306 133.493 140 0.0739793 0.474492
2020-03-08 04:08:16.348618 min lat: 0.00836277 max lat: 1.241 avg lat: 0.47448
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3353 3337 133.397 124 0.864282 0.47448
101 16 3384 3368 133.304 124 0.799985 0.474833
102 16 3420 3404 133.409 144 0.722673 0.474931
103 16 3452 3436 133.356 128 0.916916 0.474953
104 16 3484 3468 133.305 128 0.154439 0.475176
105 16 3516 3500 133.254 128 0.908337 0.475083
106 16 3553 3537 133.393 148 0.908646 0.474628
107 15 3584 3569 133.339 128 0.976593 0.474769
108 15 3618 3603 133.364 136 0.0672866 0.474941
109 16 3654 3638 133.424 140 0.946736 0.474684
110 16 3686 3670 133.375 128 0.161426 0.474637
111 16 3716 3700 133.254 120 0.689853 0.475435
112 16 3748 3732 133.207 128 0.859965 0.475279
113 16 3780 3764 133.161 128 0.710995 0.475825
114 16 3813 3797 133.151 132 0.829756 0.475785
115 16 3849 3833 133.245 144 0.062841 0.475646
116 16 3888 3872 133.441 156 0.135025 0.474524
117 16 3920 3904 133.394 128 0.151655 0.474847
118 16 3953 3937 133.382 132 0.860326 0.474784
119 16 3987 3971 133.404 136 1.11299 0.474391
2020-03-08 04:08:36.354024 min lat: 0.00836277 max lat: 1.241 avg lat: 0.475038
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 4019 4003 133.359 128 0.0520461 0.475038
121 16 4052 4036 133.347 132 0.901977 0.475312
122 16 4086 4070 133.369 136 0.204373 0.475127
123 16 4119 4103 133.357 132 0.618922 0.475332
124 16 4157 4141 133.507 152 0.184775 0.475108
125 16 4187 4171 133.399 120 0.126014 0.474958
126 16 4224 4208 133.515 148 0.837533 0.474626
127 15 4260 4245 133.629 148 0.94655 0.474256
128 16 4298 4282 133.74 148 0.783716 0.473807
129 16 4333 4317 133.788 140 0.786668 0.473581
130 16 4364 4348 133.713 124 0.860266 0.473926
131 16 4397 4381 133.7 132 0.794391 0.474063
132 16 4430 4414 133.686 132 0.897462 0.473995
133 16 4464 4448 133.704 136 0.321504 0.47396
134 16 4495 4479 133.631 124 0.816939 0.474315
135 15 4529 4514 133.677 140 0.112762 0.474173
136 16 4562 4546 133.635 128 0.773603 0.474196
137 16 4599 4583 133.74 148 0.175125 0.473752
138 16 4635 4619 133.814 144 0.870978 0.473602
139 16 4668 4652 133.8 132 0.960549 0.473509
2020-03-08 04:08:56.361061 min lat: 0.00836277 max lat: 1.241 avg lat: 0.473051
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 4703 4687 133.843 140 0.204832 0.473051
141 16 4730 4714 133.66 108 0.725876 0.474213
142 16 4766 4750 133.732 144 0.87269 0.473984
143 15 4798 4783 133.72 132 0.778536 0.474156
144 16 4829 4813 133.625 120 0.8839 0.474476
145 16 4862 4846 133.613 132 0.0780186 0.474625
146 16 4890 4874 133.465 112 0.725218 0.475027
147 15 4924 4909 133.509 140 0.167976 0.47468
148 16 4962 4946 133.607 148 0.738299 0.474658
149 16 4998 4982 133.677 144 0.740786 0.474217
150 16 5029 5013 133.612 124 0.700061 0.474532
151 16 5067 5051 133.734 152 0.295676 0.474024
152 16 5097 5081 133.643 120 0.956228 0.474259
153 16 5129 5113 133.606 128 0.95389 0.474352
154 16 5166 5150 133.699 148 0.729123 0.474306
155 16 5199 5183 133.688 132 0.761308 0.474257
156 15 5235 5220 133.78 148 0.828421 0.47418
157 16 5269 5253 133.767 132 0.225284 0.473826
158 16 5300 5284 133.705 124 0.646216 0.474161
159 16 5334 5318 133.719 136 0.714883 0.474307
2020-03-08 04:09:16.366578 min lat: 0.00836277 max lat: 1.28557 avg lat: 0.474035
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 5367 5351 133.708 132 0.859074 0.474035
161 16 5401 5385 133.722 136 0.823972 0.474187
162 16 5437 5421 133.786 144 0.165225 0.473834
163 16 5469 5453 133.75 128 1.04529 0.473949
164 16 5504 5488 133.788 140 0.272428 0.473882
165 16 5536 5520 133.753 128 0.221268 0.473974
166 16 5573 5557 133.838 148 0.709128 0.473896
167 16 5608 5592 133.875 140 0.85748 0.473673
168 16 5635 5619 133.721 108 0.165576 0.474198
169 15 5668 5653 133.734 136 0.0684903 0.47417
170 16 5702 5686 133.724 132 0.778847 0.47429
171 16 5732 5716 133.644 120 0.84184 0.474552
172 16 5764 5748 133.611 128 0.839272 0.474562
173 16 5797 5781 133.601 132 0.171637 0.474537
174 15 5835 5820 133.73 156 0.702629 0.474361
175 16 5872 5856 133.788 144 0.615781 0.474126
176 15 5904 5889 133.778 132 0.732444 0.474273
177 16 5937 5921 133.745 128 0.201052 0.474233
178 15 5975 5960 133.87 156 0.909111 0.47382
179 16 6008 5992 133.837 128 0.920218 0.473774
2020-03-08 04:09:36.370434 min lat: 0.00836277 max lat: 1.28557 avg lat: 0.473779
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 6042 6026 133.849 136 0.805651 0.473779
181 16 6072 6056 133.772 120 0.686971 0.474125
182 16 6109 6093 133.85 148 0.727594 0.473911
183 16 6138 6122 133.753 116 0.669657 0.47424
184 16 6172 6156 133.765 136 0.206898 0.473988
185 16 6209 6193 133.841 148 0.558577 0.47404
186 16 6244 6228 133.873 140 0.131761 0.473847
187 16 6279 6263 133.906 140 0.709064 0.473788
188 16 6309 6293 133.831 120 0.164817 0.473846
189 16 6341 6325 133.8 128 0.783304 0.474048
190 16 6376 6360 133.832 140 0.851214 0.473917
191 16 6407 6391 133.78 124 0.0772549 0.474167
192 16 6447 6431 133.916 160 0.775545 0.473739
193 16 6479 6463 133.885 128 0.82808 0.473782
194 16 6510 6494 133.834 124 0.810658 0.474026
195 16 6546 6530 133.886 144 0.653968 0.473844
196 16 6579 6563 133.876 132 0.829998 0.473774
197 16 6611 6595 133.846 128 0.218761 0.473871
198 16 6644 6628 133.837 132 0.887982 0.473824
199 16 6676 6660 133.807 128 0.803168 0.474039
2020-03-08 04:09:56.379317 min lat: 0.00836277 max lat: 1.28557 avg lat: 0.473846
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 6714 6698 133.898 152 0.878244 0.473846
201 16 6746 6730 133.869 128 0.768501 0.473831
202 16 6778 6762 133.839 128 0.261127 0.47385
203 16 6818 6802 133.968 160 0.825998 0.473531
204 16 6851 6835 133.958 132 0.819022 0.473366
205 16 6884 6868 133.949 132 0.0788361 0.473514
206 16 6918 6902 133.959 136 0.815359 0.473553
207 16 6946 6930 133.853 112 0.840559 0.473911
208 16 6977 6961 133.805 124 0.590258 0.474219
209 16 7012 6996 133.815 140 0.201774 0.474042
210 16 7044 7028 133.787 128 0.642904 0.474265
211 16 7081 7065 133.854 148 0.774118 0.473984
212 16 7117 7101 133.902 144 0.123638 0.473725
213 16 7157 7141 134.024 160 0.518049 0.473464
214 16 7191 7175 134.033 136 0.744141 0.473359
215 16 7227 7211 134.079 144 0.239106 0.473191
216 16 7264 7248 134.143 148 0.0833049 0.472876
217 16 7299 7283 134.169 140 0.0610498 0.472635
218 15 7328 7313 134.104 120 0.842044 0.473032
219 16 7364 7348 134.131 140 0.771269 0.472945
2020-03-08 04:10:16.416138 min lat: 0.00836277 max lat: 1.28557 avg lat: 0.472968
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 7397 7381 134.121 132 0.0725063 0.472968
221 16 7430 7414 134.111 132 0.720154 0.473083
222 16 7460 7444 134.047 120 0.730235 0.47333
223 16 7495 7479 134.074 140 0.0526659 0.47332
224 16 7527 7511 134.046 128 0.283585 0.473148
225 16 7560 7544 134.036 132 0.673437 0.473328
226 16 7597 7581 134.098 148 0.148483 0.473082
227 16 7629 7613 134.07 128 0.800568 0.473115
228 16 7666 7650 134.131 148 0.824191 0.473086
229 15 7698 7683 134.121 132 0.860073 0.473035
230 16 7731 7715 134.095 128 0.991363 0.473112
231 16 7765 7749 134.103 136 0.817214 0.473088
232 16 7798 7782 134.094 132 0.153165 0.473078
233 16 7828 7812 134.033 120 0.288812 0.473226
234 16 7862 7846 134.041 136 0.193275 0.473232
235 16 7895 7879 134.032 132 1.01324 0.473253
236 16 7927 7911 134.007 128 0.0495286 0.473434
237 16 7963 7947 134.049 144 0.0794484 0.47327
238 16 7996 7980 134.04 132 0.700819 0.473372
239 16 8025 8009 133.964 116 0.908209 0.473622
2020-03-08 04:10:36.424949 min lat: 0.00836277 max lat: 1.28557 avg lat: 0.473488
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 8061 8045 134.006 144 0.68111 0.473488
241 15 8096 8081 134.047 144 0.762315 0.473437
242 16 8129 8113 134.022 128 0.832327 0.473471
243 15 8158 8143 133.964 120 0.045964 0.473642
244 16 8194 8178 133.988 140 0.0976633 0.473517
245 16 8225 8209 133.947 124 0.684263 0.473676
246 16 8262 8246 134.004 148 0.721014 0.473495
247 16 8299 8283 134.061 148 0.819546 0.473259
248 16 8332 8316 134.052 132 0.79432 0.473305
249 15 8366 8351 134.075 140 0.048162 0.473156
250 16 8404 8388 134.131 148 0.084001 0.473187
251 16 8434 8418 134.075 120 0.720086 0.473294
252 16 8469 8453 134.098 140 0.654103 0.473274
253 16 8503 8487 134.105 136 0.672926 0.473223
254 16 8539 8523 134.144 144 0.14165 0.473034
255 16 8568 8552 134.073 116 0.921784 0.473101
256 16 8600 8584 134.049 128 1.11883 0.473154
257 16 8628 8612 133.964 112 1.13915 0.47337
258 16 8660 8644 133.94 128 0.935785 0.473647
259 16 8692 8676 133.917 128 0.782985 0.473764
2020-03-08 04:10:56.431865 min lat: 0.00821512 max lat: 1.28557 avg lat: 0.473652
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 8728 8712 133.956 144 0.886369 0.473652
261 16 8762 8746 133.964 136 0.699452 0.473737
262 16 8794 8778 133.941 128 0.689826 0.473825
263 16 8823 8807 133.873 116 0.172319 0.473919
264 16 8858 8842 133.896 140 0.0504963 0.47379
265 16 8890 8874 133.873 128 0.85094 0.473976
266 16 8921 8905 133.836 124 0.711171 0.474214
267 15 8951 8936 133.799 124 0.847536 0.474271
268 16 8983 8967 133.763 124 0.798836 0.474429
269 16 9016 9000 133.755 132 0.931471 0.474381
270 16 9048 9032 133.734 128 0.0520949 0.474247
271 16 9083 9067 133.757 140 0.799474 0.474449
272 16 9116 9100 133.75 132 0.194583 0.47442
273 16 9144 9128 133.67 112 0.967392 0.474648
274 16 9177 9161 133.664 132 0.0933709 0.47475
275 16 9211 9195 133.672 136 0.0567927 0.474748
276 16 9244 9228 133.666 132 0.698015 0.474777
277 16 9278 9262 133.674 136 0.0549317 0.474761
278 16 9313 9297 133.697 140 0.687856 0.474667
279 16 9348 9332 133.719 140 0.873569 0.474444
2020-03-08 04:11:16.440225 min lat: 0.00821512 max lat: 1.28557 avg lat: 0.47439
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 9378 9362 133.67 120 0.0656221 0.47439
281 16 9411 9395 133.663 132 0.238052 0.474677
282 16 9445 9429 133.671 136 0.791446 0.474787
283 15 9474 9459 133.623 120 0.13692 0.474976
284 15 9507 9492 133.617 132 0.00851741 0.474967
285 16 9542 9526 133.625 136 0.639127 0.474988
286 15 9578 9563 133.675 148 0.885825 0.474767
287 16 9613 9597 133.683 136 0.792025 0.474791
288 16 9652 9636 133.76 156 0.0746526 0.474401
289 16 9685 9669 133.754 132 0.855458 0.474457
290 16 9720 9704 133.775 140 0.961866 0.474431
291 16 9752 9736 133.756 128 0.125018 0.474386
292 16 9782 9766 133.708 120 0.673914 0.474715
293 16 9815 9799 133.703 132 0.773804 0.474667
294 16 9848 9832 133.696 132 0.128949 0.474675
295 15 9886 9871 133.772 156 0.0617269 0.474499
296 15 9915 9900 133.711 116 0.741222 0.474653
297 16 9952 9936 133.746 144 0.236945 0.474417
298 16 9988 9972 133.78 144 0.0498526 0.47441
299 16 10025 10009 133.827 148 0.182187 0.474246
2020-03-08 04:11:36.448793 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.474384
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 10054 10038 133.768 116 0.0567709 0.474384
301 16 10091 10075 133.815 148 0.318893 0.474225
302 16 10122 10106 133.782 124 0.82665 0.474335
303 16 10151 10135 133.724 116 0.575964 0.474601
304 15 10184 10169 133.731 136 0.529074 0.474756
305 16 10219 10203 133.738 136 0.813308 0.474582
306 15 10250 10235 133.719 128 0.255949 0.474599
307 16 10293 10277 133.831 168 0.125106 0.474227
308 15 10331 10316 133.902 156 1.02491 0.473992
309 16 10362 10346 133.857 120 0.950659 0.474122
310 16 10395 10379 133.85 132 0.0741445 0.474115
311 16 10428 10412 133.844 132 0.918745 0.474103
312 16 10460 10444 133.825 128 0.6814 0.474257
313 16 10497 10481 133.87 148 0.0990792 0.474041
314 16 10533 10517 133.902 144 0.0709087 0.473945
315 16 10567 10551 133.908 136 0.768953 0.473895
316 16 10599 10583 133.89 128 0.9036 0.47405
317 16 10628 10612 133.833 116 0.0670772 0.474018
318 16 10662 10646 133.84 136 0.13329 0.474204
319 15 10696 10681 133.859 140 0.229383 0.474197
2020-03-08 04:11:56.458317 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.474212
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 10728 10712 133.828 124 0.930734 0.474212
321 16 10761 10745 133.822 132 0.0505491 0.474265
322 16 10798 10782 133.866 148 0.903823 0.474141
323 16 10830 10814 133.848 128 0.807813 0.474186
324 15 10864 10849 133.867 140 0.831532 0.47415
325 16 10900 10884 133.886 140 0.296874 0.473992
326 16 10935 10919 133.905 140 0.063832 0.474013
327 16 10965 10949 133.862 120 0.79645 0.474126
328 16 10998 10982 133.856 132 0.678145 0.474212
329 16 11031 11015 133.85 132 0.626806 0.474218
330 15 11061 11046 133.82 124 0.0471994 0.474352
331 16 11090 11074 133.754 112 0.68372 0.474614
332 16 11125 11109 133.771 140 0.847348 0.474516
333 16 11158 11142 133.766 132 0.75611 0.474536
334 15 11195 11180 133.82 152 0.848292 0.47444
335 16 11227 11211 133.79 124 0.105439 0.474316
336 16 11259 11243 133.773 128 0.0813823 0.474474
337 16 11292 11276 133.768 132 0.0459976 0.474436
338 16 11324 11308 133.75 128 0.893798 0.47453
339 16 11354 11338 133.71 120 0.79884 0.474674
2020-03-08 04:12:16.472073 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.474712
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 15 11385 11370 133.692 128 0.0682948 0.474712
341 16 11417 11401 133.663 124 0.742561 0.474946
342 16 11449 11433 133.647 128 0.185589 0.474875
343 16 11484 11468 133.665 140 0.902721 0.474839
344 16 11516 11500 133.649 128 0.0688423 0.474803
345 16 11552 11536 133.679 144 0.836811 0.474812
346 16 11588 11572 133.708 144 0.137557 0.474792
347 16 11619 11603 133.68 124 0.87439 0.474779
348 16 11658 11642 133.744 156 0.191296 0.474667
349 16 11693 11677 133.762 140 0.702421 0.474598
350 16 11723 11707 133.723 120 0.196983 0.474567
351 16 11754 11738 133.695 124 0.167663 0.47476
352 16 11791 11775 133.736 148 0.109859 0.474685
353 16 11824 11808 133.731 132 0.750969 0.47472
354 15 11854 11839 133.703 124 0.14361 0.474778
355 16 11897 11881 133.799 168 0.688509 0.474499
356 16 11936 11920 133.861 156 0.677836 0.474238
357 16 11968 11952 133.844 128 0.0714125 0.474323
358 16 12007 11991 133.906 156 0.834854 0.474138
359 15 12034 12019 133.845 112 0.043868 0.474195
2020-03-08 04:12:36.478934 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.474357
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 12066 12050 133.817 124 0.855982 0.474357
361 15 12100 12085 133.835 140 0.88804 0.474415
362 15 12139 12124 133.896 156 0.736932 0.474188
363 16 12173 12157 133.89 132 0.707198 0.474193
364 16 12212 12196 133.951 156 0.825003 0.473958
365 15 12249 12234 134 152 0.805106 0.473776
366 16 12284 12268 134.005 136 0.0502388 0.473744
367 16 12315 12299 133.978 124 0.245479 0.473661
368 16 12344 12328 133.929 116 0.867353 0.473996
369 16 12379 12363 133.945 140 0.990894 0.473867
370 16 12413 12397 133.95 136 0.15254 0.473861
371 16 12448 12432 133.967 140 0.734665 0.473869
372 16 12485 12469 134.004 148 0.640108 0.473788
373 16 12518 12502 133.999 132 0.912343 0.473676
374 16 12553 12537 134.015 140 0.71889 0.473748
375 16 12585 12569 133.998 128 1.03926 0.473665
376 16 12624 12608 134.057 156 0.802228 0.473576
377 16 12655 12639 134.03 124 0.141191 0.473628
378 16 12689 12673 134.035 136 0.904441 0.473585
379 16 12725 12709 134.062 144 0.16921 0.473534
2020-03-08 04:12:56.485958 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.473547
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 12758 12742 134.056 132 0.71918 0.473547
381 15 12794 12779 134.092 148 0.225409 0.47336
382 16 12830 12814 134.108 140 0.682743 0.473427
383 15 12861 12846 134.092 128 0.196221 0.473495
384 15 12897 12882 134.118 144 0.662155 0.473363
385 16 12931 12915 134.112 132 0.751263 0.473335
386 16 12967 12951 134.138 144 0.153492 0.473265
387 16 13006 12990 134.194 156 0.154775 0.473071
388 16 13042 13026 134.219 144 0.194374 0.472919
389 16 13077 13061 134.234 140 0.287094 0.472767
390 16 13117 13101 134.3 160 0.62969 0.472774
391 16 13153 13137 134.325 144 1.07428 0.472587
392 16 13185 13169 134.308 128 0.894408 0.472724
393 16 13216 13200 134.282 124 0.713725 0.472802
394 16 13247 13231 134.256 124 0.314803 0.472781
395 15 13279 13264 134.25 132 0.916748 0.472883
396 16 13315 13299 134.263 140 0.911682 0.472751
397 16 13346 13330 134.238 124 0.979752 0.472926
398 16 13382 13366 134.262 144 0.711776 0.472886
399 16 13416 13400 134.266 136 0.0573474 0.472696
2020-03-08 04:13:16.493917 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.472771
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 13450 13434 134.27 136 0.859607 0.472771
401 16 13483 13467 134.264 132 0.667416 0.472849
402 15 13516 13501 134.268 136 0.0684735 0.472756
403 16 13548 13532 134.243 124 0.981625 0.472919
404 16 13580 13564 134.227 128 0.941701 0.472999
405 16 13614 13598 134.231 136 0.846183 0.47295
406 15 13649 13634 134.255 144 0.669232 0.472927
407 16 13684 13668 134.259 136 0.739767 0.472925
408 16 13716 13700 134.244 128 0.762265 0.472941
409 16 13748 13732 134.228 128 0.649967 0.473048
410 16 13780 13764 134.213 128 0.788554 0.47304
411 16 13810 13794 134.178 120 0.162429 0.473092
412 16 13842 13826 134.163 128 0.825973 0.473208
413 16 13872 13856 134.129 120 0.628105 0.473408
414 16 13905 13889 134.124 132 0.0464738 0.473404
415 16 13941 13925 134.147 144 0.139263 0.473245
416 16 13972 13956 134.123 124 0.717724 0.473413
417 16 14001 13985 134.079 116 0.990372 0.473427
418 16 14031 14015 134.046 120 0.076273 0.473565
419 16 14065 14049 134.05 136 0.822905 0.473641
2020-03-08 04:13:36.502447 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.473569
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 14100 14084 134.064 140 0.888226 0.473569
421 16 14135 14119 134.079 140 0.863827 0.473574
422 15 14169 14154 134.093 140 0.764471 0.4735
423 15 14205 14190 134.113 144 0.0485052 0.47338
424 16 14239 14223 134.108 132 0.880788 0.473372
425 16 14273 14257 134.112 136 0.851865 0.473417
426 16 14308 14292 134.126 140 0.778321 0.473365
427 16 14336 14320 134.074 112 0.699846 0.473582
428 16 14376 14360 134.135 160 0.662047 0.473371
429 16 14409 14393 134.128 132 0.80638 0.473395
430 16 14441 14425 134.114 128 0.661736 0.473451
431 16 14475 14459 134.118 136 0.821582 0.473375
432 16 14511 14495 134.141 144 0.623924 0.473341
433 16 14544 14528 134.136 132 0.133769 0.473283
434 16 14580 14564 134.159 144 0.0579914 0.47326
435 16 14612 14596 134.145 128 0.047155 0.4733
436 15 14643 14628 134.131 128 0.0690828 0.473359
437 16 14676 14660 134.116 128 0.886116 0.473405
438 16 14712 14696 134.139 144 0.763484 0.473301
439 16 14745 14729 134.134 132 0.895585 0.473333
2020-03-08 04:13:56.523028 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.473417
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 14776 14760 134.11 124 1.01057 0.473417
441 16 14808 14792 134.096 128 0.247927 0.473425
442 16 14841 14825 134.091 132 0.742684 0.473471
443 16 14875 14859 134.095 136 0.760651 0.473463
444 16 14904 14888 134.055 116 0.847611 0.473571
445 16 14936 14920 134.041 128 0.180171 0.473594
446 16 14974 14958 134.081 152 0.0698579 0.473498
447 16 15010 14994 134.103 144 0.158024 0.473394
448 16 15044 15028 134.107 136 0.0708973 0.473373
449 16 15080 15064 134.129 144 0.720252 0.473348
450 16 15116 15100 134.151 144 0.258729 0.473282
451 16 15148 15132 134.137 128 0.0758505 0.473335
452 16 15185 15169 134.168 148 0.196685 0.473168
453 16 15222 15206 134.198 148 0.127752 0.473132
454 16 15259 15243 134.229 148 0.0548841 0.472988
455 16 15294 15278 134.241 140 0.195632 0.472949
456 16 15326 15310 134.228 128 0.851891 0.472973
457 16 15361 15345 134.24 140 0.889282 0.472938
458 16 15396 15380 134.253 140 0.0482713 0.472848
459 16 15432 15416 134.274 144 0.653691 0.472897
2020-03-08 04:14:16.528328 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.472993
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 15462 15446 134.243 120 0.717287 0.472993
461 16 15495 15479 134.238 132 0.181655 0.472955
462 16 15528 15512 134.232 132 0.836749 0.472965
463 16 15559 15543 134.21 124 0.912579 0.473044
464 16 15597 15581 134.248 152 0.940377 0.472942
465 15 15629 15614 134.243 132 0.690721 0.473003
466 16 15666 15650 134.264 144 0.783025 0.472897
467 16 15700 15684 134.267 136 0.950257 0.472817
468 16 15729 15713 134.228 116 0.0454483 0.472903
469 16 15766 15750 134.257 148 0.140922 0.472863
470 16 15801 15785 134.269 140 0.0680671 0.47289
471 16 15834 15818 134.264 132 0.82093 0.472864
472 16 15869 15853 134.276 140 0.928222 0.472811
473 16 15906 15890 134.305 148 0.678418 0.472805
474 16 15943 15927 134.334 148 0.800747 0.472668
475 16 15973 15957 134.304 120 0.908088 0.472756
476 16 16005 15989 134.29 128 0.619162 0.472867
477 16 16035 16019 134.26 120 0.130745 0.472862
478 16 16071 16055 134.281 144 0.0838108 0.472889
479 16 16106 16090 134.293 140 0.0818253 0.472724
2020-03-08 04:14:36.538910 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.472671
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 16142 16126 134.313 144 0.78565 0.472671
481 16 16175 16159 134.308 132 0.0648891 0.472641
482 16 16210 16194 134.32 140 0.142354 0.472684
483 15 16247 16232 134.356 152 0.690424 0.472611
484 16 16280 16264 134.343 128 0.620573 0.472676
485 16 16312 16296 134.33 128 1.01156 0.472619
486 16 16347 16331 134.341 140 0.787834 0.472603
487 16 16382 16366 134.353 140 0.04787 0.47261
488 16 16414 16398 134.34 128 0.216779 0.472636
489 16 16446 16430 134.327 128 0.272641 0.472565
490 16 16481 16465 134.338 140 0.911449 0.472617
491 16 16511 16495 134.309 120 0.134488 0.472747
492 16 16543 16527 134.296 128 1.02804 0.472679
493 16 16573 16557 134.267 120 0.839804 0.472827
494 16 16607 16591 134.271 136 0.769509 0.47291
495 16 16640 16624 134.266 132 0.303948 0.472781
496 16 16678 16662 134.302 152 0.0835276 0.47275
497 16 16714 16698 134.321 144 0.786149 0.472696
498 16 16745 16729 134.3 124 0.707991 0.472805
499 16 16777 16761 134.288 128 0.22756 0.472735
2020-03-08 04:14:56.543533 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.47277
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 16809 16793 134.275 128 0.172409 0.47277
501 16 16840 16824 134.254 124 0.0822042 0.472909
502 16 16873 16857 134.25 132 0.795518 0.47294
503 16 16909 16893 134.269 144 0.0695127 0.472889
504 16 16944 16928 134.28 140 0.794149 0.472859
505 16 16981 16965 134.307 148 0.655628 0.472754
506 16 17010 16994 134.271 116 0.580916 0.472939
507 16 17048 17032 134.306 152 0.660026 0.472807
508 16 17084 17068 134.325 144 0.855374 0.472647
509 16 17116 17100 134.313 128 0.350447 0.47269
510 16 17146 17130 134.285 120 0.748529 0.472831
511 16 17183 17167 134.311 148 0.757828 0.472747
512 16 17216 17200 134.307 132 0.906383 0.472754
513 16 17247 17231 134.287 124 0.761509 0.472837
514 16 17280 17264 134.282 132 0.837028 0.47284
515 15 17312 17297 134.278 132 0.968347 0.472885
516 15 17344 17329 134.265 128 1.11924 0.47286
517 16 17376 17360 134.246 124 0.745734 0.472992
518 16 17413 17397 134.272 148 0.701706 0.472932
519 16 17442 17426 134.237 116 0.611102 0.473064
2020-03-08 04:15:16.548746 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.473031
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 17474 17458 134.225 128 0.840809 0.473031
521 15 17506 17491 134.22 132 0.222493 0.473045
522 16 17543 17527 134.239 144 0.901258 0.472949
523 16 17581 17565 134.273 152 0.81055 0.472926
524 16 17616 17600 134.284 140 0.0574539 0.472791
525 16 17653 17637 134.31 148 1.00109 0.472656
526 16 17688 17672 134.321 140 0.804263 0.472727
527 16 17720 17704 134.309 128 0.148273 0.472726
528 16 17754 17738 134.312 136 0.701567 0.472733
529 16 17785 17769 134.292 124 0.710798 0.47282
530 16 17816 17800 134.273 124 0.199646 0.472798
531 16 17845 17829 134.238 116 0.98901 0.472893
532 16 17883 17867 134.272 152 0.0550675 0.472875
533 16 17913 17897 134.245 120 0.754817 0.472953
534 16 17948 17932 134.255 140 0.939429 0.472884
535 16 17981 17965 134.251 132 1.12066 0.472872
536 16 18015 17999 134.254 136 0.218078 0.472926
537 16 18049 18033 134.257 136 0.181749 0.472942
538 16 18082 18066 134.253 132 0.73172 0.472987
539 16 18116 18100 134.256 136 0.965622 0.472895
2020-03-08 04:15:36.553794 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.47297
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 18148 18132 134.245 128 0.0514879 0.47297
541 16 18181 18165 134.241 132 0.175265 0.472992
542 16 18213 18197 134.229 128 0.0910136 0.473062
543 16 18246 18230 134.225 132 0.808678 0.47307
544 16 18276 18260 134.199 120 0.167186 0.473168
545 16 18314 18298 134.231 152 0.735309 0.473034
546 16 18349 18333 134.242 140 0.658261 0.47304
547 16 18385 18369 134.259 144 0.0839557 0.472923
548 16 18418 18402 134.255 132 0.782334 0.472971
549 16 18452 18436 134.258 136 0.202949 0.472862
550 16 18487 18471 134.269 140 0.680696 0.47295
551 15 18521 18506 134.279 140 0.138018 0.472799
552 16 18552 18536 134.253 120 0.962779 0.472901
553 16 18586 18570 134.256 136 0.789027 0.47295
554 16 18616 18600 134.231 120 0.133948 0.472952
555 16 18653 18637 134.255 148 0.342443 0.472917
556 16 18688 18672 134.266 140 0.0801631 0.472948
557 16 18718 18702 134.24 120 0.70385 0.473009
558 16 18753 18737 134.25 140 0.156122 0.472998
559 16 18784 18768 134.232 124 0.0722608 0.47304
2020-03-08 04:15:56.557483 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.473061
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 18816 18800 134.221 128 0.872465 0.473061
561 15 18852 18837 134.245 148 0.861297 0.472987
562 16 18886 18870 134.241 132 0.907389 0.473017
563 16 18919 18903 134.236 132 0.829337 0.47301
564 16 18952 18936 134.232 132 0.737176 0.473067
565 16 18984 18968 134.221 128 0.0678889 0.473082
566 16 19020 19004 134.239 144 0.0815913 0.473049
567 16 19053 19037 134.235 132 0.822984 0.473047
568 16 19091 19075 134.266 152 0.683587 0.472959
569 16 19121 19105 134.241 120 1.03885 0.47299
570 16 19159 19143 134.272 152 0.0600585 0.472908
571 16 19190 19174 134.254 124 0.736523 0.472955
572 16 19224 19208 134.257 136 0.73722 0.472972
573 16 19257 19241 134.253 132 0.0487991 0.472953
574 16 19289 19273 134.242 128 0.24744 0.472992
575 16 19324 19308 134.252 140 0.391084 0.472896
576 15 19354 19339 134.234 124 0.663745 0.473089
577 16 19391 19375 134.251 144 0.748631 0.472986
578 16 19424 19408 134.247 132 1.20114 0.472932
579 16 19461 19445 134.27 148 0.828344 0.472912
2020-03-08 04:16:16.565398 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.472773
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 19498 19482 134.294 148 0.172264 0.472773
581 16 19527 19511 134.263 116 0.709465 0.472963
582 15 19562 19547 134.279 144 0.725635 0.472916
583 16 19599 19583 134.296 144 0.057383 0.472805
584 16 19630 19614 134.278 124 0.734971 0.472896
585 16 19664 19648 134.281 136 0.775093 0.472893
586 16 19694 19678 134.257 120 0.56365 0.473032
587 16 19728 19712 134.26 136 0.853611 0.472951
588 16 19762 19746 134.263 136 0.798978 0.472975
589 15 19798 19783 134.286 148 0.0450345 0.472877
590 16 19833 19817 134.288 136 0.078982 0.472808
591 16 19865 19849 134.278 128 0.219248 0.472797
592 16 19897 19881 134.267 128 0.855589 0.472938
593 16 19930 19914 134.263 132 0.824729 0.472954
594 16 19967 19951 134.286 148 0.781349 0.472909
595 16 20000 19984 134.282 132 0.160079 0.472887
596 16 20034 20018 134.285 136 0.634084 0.472905
597 16 20070 20054 134.301 144 0.897115 0.472823
598 16 20101 20085 134.284 124 0.791925 0.47289
599 16 20133 20117 134.273 128 0.0607163 0.472916
2020-03-08 04:16:36.574697 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.472884
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 16 20161 20145 134.236 112 0.778877 0.472884
Total time run: 600.576
Total reads made: 20161
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 134.278
Average IOPS: 33
Stddev IOPS: 2.62918
Max IOPS: 42
Min IOPS: 27
Average Latency(s): 0.473116
Max latency(s): 1.35937
Min latency(s): 0.00821512
Total time run: 600.576
Total reads made: 20161
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 134.278
Average IOPS: 33
Stddev IOPS: 2.62918
Max IOPS: 42
Min IOPS: 27
Average Latency(s): 0.473116
Max latency(s): 1.35937
Min latency(s): 0.00821512
Cleanup, pool with 8 PGs
podman exec ceph-mon-f5-422-01 rados -p testbench cleanup
Performance Baseline, pool with 256 PGs
Since the cluster is brand new and largely unused, I purged everything. Not because I had to, but because I could ™ :-)
I went on to create a single, replica 3, pool testbench with 256 PGs.
I also set the application type to rgw.
Both of these were mainly done to shut up the (expected) warnings.
Now I have 96 PGs per OSD.
Performance Measurements with rados bench, pool with 256 PGs
All rados bench tests were run on f5-422-01.
Sequential Write Test, pool with 256 PGs
podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 write --no-cleanup | tee --append $(date +%F)-rados-test-write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_f5-422-01.internal.pcfe.net_3074
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 46 30 119.975 120 0.359667 0.380986
2 16 78 62 123.977 128 0.347748 0.432131
3 16 112 96 127.978 136 0.749442 0.438789
4 16 140 124 123.979 112 0.24305 0.48014
5 16 178 162 129.575 152 0.356074 0.463551
6 16 201 185 123.308 92 0.136012 0.477297
7 16 232 216 123.401 124 0.68408 0.49937
8 16 268 252 125.973 144 0.197888 0.494673
9 16 304 288 127.974 144 0.469769 0.487006
10 16 342 326 130.374 152 0.755879 0.479612
11 16 381 365 132.701 156 0.410596 0.470903
12 16 418 402 133.969 148 0.518643 0.469246
13 16 451 435 133.816 132 0.155759 0.469258
14 16 479 463 132.257 112 0.266196 0.470504
15 16 519 503 134.105 160 0.341318 0.468726
16 16 546 530 132.472 108 0.212679 0.473273
17 16 574 558 131.267 112 0.568618 0.482085
18 16 601 585 129.973 108 0.234361 0.483411
19 16 636 620 130.499 140 0.415921 0.483334
2020-03-09 13:47:33.737970 min lat: 0.0717105 max lat: 1.31601 avg lat: 0.484174
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 669 653 130.573 132 0.747809 0.484174
21 16 699 683 130.068 120 0.429075 0.484329
22 16 735 719 130.7 144 0.330224 0.483207
23 16 764 748 130.061 116 0.45127 0.486249
24 16 799 783 130.473 140 0.608222 0.484097
25 16 834 818 130.852 140 0.276788 0.484868
26 16 871 855 131.507 148 0.643607 0.482013
27 16 903 887 131.373 128 0.569667 0.48285
28 16 932 916 130.82 116 0.255666 0.483143
29 16 966 950 130.973 136 0.588522 0.48515
30 16 998 982 130.872 128 0.65502 0.483678
31 16 1024 1008 130.004 104 0.762551 0.488108
32 16 1044 1028 128.439 80 0.375524 0.492252
33 16 1066 1050 127.214 88 0.356193 0.49718
34 16 1096 1080 127.001 120 0.337509 0.498001
35 16 1122 1106 126.343 104 0.186368 0.501584
36 16 1147 1131 125.611 100 0.51983 0.504995
37 16 1181 1165 125.891 136 0.565213 0.504746
38 16 1222 1206 126.893 164 0.120005 0.501804
39 16 1264 1248 127.946 168 0.218359 0.497675
2020-03-09 13:47:53.750444 min lat: 0.0717105 max lat: 1.46248 avg lat: 0.49587
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1295 1279 127.847 124 0.460262 0.49587
41 16 1333 1317 128.435 152 0.643408 0.494639
42 16 1367 1351 128.612 136 0.398122 0.493743
43 16 1405 1389 129.155 152 0.430469 0.492381
44 16 1437 1421 129.129 128 0.206729 0.492294
45 16 1466 1450 128.836 116 0.0939879 0.493724
46 16 1496 1480 128.642 120 0.532239 0.493498
47 16 1535 1519 129.224 156 0.30192 0.49243
48 16 1572 1556 129.614 148 0.303496 0.49201
49 16 1602 1586 129.416 120 0.790806 0.491831
50 16 1632 1616 129.228 120 0.719665 0.491551
51 16 1661 1645 128.968 116 0.49687 0.494285
52 16 1691 1675 128.794 120 0.643789 0.493206
53 16 1719 1703 128.477 112 0.174905 0.49502
54 16 1753 1737 128.615 136 0.343396 0.494993
55 16 1781 1765 128.313 112 0.218915 0.495612
56 16 1813 1797 128.307 128 0.651625 0.49658
57 16 1840 1824 127.949 108 0.0950628 0.496028
58 16 1878 1862 128.363 152 0.364567 0.496509
59 16 1914 1898 128.627 144 0.322983 0.494879
2020-03-09 13:48:13.757359 min lat: 0.0716476 max lat: 1.46248 avg lat: 0.495193
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 1945 1929 128.55 124 0.277205 0.495193
61 16 1969 1953 128.015 96 0.15123 0.495979
62 16 1994 1978 127.563 100 0.863083 0.497656
63 16 2017 2001 126.998 92 0.740748 0.500637
64 16 2044 2028 126.701 108 0.132614 0.50229
65 16 2080 2064 126.966 144 0.199869 0.501354
66 16 2122 2106 127.587 168 0.505488 0.500336
67 16 2156 2140 127.712 136 0.513155 0.499443
68 16 2194 2178 128.069 152 0.251558 0.497586
69 16 2225 2209 128.01 124 1.05985 0.49788
70 16 2253 2237 127.781 112 0.528519 0.498064
71 16 2288 2272 127.953 140 0.273438 0.498547
72 16 2321 2305 128.009 132 0.919142 0.497856
73 16 2358 2342 128.282 148 0.260887 0.496907
74 16 2391 2375 128.331 132 0.440161 0.497008
75 16 2426 2410 128.487 140 0.195383 0.49716
76 16 2458 2442 128.48 128 0.269111 0.495936
77 16 2491 2475 128.525 132 0.180516 0.495634
78 16 2527 2511 128.723 144 0.272189 0.495631
79 16 2561 2545 128.814 136 0.725491 0.495048
2020-03-09 13:48:33.762806 min lat: 0.0716476 max lat: 1.46248 avg lat: 0.495577
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 2586 2570 128.453 100 0.497292 0.495577
81 16 2614 2598 128.25 112 0.242421 0.49702
82 16 2635 2619 127.708 84 0.66775 0.498904
83 16 2664 2648 127.566 116 0.0780687 0.49973
84 16 2690 2674 127.285 104 0.254723 0.500546
85 16 2722 2706 127.294 128 0.393947 0.500498
86 16 2747 2731 126.976 100 0.229089 0.501614
87 16 2780 2764 127.034 132 0.087223 0.501226
88 16 2816 2800 127.226 144 0.645412 0.501348
89 16 2846 2830 127.145 120 1.23799 0.501021
90 16 2880 2864 127.243 136 0.621924 0.501547
91 16 2913 2897 127.295 132 0.125684 0.501073
92 16 2939 2923 127.04 104 0.859025 0.502364
93 16 2974 2958 127.179 140 0.571539 0.50204
94 16 3007 2991 127.23 132 0.188611 0.501303
95 16 3037 3021 127.153 120 0.192009 0.500719
96 16 3077 3061 127.493 160 0.484729 0.50058
97 16 3110 3094 127.531 132 0.734255 0.500618
98 16 3148 3132 127.78 152 0.433723 0.499386
99 16 3179 3163 127.742 124 0.431907 0.4993
2020-03-09 13:48:53.777399 min lat: 0.0716476 max lat: 1.78416 avg lat: 0.500599
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3206 3190 127.544 108 0.665649 0.500599
101 16 3237 3221 127.509 124 0.369357 0.500074
102 16 3274 3258 127.709 148 0.198156 0.499872
103 16 3315 3299 128.061 164 0.438459 0.498676
104 16 3357 3341 128.445 168 0.223767 0.496934
105 16 3390 3374 128.478 132 0.451224 0.496417
106 16 3431 3415 128.813 164 0.456859 0.495683
107 16 3465 3449 128.88 136 0.371715 0.495341
108 16 3503 3487 129.094 152 0.301211 0.494671
109 16 3540 3524 129.267 148 0.236523 0.493584
110 16 3575 3559 129.365 140 0.578098 0.493294
111 16 3608 3592 129.388 132 0.131863 0.493406
112 16 3644 3628 129.518 144 0.312151 0.493075
113 16 3681 3665 129.681 148 0.593661 0.492712
114 16 3709 3693 129.526 112 0.772584 0.492846
115 16 3750 3734 129.826 164 0.334616 0.491762
116 16 3785 3769 129.913 140 0.215087 0.491054
117 16 3816 3800 129.862 124 0.302777 0.49167
118 16 3846 3830 129.778 120 0.357825 0.492301
119 16 3880 3864 129.83 136 0.539661 0.491452
2020-03-09 13:49:13.783504 min lat: 0.0714435 max lat: 1.78416 avg lat: 0.492061
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 3911 3895 129.78 124 0.369217 0.492061
121 16 3950 3934 129.996 156 0.337596 0.491487
122 16 3985 3969 130.077 140 0.325811 0.491144
123 16 4015 3999 129.995 120 0.234159 0.490601
124 16 4050 4034 130.075 140 0.351187 0.49128
125 16 4075 4059 129.834 100 0.612619 0.491183
126 16 4109 4093 129.883 136 0.742749 0.491513
127 16 4142 4126 129.899 132 0.459294 0.491227
128 16 4179 4163 130.04 148 0.571095 0.491236
129 16 4219 4203 130.272 160 0.457634 0.490343
130 16 4250 4234 130.224 124 0.745385 0.490243
131 16 4275 4259 129.993 100 0.114989 0.490629
132 16 4313 4297 130.159 152 0.467331 0.490717
133 16 4344 4328 130.112 124 0.367843 0.490623
134 16 4380 4364 130.216 144 0.554696 0.490194
135 16 4422 4406 130.495 168 0.187854 0.48958
136 16 4452 4436 130.417 120 0.768817 0.489632
137 16 4491 4475 130.603 156 0.413473 0.489357
138 16 4535 4519 130.931 176 0.273079 0.488159
139 16 4576 4560 131.168 164 0.532096 0.487224
2020-03-09 13:49:33.792025 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.487542
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 4601 4585 130.946 100 0.343155 0.487542
141 16 4638 4622 131.065 148 0.49814 0.487485
142 16 4671 4655 131.071 132 0.629704 0.487162
143 16 4700 4684 130.965 116 0.212529 0.487468
144 16 4736 4720 131.056 144 0.460629 0.487877
145 16 4767 4751 131.007 124 0.436478 0.487351
146 16 4808 4792 131.231 164 0.482506 0.487213
147 16 4845 4829 131.345 148 0.554865 0.486456
148 16 4880 4864 131.403 140 0.83767 0.485906
149 16 4913 4897 131.407 132 0.23633 0.48595
150 16 4945 4929 131.384 128 0.513121 0.486439
151 16 4974 4958 131.281 116 0.565548 0.486642
152 16 5006 4990 131.259 128 0.171883 0.486551
153 16 5041 5025 131.316 140 0.435409 0.486609
154 16 5076 5060 131.372 140 0.158076 0.485858
155 16 5111 5095 131.427 140 0.304771 0.486107
156 16 5147 5131 131.508 144 0.374117 0.485598
157 16 5173 5157 131.332 104 0.0826305 0.48585
158 16 5206 5190 131.336 132 0.310246 0.486187
159 16 5245 5229 131.491 156 0.166397 0.485676
2020-03-09 13:49:53.801959 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.485703
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 5278 5262 131.494 132 0.743372 0.485703
161 16 5311 5295 131.497 132 0.518896 0.485829
162 16 5348 5332 131.599 148 0.559159 0.485547
163 16 5380 5364 131.576 128 0.272844 0.48551
164 16 5412 5396 131.554 128 0.984581 0.485553
165 16 5447 5431 131.605 140 0.686722 0.485379
166 16 5477 5461 131.535 120 0.300734 0.485402
167 16 5521 5505 131.801 176 0.4637 0.484717
168 16 5558 5542 131.896 148 0.381065 0.484266
169 16 5602 5586 132.156 176 0.746587 0.483521
170 16 5640 5624 132.272 152 0.370748 0.483163
171 16 5676 5660 132.34 144 0.2788 0.482865
172 16 5707 5691 132.292 124 0.408688 0.482885
173 16 5746 5730 132.428 156 0.584259 0.482697
174 16 5782 5766 132.495 144 0.0789463 0.482216
175 16 5816 5800 132.515 136 0.2359 0.482381
176 16 5849 5833 132.512 132 1.08834 0.482006
177 16 5882 5866 132.508 132 0.472668 0.482455
178 16 5921 5905 132.639 156 0.321792 0.481885
179 16 5960 5944 132.769 156 0.565195 0.481509
2020-03-09 13:50:13.811540 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.481274
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 5994 5978 132.787 136 0.470208 0.481274
181 16 6023 6007 132.694 116 0.390591 0.481245
182 16 6060 6044 132.778 148 0.1815 0.481144
183 16 6096 6080 132.839 144 0.335024 0.48077
184 16 6124 6108 132.726 112 0.534821 0.481351
185 16 6167 6151 132.938 172 0.335853 0.48095
186 16 6201 6185 132.954 136 0.386932 0.480916
187 16 6239 6223 133.056 152 0.972335 0.480284
188 16 6269 6253 132.987 120 0.376611 0.480494
189 16 6313 6297 133.214 176 0.0865965 0.479733
190 16 6339 6323 133.06 104 0.147103 0.479894
191 16 6377 6361 133.159 152 0.24453 0.480105
192 16 6404 6388 133.028 108 0.218203 0.48002
193 16 6444 6428 133.168 160 0.106025 0.479846
194 16 6477 6461 133.161 132 1.00194 0.479703
195 16 6514 6498 133.237 148 0.805688 0.479583
196 16 6550 6534 133.292 144 0.520331 0.479262
197 16 6588 6572 133.387 152 0.408412 0.478834
198 16 6624 6608 133.44 144 0.271075 0.478981
199 16 6664 6648 133.572 160 0.196374 0.478292
2020-03-09 13:50:33.817509 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.478414
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 6697 6681 133.564 132 0.65436 0.478414
201 16 6739 6723 133.735 168 0.274473 0.478079
202 16 6772 6756 133.727 132 0.753022 0.477889
203 16 6806 6790 133.738 136 0.538447 0.477985
204 16 6840 6824 133.749 136 0.373896 0.477825
205 16 6879 6863 133.857 156 0.320515 0.47767
206 16 6917 6901 133.945 152 0.33905 0.477112
207 16 6948 6932 133.897 124 0.745551 0.477341
208 16 6986 6970 133.984 152 0.217822 0.477273
209 16 7024 7008 134.07 152 0.580256 0.476888
210 16 7061 7045 134.136 148 0.438728 0.476647
211 16 7097 7081 134.182 144 0.610789 0.476413
212 16 7123 7107 134.039 104 0.309203 0.476553
213 16 7163 7147 134.161 160 0.168109 0.476489
214 16 7195 7179 134.132 128 0.389336 0.476631
215 16 7225 7209 134.066 120 0.608599 0.476835
216 16 7257 7241 134.036 128 0.376109 0.477007
217 16 7284 7268 133.916 108 0.674074 0.477033
218 16 7320 7304 133.962 144 0.518294 0.477249
219 16 7355 7339 133.989 140 0.255822 0.477096
2020-03-09 13:50:53.827076 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.476958
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 7390 7374 134.016 140 0.359536 0.476958
221 16 7422 7406 133.988 128 0.235458 0.47688
222 16 7461 7445 134.087 156 0.552057 0.476534
223 16 7500 7484 134.185 156 0.122898 0.476439
224 16 7540 7524 134.3 160 0.677597 0.476169
225 16 7577 7561 134.361 148 0.478742 0.47592
226 16 7605 7589 134.262 112 0.452826 0.475701
227 16 7641 7625 134.305 144 0.361525 0.475857
228 16 7678 7662 134.365 148 0.232513 0.475916
229 16 7713 7697 134.389 140 0.377874 0.475651
230 16 7751 7735 134.466 152 0.283504 0.475249
231 16 7783 7767 134.438 128 0.73329 0.475222
232 16 7820 7804 134.496 148 0.585296 0.475329
233 16 7857 7841 134.554 148 0.298701 0.475105
234 16 7894 7878 134.611 148 0.303633 0.474928
235 16 7928 7912 134.617 136 0.207075 0.474899
236 16 7965 7949 134.673 148 0.586429 0.474658
237 16 7997 7981 134.645 128 0.233565 0.474662
238 16 8038 8022 134.767 164 0.301697 0.474489
239 16 8079 8063 134.889 164 0.467994 0.474005
2020-03-09 13:51:13.835219 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.474078
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 8113 8097 134.893 136 0.301468 0.474078
241 16 8153 8137 134.997 160 0.114605 0.473718
242 16 8193 8177 135.099 160 0.289684 0.47336
243 16 8225 8209 135.07 128 0.704905 0.473141
244 16 8265 8249 135.172 160 0.289248 0.473148
245 16 8303 8287 135.241 152 0.56666 0.472784
246 16 8331 8315 135.146 112 0.479352 0.472771
247 16 8362 8346 135.101 124 0.299596 0.472932
248 16 8388 8372 134.975 104 0.224243 0.473381
249 16 8424 8408 135.012 144 0.576815 0.473601
250 16 8452 8436 134.919 112 0.219891 0.473646
251 16 8481 8465 134.844 116 0.231252 0.473778
252 16 8522 8506 134.959 164 0.250522 0.473693
253 16 8563 8547 135.074 164 0.174336 0.473367
254 16 8591 8575 134.983 112 0.81775 0.473567
255 16 8630 8614 135.065 156 0.411743 0.47345
256 16 8656 8640 134.944 104 0.72775 0.473448
257 16 8689 8673 134.932 132 0.594755 0.473645
258 16 8728 8712 135.014 156 0.465425 0.473708
259 16 8761 8745 135.002 132 0.324839 0.473496
2020-03-09 13:51:33.841271 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.473384
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 8798 8782 135.052 148 0.372748 0.473384
261 16 8830 8814 135.025 128 0.821787 0.47336
262 16 8856 8840 134.906 104 0.394544 0.473502
263 16 8890 8874 134.91 136 0.582109 0.473994
264 16 8928 8912 134.975 152 0.40272 0.473827
265 16 8967 8951 135.054 156 0.154145 0.473271
266 16 9000 8984 135.043 132 0.888531 0.473354
267 16 9036 9020 135.076 144 0.140036 0.473409
268 16 9069 9053 135.064 132 0.152656 0.473387
269 16 9100 9084 135.023 124 0.741014 0.473505
270 16 9137 9121 135.071 148 0.252373 0.47329
271 16 9180 9164 135.207 172 0.349037 0.472971
272 16 9203 9187 135.049 92 0.307449 0.47316
273 16 9241 9225 135.11 152 0.127631 0.473415
274 16 9279 9263 135.172 152 0.376798 0.47296
275 16 9305 9289 135.058 104 0.982742 0.473267
276 16 9340 9324 135.076 140 0.226019 0.473228
277 16 9378 9362 135.137 152 0.518103 0.473143
278 16 9395 9379 134.895 68 0.947851 0.47367
279 16 9432 9416 134.942 148 0.173529 0.473801
2020-03-09 13:51:53.846751 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.47375
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 9464 9448 134.917 128 0.218692 0.47375
281 16 9493 9477 134.85 116 0.175167 0.473961
282 16 9536 9520 134.98 172 0.325448 0.473871
283 16 9570 9554 134.984 136 0.30484 0.473574
284 16 9600 9584 134.931 120 0.766058 0.473857
285 16 9624 9608 134.794 96 0.565996 0.474319
286 16 9654 9638 134.742 120 0.164893 0.474432
287 16 9690 9674 134.774 144 0.156769 0.474394
288 16 9716 9700 134.667 104 0.18715 0.474512
289 16 9750 9734 134.672 136 0.652454 0.474872
290 16 9779 9763 134.607 116 0.0811866 0.474867
291 16 9812 9796 134.598 132 0.304357 0.475065
292 16 9850 9834 134.658 152 0.780671 0.474751
293 16 9888 9872 134.717 152 0.81104 0.474651
294 16 9925 9909 134.762 148 1.01698 0.474468
295 16 9956 9940 134.725 124 0.911035 0.474546
296 16 9992 9976 134.756 144 0.572994 0.474646
297 16 10032 10016 134.84 160 0.179373 0.474415
298 16 10060 10044 134.764 112 0.371462 0.474367
299 16 10099 10083 134.835 156 0.6202 0.474196
2020-03-09 13:52:13.856173 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.474124
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 10135 10119 134.865 144 0.384481 0.474124
301 16 10170 10154 134.882 140 0.385599 0.474122
302 15 10201 10186 134.859 128 0.245145 0.473929
303 16 10246 10230 134.995 176 0.263429 0.473822
304 16 10277 10261 134.959 124 0.952323 0.47358
305 16 10315 10299 135.014 152 0.808054 0.47368
306 16 10348 10332 135.004 132 0.304334 0.473674
307 16 10384 10368 135.034 144 0.406567 0.473544
308 16 10418 10402 135.037 136 0.169952 0.473345
309 16 10446 10430 134.962 112 1.2108 0.47354
310 16 10488 10472 135.068 168 0.0938959 0.473156
311 16 10523 10507 135.084 140 0.364407 0.47338
312 16 10560 10544 135.126 148 0.503998 0.473255
313 16 10589 10573 135.064 116 0.544923 0.473554
314 16 10622 10606 135.055 132 0.59883 0.473577
315 16 10658 10642 135.083 144 0.932141 0.473393
316 16 10693 10677 135.098 140 0.379453 0.473252
317 16 10733 10717 135.177 160 0.100403 0.472957
318 16 10769 10753 135.204 144 0.487444 0.473017
319 16 10804 10788 135.219 140 0.165549 0.472857
2020-03-09 13:52:33.859926 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.472819
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 10839 10823 135.234 140 0.896992 0.472819
321 16 10871 10855 135.212 128 0.820972 0.472819
322 16 10908 10892 135.251 148 0.193181 0.472727
323 16 10948 10932 135.328 160 0.293782 0.472619
324 16 10976 10960 135.256 112 0.373327 0.472698
325 16 11016 11000 135.332 160 0.110501 0.472549
326 16 11049 11033 135.321 132 0.434871 0.472617
327 16 11088 11072 135.385 156 0.120279 0.472248
328 16 11123 11107 135.398 140 0.162674 0.472318
329 16 11153 11137 135.352 120 0.340746 0.472257
330 16 11186 11170 135.341 132 0.279803 0.47247
331 16 11224 11208 135.39 152 0.256636 0.47234
332 16 11246 11230 135.247 88 0.344452 0.472721
333 16 11278 11262 135.226 128 0.559596 0.47266
334 16 11309 11293 135.192 124 0.322432 0.47305
335 16 11352 11336 135.301 172 0.401244 0.472582
336 16 11385 11369 135.291 132 0.260196 0.472609
337 16 11416 11400 135.258 124 0.937847 0.472758
338 16 11450 11434 135.26 136 0.156418 0.472695
339 16 11481 11465 135.227 124 0.416435 0.472927
2020-03-09 13:52:53.868860 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.472805
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 11519 11503 135.276 152 0.735485 0.472805
341 16 11554 11538 135.289 140 0.713162 0.472787
342 16 11586 11570 135.268 128 0.395527 0.47277
343 16 11613 11597 135.188 108 0.16304 0.473001
344 16 11648 11632 135.202 140 0.498541 0.473054
345 16 11685 11669 135.239 148 0.431977 0.472929
346 16 11719 11703 135.241 136 0.365017 0.472964
347 16 11749 11733 135.197 120 0.208045 0.472912
348 16 11793 11777 135.314 176 0.288502 0.472714
349 16 11822 11806 135.259 116 0.615701 0.472621
350 16 11862 11846 135.329 160 0.433324 0.472623
351 16 11892 11876 135.285 120 0.460653 0.472535
352 16 11932 11916 135.355 160 0.187522 0.472529
353 16 11963 11947 135.323 124 0.252118 0.472454
354 16 12002 11986 135.381 156 0.579114 0.472448
355 16 12036 12020 135.383 136 0.411726 0.472376
356 16 12064 12048 135.317 112 0.261821 0.47244
357 16 12098 12082 135.319 136 0.696116 0.472521
358 16 12121 12105 135.198 92 1.37038 0.472704
359 16 12161 12145 135.267 160 0.428137 0.47286
2020-03-09 13:53:13.875746 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.472768
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 12198 12182 135.302 148 0.322379 0.472768
361 16 12232 12216 135.304 136 0.931174 0.472525
362 16 12263 12247 135.273 124 0.231982 0.472463
363 16 12303 12287 135.341 160 0.42676 0.472412
364 16 12341 12325 135.386 152 0.46565 0.47234
365 16 12368 12352 135.311 108 0.382646 0.472508
366 16 12402 12386 135.313 136 0.455886 0.472637
367 16 12433 12417 135.282 124 0.0852436 0.472558
368 16 12473 12457 135.349 160 0.560605 0.472514
369 16 12506 12490 135.34 132 0.260523 0.472529
370 16 12549 12533 135.439 172 0.226338 0.472307
371 16 12582 12566 135.43 132 0.393776 0.471963
372 16 12617 12601 135.442 140 0.350945 0.47224
373 16 12655 12639 135.486 152 0.169618 0.471982
374 16 12695 12679 135.552 160 0.398727 0.471771
375 16 12721 12705 135.467 104 0.416972 0.471844
376 16 12762 12746 135.543 164 0.255434 0.471926
377 16 12798 12782 135.565 144 0.481763 0.471841
378 16 12830 12814 135.545 128 0.377422 0.471762
379 16 12872 12856 135.63 168 0.270653 0.471591
2020-03-09 13:53:33.881853 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.471633
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 12895 12879 135.516 92 0.6754 0.471633
381 16 12932 12916 135.548 148 0.365135 0.471846
382 16 12968 12952 135.569 144 0.476505 0.471797
383 16 13008 12992 135.633 160 0.514332 0.471609
384 16 13044 13028 135.654 144 0.252104 0.471462
385 16 13084 13068 135.717 160 0.181763 0.471242
386 16 13113 13097 135.666 116 0.188631 0.47131
387 16 13150 13134 135.698 148 0.500839 0.471274
388 16 13185 13169 135.709 140 0.144931 0.471286
389 16 13221 13205 135.73 144 0.606974 0.471286
390 16 13260 13244 135.782 156 0.414916 0.470931
391 16 13298 13282 135.823 152 0.403291 0.470906
392 16 13322 13306 135.72 96 0.32187 0.47124
393 16 13358 13342 135.741 144 0.53494 0.471113
394 16 13390 13374 135.722 128 0.343039 0.47122
395 16 13424 13408 135.722 136 0.366867 0.471182
396 16 13456 13440 135.702 128 0.307662 0.471252
397 16 13484 13468 135.643 112 0.136399 0.471323
398 16 13527 13511 135.734 172 0.412355 0.471182
399 16 13560 13544 135.724 132 0.444682 0.471163
2020-03-09 13:53:53.895854 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.47118
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 13592 13576 135.705 128 0.209861 0.47118
401 16 13633 13617 135.776 164 0.285086 0.471092
402 16 13665 13649 135.756 128 0.118248 0.47104
403 16 13702 13686 135.786 148 0.717739 0.471018
404 16 13732 13716 135.747 120 0.510652 0.471057
405 16 13768 13752 135.768 144 0.154975 0.471053
406 16 13806 13790 135.807 152 0.456511 0.470924
407 16 13842 13826 135.828 144 0.230021 0.470898
408 16 13878 13862 135.847 144 0.390341 0.47073
409 16 13912 13896 135.848 136 0.223302 0.470697
410 15 13945 13930 135.848 136 0.332684 0.470824
411 16 13983 13967 135.877 148 0.492018 0.470817
412 16 14016 14000 135.868 132 0.561557 0.470724
413 16 14049 14033 135.859 132 0.269506 0.470821
414 16 14081 14065 135.839 128 0.353223 0.470723
415 16 14104 14088 135.734 92 0.83534 0.470963
416 16 14135 14119 135.705 124 0.540999 0.471235
417 16 14174 14158 135.753 156 0.760297 0.471222
418 16 14209 14193 135.764 140 0.265108 0.471054
419 16 14244 14228 135.774 140 0.322467 0.471034
2020-03-09 13:54:13.902195 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.470908
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 14282 14266 135.812 152 0.312589 0.470908
421 16 14324 14308 135.889 168 0.477891 0.470726
422 16 14357 14341 135.879 132 0.893055 0.470598
423 16 14393 14377 135.898 144 0.284082 0.470633
424 16 14433 14417 135.955 160 0.158381 0.470417
425 16 14466 14450 135.946 132 0.41506 0.470367
426 16 14498 14482 135.927 128 0.263931 0.470394
427 16 14533 14517 135.936 140 0.201927 0.470579
428 16 14573 14557 135.992 160 0.322358 0.470369
429 16 14615 14599 136.067 168 0.394001 0.470152
430 16 14648 14632 136.057 132 0.91336 0.470056
431 16 14690 14674 136.131 168 0.461153 0.469947
432 16 14717 14701 136.066 108 0.277285 0.469965
433 16 14747 14731 136.029 120 0.659027 0.470118
434 16 14783 14767 136.047 144 0.109223 0.470067
435 16 14821 14805 136.084 152 0.280136 0.470037
436 16 14856 14840 136.093 140 0.436811 0.469963
437 16 14894 14878 136.129 152 0.223475 0.469821
438 16 14931 14915 136.156 148 0.511638 0.469668
439 16 14975 14959 136.246 176 0.157312 0.469545
2020-03-09 13:54:33.911097 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.469359
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 15008 14992 136.236 132 0.238473 0.469359
441 16 15044 15028 136.254 144 0.431187 0.469515
442 16 15083 15067 136.298 156 0.19367 0.46924
443 16 15114 15098 136.27 124 0.533306 0.46928
444 16 15157 15141 136.351 172 0.520169 0.469149
445 16 15192 15176 136.359 140 0.285858 0.469029
446 16 15225 15209 136.349 132 0.413597 0.469004
447 16 15255 15239 136.312 120 0.297951 0.469232
448 16 15288 15272 136.303 132 0.378513 0.469229
449 16 15324 15308 136.32 144 0.127919 0.469246
450 16 15356 15340 136.301 128 0.765494 0.469174
451 16 15399 15383 136.38 172 0.118877 0.468965
452 16 15423 15407 136.291 96 1.17101 0.469233
453 16 15463 15447 136.342 160 0.270425 0.469159
454 16 15500 15484 136.368 148 0.722035 0.468955
455 16 15539 15523 136.411 156 0.376699 0.468883
456 16 15577 15561 136.445 152 0.266948 0.468724
457 16 15604 15588 136.383 108 0.362744 0.468817
458 16 15635 15619 136.356 124 0.400404 0.469087
459 16 15675 15659 136.407 160 0.219797 0.468832
2020-03-09 13:54:53.919461 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.468942
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 15705 15689 136.371 120 0.517915 0.468942
461 16 15751 15735 136.474 184 0.314449 0.468747
462 16 15785 15769 136.473 136 0.254378 0.468726
463 16 15821 15805 136.489 144 0.554271 0.468577
464 16 15862 15846 136.548 164 0.486457 0.468442
465 16 15887 15871 136.469 100 0.871659 0.468612
466 16 15916 15900 136.424 116 0.593023 0.468817
467 16 15953 15937 136.449 148 0.201829 0.468741
468 16 15993 15977 136.498 160 0.324658 0.468577
469 16 16033 16017 136.548 160 0.112061 0.468452
470 16 16067 16051 136.547 136 0.311924 0.468319
471 16 16094 16078 136.486 108 1.21297 0.46868
472 16 16131 16115 136.511 148 0.105206 0.468558
473 16 16170 16154 136.552 156 0.245129 0.468456
474 16 16202 16186 136.534 128 0.40674 0.468355
475 16 16233 16217 136.507 124 0.18983 0.46859
476 16 16265 16249 136.489 128 0.32931 0.46855
477 16 16299 16283 136.488 136 0.319409 0.468624
478 16 16345 16329 136.588 184 0.352787 0.468407
479 16 16372 16356 136.528 108 1.16864 0.468421
2020-03-09 13:55:13.932902 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.468281
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 16414 16398 136.593 168 0.713134 0.468281
481 16 16450 16434 136.609 144 0.380134 0.468194
482 16 16486 16470 136.624 144 0.506203 0.4682
483 16 16521 16505 136.631 140 0.202173 0.468043
484 16 16560 16544 136.671 156 0.094533 0.468021
485 16 16591 16575 136.645 124 0.771024 0.467972
486 16 16630 16614 136.684 156 0.238991 0.467953
487 16 16669 16653 136.722 156 0.314566 0.467874
488 16 16703 16687 136.72 136 0.5175 0.467886
489 16 16738 16722 136.727 140 0.358545 0.467782
490 16 16778 16762 136.775 160 0.301999 0.467673
491 16 16812 16796 136.773 136 0.438576 0.467625
492 16 16855 16839 136.844 172 0.315865 0.46749
493 16 16895 16879 136.891 160 0.155628 0.467266
494 16 16932 16916 136.914 148 0.349489 0.467335
495 16 16962 16946 136.88 120 0.530849 0.467282
496 16 16996 16980 136.878 136 0.951513 0.467268
497 16 17034 17018 136.908 152 0.438845 0.467129
498 16 17078 17062 136.987 176 0.113521 0.467034
499 16 17117 17101 137.025 156 0.929013 0.466866
2020-03-09 13:55:33.943960 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.466914
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 17145 17129 136.974 112 0.551434 0.466914
501 16 17176 17160 136.948 124 0.220509 0.466946
502 16 17205 17189 136.907 116 0.708215 0.467283
503 16 17239 17223 136.905 136 0.223902 0.467277
504 16 17275 17259 136.919 144 0.149271 0.467175
505 16 17308 17292 136.909 132 0.527162 0.467279
506 16 17343 17327 136.915 140 0.457346 0.467162
507 16 17377 17361 136.913 136 0.781563 0.467248
508 16 17416 17400 136.95 156 0.295068 0.467058
509 16 17443 17427 136.893 108 0.207526 0.467181
510 16 17478 17462 136.899 140 0.39138 0.467222
511 16 17518 17502 136.945 160 0.352586 0.467175
512 16 17547 17531 136.904 116 0.260512 0.4673
513 16 17589 17573 136.964 168 0.390446 0.467094
514 16 17622 17606 136.954 132 0.160975 0.467091
515 16 17657 17641 136.96 140 0.243762 0.467045
516 16 17692 17676 136.966 140 0.445149 0.467003
517 16 17723 17707 136.941 124 0.373801 0.46705
518 16 17760 17744 136.962 148 0.163027 0.466981
519 16 17798 17782 136.991 152 0.24347 0.466915
2020-03-09 13:55:53.951850 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.466837
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 17837 17821 137.027 156 0.433931 0.466837
521 16 17864 17848 136.97 108 0.160606 0.466883
522 16 17906 17890 137.03 168 0.361994 0.466879
523 16 17942 17926 137.043 144 0.541809 0.466788
524 16 17974 17958 137.026 128 0.687222 0.466877
525 16 18006 17990 137.008 128 0.187099 0.466867
526 16 18033 18017 136.953 108 0.605651 0.466957
527 16 18061 18045 136.905 112 0.162039 0.467175
528 16 18093 18077 136.889 128 0.529875 0.467288
529 16 18128 18112 136.894 140 0.280982 0.467226
530 16 18166 18150 136.923 152 0.485335 0.46728
531 16 18201 18185 136.929 140 0.486643 0.467076
532 16 18243 18227 136.987 168 0.110459 0.466997
533 16 18267 18251 136.91 96 0.697684 0.46696
534 16 18309 18293 136.967 168 0.569496 0.467099
535 16 18345 18329 136.98 144 0.274834 0.466923
536 16 18378 18362 136.971 132 0.618873 0.46704
537 16 18411 18395 136.961 132 0.178024 0.466999
538 16 18452 18436 137.012 164 0.334936 0.466864
539 16 18483 18467 136.987 124 0.362084 0.466863
2020-03-09 13:56:13.965891 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.466925
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 18516 18500 136.978 132 0.141543 0.466925
541 16 18551 18535 136.984 140 0.194767 0.466926
542 16 18583 18567 136.967 128 0.30533 0.466911
543 16 18617 18601 136.965 136 0.242655 0.46699
544 16 18651 18635 136.963 136 0.543032 0.466962
545 16 18681 18665 136.932 120 0.369876 0.467076
546 16 18714 18698 136.923 132 0.757877 0.467201
547 16 18742 18726 136.878 112 0.623998 0.467199
548 16 18778 18762 136.89 144 0.598083 0.467294
549 16 18822 18806 136.962 176 0.317329 0.467033
550 16 18861 18845 136.996 156 0.30958 0.466969
551 16 18900 18884 137.03 156 0.107902 0.46677
552 16 18929 18913 136.992 116 0.248413 0.466878
553 16 18972 18956 137.055 172 0.213509 0.466795
554 16 19008 18992 137.068 144 0.809786 0.46673
555 16 19042 19026 137.066 136 0.562657 0.466716
556 16 19074 19058 137.049 128 0.665218 0.46665
557 16 19114 19098 137.091 160 0.205075 0.466655
558 16 19151 19135 137.11 148 0.288084 0.466515
559 16 19181 19165 137.079 120 0.221551 0.466553
2020-03-09 13:56:33.971756 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.46663
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 19220 19204 137.113 156 0.544361 0.46663
561 16 19263 19247 137.175 172 0.15661 0.466293
562 16 19297 19281 137.173 136 0.586363 0.466316
563 16 19332 19316 137.178 140 0.265647 0.466271
564 16 19370 19354 137.204 152 0.106856 0.466203
565 16 19404 19388 137.202 136 0.68798 0.466209
566 16 19448 19432 137.27 176 0.591428 0.46602
567 16 19478 19462 137.239 120 0.60812 0.466126
568 16 19517 19501 137.272 156 0.390487 0.466033
569 16 19556 19540 137.305 156 0.447418 0.465876
570 16 19594 19578 137.331 152 0.761197 0.465827
571 16 19635 19619 137.378 164 0.388597 0.465695
572 16 19655 19639 137.277 80 0.611985 0.465841
573 16 19697 19681 137.331 168 0.599114 0.465891
574 16 19721 19705 137.259 96 0.489929 0.465954
575 16 19763 19747 137.312 168 0.306763 0.465967
576 16 19799 19783 137.324 144 0.634426 0.465853
577 16 19840 19824 137.369 164 0.237302 0.465778
578 16 19876 19860 137.381 144 0.979485 0.465645
579 16 19910 19894 137.378 136 0.593116 0.46572
2020-03-09 13:56:53.981010 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.465694
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 19945 19929 137.383 140 0.264241 0.465694
581 16 19982 19966 137.401 148 0.674185 0.465603
582 16 20019 20003 137.419 148 0.529329 0.465548
583 16 20051 20035 137.403 128 0.268896 0.465571
584 16 20093 20077 137.455 168 0.538054 0.465439
585 16 20127 20111 137.452 136 0.78359 0.465388
586 16 20169 20153 137.504 168 0.195777 0.465281
587 16 20200 20184 137.481 124 0.266851 0.465221
588 16 20238 20222 137.506 152 0.398561 0.465175
589 16 20271 20255 137.497 132 0.495558 0.465266
590 16 20309 20293 137.521 152 0.58999 0.465193
591 16 20340 20324 137.498 124 0.437875 0.465217
592 16 20385 20369 137.57 180 0.341231 0.465085
593 16 20413 20397 137.527 112 1.07334 0.465053
594 16 20453 20437 137.565 160 0.498946 0.46508
595 16 20487 20471 137.562 136 0.489279 0.465053
596 16 20523 20507 137.572 144 0.249381 0.464951
597 16 20550 20534 137.523 108 0.679844 0.465051
598 16 20586 20570 137.534 144 0.621874 0.465158
599 16 20621 20605 137.537 140 0.366411 0.4651
2020-03-09 13:57:13.991680 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.465053
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 15 20649 20634 137.501 116 0.477456 0.465053
Total time run: 600.444
Total writes made: 20649
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 137.558
Stddev Bandwidth: 19.6916
Max bandwidth (MB/sec): 184
Min bandwidth (MB/sec): 68
Average IOPS: 34
Stddev IOPS: 4.9229
Max IOPS: 46
Min IOPS: 17
Average Latency(s): 0.465239
Stddev Latency(s): 0.23455
Max latency(s): 1.78416
Min latency(s): 0.0712215
Total time run: 600.444
Total writes made: 20649
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 137.558
Stddev Bandwidth: 19.6916
Max bandwidth (MB/sec): 184
Min bandwidth (MB/sec): 68
Average IOPS: 34
Stddev IOPS: 4.9229
Max IOPS: 46
Min IOPS: 17
Average Latency(s): 0.465239
Stddev Latency(s): 0.23455
Max latency(s): 1.78416
Min latency(s): 0.0712215
Sequential Read Test, pool with 256 PGs
podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 seq | tee --append $(date +%F)-rados-test-read-seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 47 31 122.867 124 0.892219 0.299688
2 15 79 64 127.402 132 0.272987 0.408774
3 16 109 93 123.607 116 0.445915 0.442824
4 16 139 123 122.704 120 0.0606516 0.466495
5 16 169 153 122.16 120 0.890098 0.477531
6 16 201 185 123.12 128 0.197168 0.478944
7 15 231 216 123.223 124 0.27612 0.473009
8 16 266 250 124.815 136 0.16127 0.477635
9 16 297 281 124.722 124 1.67437 0.474589
10 16 330 314 125.446 132 0.307985 0.473194
11 16 362 346 125.657 128 0.416032 0.482695
12 16 393 377 125.511 124 0.292082 0.482124
13 16 426 410 126.003 132 0.0735909 0.481627
14 16 457 441 125.858 124 0.101919 0.483594
15 16 486 470 125.183 116 0.0970761 0.487313
16 16 517 501 125.091 124 0.148924 0.49301
17 16 547 531 124.791 120 1.38568 0.492822
18 16 578 562 124.746 124 0.266846 0.491729
19 16 613 597 125.547 140 1.44537 0.491227
2020-03-09 14:14:15.531643 min lat: 0.0209854 max lat: 2.10739 avg lat: 0.491662
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 643 627 125.269 120 1.62711 0.491662
21 16 672 656 124.828 116 0.0461491 0.495234
22 16 701 685 124.426 116 0.154277 0.496283
23 16 733 717 124.576 128 0.229963 0.494833
24 16 765 749 124.718 128 0.125264 0.497316
25 16 798 782 125.008 132 0.160512 0.495727
26 16 832 816 125.43 136 0.331566 0.492769
27 16 868 852 126.117 144 0.0495642 0.490482
28 16 901 885 126.326 132 0.0995234 0.495103
29 16 928 912 125.694 108 1.47158 0.49533
30 16 961 945 125.903 132 0.103094 0.495363
31 16 991 975 125.712 120 0.145827 0.496937
32 16 1022 1006 125.658 124 0.2493 0.49566
33 16 1052 1036 125.486 120 1.0992 0.49861
34 16 1082 1066 125.325 120 0.196652 0.497979
35 16 1120 1104 126.081 152 0.0441845 0.495837
36 15 1149 1134 125.904 120 1.17875 0.497344
37 16 1179 1163 125.634 116 0.307539 0.499983
38 16 1212 1196 125.756 132 1.46409 0.497412
39 16 1253 1237 126.735 164 0.0898571 0.494788
2020-03-09 14:14:35.553049 min lat: 0.0209854 max lat: 2.10739 avg lat: 0.495789
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1284 1268 126.666 124 0.280382 0.495789
41 16 1316 1300 126.698 128 1.44724 0.494111
42 16 1353 1337 127.204 148 1.58673 0.494859
43 16 1385 1369 127.223 128 0.735345 0.494403
44 16 1419 1403 127.422 136 0.30188 0.492974
45 16 1450 1434 127.345 124 0.0577048 0.494533
46 16 1487 1471 127.793 148 1.29992 0.492074
47 16 1518 1502 127.712 124 1.28863 0.491668
48 16 1550 1534 127.718 128 0.632717 0.492283
49 16 1582 1566 127.723 128 0.338998 0.491939
50 16 1615 1599 127.808 132 1.09127 0.492657
51 16 1646 1630 127.733 124 0.275021 0.49258
52 16 1681 1665 127.967 140 1.54424 0.490845
53 16 1717 1701 128.269 144 0.206363 0.490046
54 15 1747 1732 128.186 124 0.232308 0.490402
55 16 1780 1764 128.182 128 1.38793 0.490511
56 16 1813 1797 128.25 132 0.314296 0.490262
57 16 1844 1828 128.175 124 0.360981 0.490292
58 16 1871 1855 127.827 108 0.414088 0.490588
59 15 1903 1888 127.893 132 0.376294 0.492249
2020-03-09 14:14:55.560281 min lat: 0.0204813 max lat: 2.10739 avg lat: 0.4939
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 1930 1914 127.495 104 0.0389062 0.4939
61 16 1963 1947 127.568 132 0.39855 0.493329
62 16 1995 1979 127.574 128 0.172692 0.493467
63 16 2028 2012 127.644 132 0.0292764 0.492148
64 15 2062 2047 127.835 140 0.230653 0.491703
65 16 2092 2076 127.653 116 0.369797 0.492676
66 16 2127 2111 127.838 140 1.51087 0.49297
67 16 2158 2142 127.78 124 0.263258 0.492571
68 15 2187 2172 127.662 120 0.0548647 0.4928
69 16 2222 2206 127.781 136 0.0629204 0.492871
70 16 2255 2239 127.837 132 0.0686457 0.492765
71 16 2290 2274 128.008 140 0.306493 0.491341
72 16 2320 2304 127.897 120 1.90779 0.492541
73 16 2350 2334 127.788 120 0.271317 0.493013
74 16 2381 2365 127.736 124 0.222788 0.493108
75 16 2415 2399 127.845 136 0.0265713 0.492665
76 16 2445 2429 127.741 120 0.212303 0.493418
77 16 2482 2466 128.003 148 0.097387 0.492576
78 16 2518 2502 128.207 144 1.34222 0.492447
79 16 2545 2529 127.951 108 1.61514 0.493918
2020-03-09 14:15:15.572748 min lat: 0.0203744 max lat: 2.1132 avg lat: 0.493859
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 16 2577 2561 127.951 128 1.12898 0.493859
81 16 2612 2596 128.099 140 1.28265 0.492462
82 16 2643 2627 128.047 124 0.204044 0.492216
83 16 2681 2665 128.335 152 0.0690425 0.490737
84 16 2713 2697 128.328 128 0.889824 0.491834
85 16 2744 2728 128.277 124 0.508321 0.492136
86 16 2775 2759 128.227 124 1.16227 0.492862
87 16 2806 2790 128.178 124 1.233 0.492743
88 16 2841 2825 128.31 140 0.12701 0.491969
89 16 2875 2859 128.397 136 0.243668 0.491223
90 16 2906 2890 128.348 124 0.297884 0.491903
91 16 2935 2919 128.212 116 0.28308 0.492077
92 16 2969 2953 128.296 136 1.70304 0.491982
93 16 3003 2987 128.379 136 0.148716 0.491434
94 16 3038 3022 128.502 140 0.0211009 0.490931
95 16 3067 3051 128.371 116 0.269733 0.491387
96 16 3099 3083 128.366 128 0.201617 0.49092
97 16 3131 3115 128.363 128 0.342916 0.491726
98 16 3161 3145 128.277 120 1.42997 0.492707
99 16 3196 3180 128.395 140 0.0253981 0.491718
2020-03-09 14:15:35.580023 min lat: 0.0203744 max lat: 2.15539 avg lat: 0.492147
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3227 3211 128.351 124 0.0467987 0.492147
101 16 3263 3247 128.506 144 0.164445 0.49198
102 16 3292 3276 128.383 116 1.48138 0.491628
103 16 3332 3316 128.69 160 0.0201424 0.490751
104 16 3361 3345 128.568 116 0.314573 0.491364
105 16 3395 3379 128.638 136 0.0257904 0.490323
106 16 3429 3413 128.706 136 0.0459659 0.490438
107 16 3462 3446 128.736 132 1.79781 0.490499
108 16 3495 3479 128.765 132 0.0469213 0.490293
109 16 3530 3514 128.867 140 1.33188 0.490484
110 16 3563 3547 128.883 132 0.257433 0.490693
111 16 3592 3576 128.766 116 0.12864 0.490938
112 16 3626 3610 128.829 136 0.215556 0.490451
113 16 3660 3644 128.892 136 0.0201329 0.49001
114 16 3695 3679 128.99 140 1.55332 0.489797
115 16 3724 3708 128.877 116 0.229602 0.490283
116 16 3753 3737 128.766 116 0.166641 0.490963
117 16 3785 3769 128.759 128 0.199312 0.491129
118 16 3815 3799 128.685 120 0.18168 0.491448
119 16 3846 3830 128.645 124 1.5559 0.49102
2020-03-09 14:15:55.620068 min lat: 0.0200594 max lat: 2.15539 avg lat: 0.490487
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 3883 3867 128.783 148 1.90927 0.490487
121 16 3913 3897 128.71 120 0.205211 0.490891
122 16 3947 3931 128.769 136 0.302697 0.490809
123 16 3978 3962 128.728 124 1.1433 0.491541
124 16 4010 3994 128.722 128 0.180435 0.491076
125 16 4046 4030 128.843 144 1.20192 0.491044
126 16 4075 4059 128.74 116 1.31125 0.491106
127 16 4107 4091 128.734 128 0.050824 0.491015
128 16 4138 4122 128.696 124 0.0350849 0.491038
129 16 4172 4156 128.751 136 1.81892 0.490808
130 16 4202 4186 128.683 120 1.25662 0.491693
131 16 4233 4217 128.644 124 0.20783 0.49147
132 16 4264 4248 128.598 124 1.28707 0.492114
133 16 4302 4286 128.773 152 0.19773 0.491398
134 16 4335 4319 128.797 132 0.0550823 0.490833
135 16 4371 4355 128.91 144 1.50193 0.490568
136 16 4407 4391 129.02 144 0.0649788 0.49009
137 16 4440 4424 129.042 132 0.0972908 0.490044
138 16 4474 4458 129.092 136 1.45655 0.490096
139 16 4509 4493 129.17 140 0.0410438 0.489488
2020-03-09 14:16:15.644998 min lat: 0.0200542 max lat: 2.29396 avg lat: 0.489154
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 4542 4526 129.19 132 0.26454 0.489154
141 15 4573 4558 129.181 128 0.0215077 0.489651
142 16 4604 4588 129.117 120 0.186569 0.489814
143 16 4633 4617 129.025 116 1.56875 0.490359
144 16 4665 4649 129.018 128 0.0422559 0.490218
145 15 4697 4682 129.038 132 0.222384 0.490184
146 16 4724 4708 128.866 104 1.6473 0.490947
147 16 4759 4743 128.942 140 0.0264929 0.490668
148 16 4790 4774 128.908 124 0.240668 0.490269
149 16 4824 4808 128.955 136 0.0527974 0.490019
150 16 4856 4840 128.949 128 0.313878 0.490551
151 15 4886 4871 128.913 124 0.42731 0.490692
152 16 4920 4904 128.929 132 0.163081 0.491003
153 16 4954 4938 128.975 136 1.49151 0.490549
154 16 4986 4970 128.969 128 0.267954 0.490443
155 16 5020 5004 129.014 136 1.67769 0.490629
156 16 5058 5042 129.161 152 0.0454149 0.489958
157 16 5091 5075 129.179 132 1.41498 0.490013
158 16 5125 5109 129.222 136 0.0354356 0.489436
159 15 5164 5149 129.414 160 0.0780695 0.489058
2020-03-09 14:16:35.658777 min lat: 0.0198839 max lat: 2.29396 avg lat: 0.488735
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 5201 5185 129.505 144 0.321195 0.488735
161 16 5238 5222 129.62 148 0.03819 0.488391
162 16 5272 5256 129.659 136 0.33829 0.487809
163 16 5303 5287 129.624 124 0.254569 0.487856
164 16 5331 5315 129.517 112 0.254206 0.488152
165 16 5358 5342 129.386 108 0.202445 0.489093
166 16 5387 5371 129.306 116 1.3374 0.489476
167 16 5420 5404 129.322 132 0.308261 0.489442
168 16 5454 5438 129.361 136 0.101547 0.489295
169 16 5488 5472 129.4 136 0.0542278 0.489301
170 16 5519 5503 129.369 124 1.62126 0.489662
171 16 5557 5541 129.501 152 0.0634993 0.488651
172 16 5593 5577 129.583 144 1.84887 0.488236
173 16 5625 5609 129.573 128 0.489666 0.488547
174 16 5654 5638 129.494 116 0.30284 0.488526
175 16 5682 5666 129.394 112 1.76192 0.489122
176 15 5713 5698 129.384 128 0.541846 0.488987
177 16 5749 5733 129.444 140 1.95049 0.488991
178 16 5776 5760 129.324 108 0.223297 0.489616
179 16 5808 5792 129.316 128 0.0284227 0.489446
2020-03-09 14:16:55.668785 min lat: 0.0198839 max lat: 2.29396 avg lat: 0.489424
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 5841 5825 129.331 132 0.0395301 0.489424
181 16 5870 5854 129.256 116 1.85892 0.489997
182 15 5903 5888 129.285 136 0.0311012 0.490041
183 16 5939 5923 129.343 140 0.0429795 0.489662
184 16 5966 5950 129.227 108 0.289396 0.489738
185 16 5996 5980 129.177 120 0.257256 0.49011
186 16 6027 6011 129.149 124 1.47025 0.490253
187 15 6059 6044 129.164 132 0.222182 0.489788
188 16 6091 6075 129.136 124 0.0282552 0.489668
189 16 6119 6103 129.046 112 0.10614 0.490754
190 16 6155 6139 129.124 144 1.53179 0.490333
191 16 6185 6169 129.076 120 0.281941 0.490207
192 16 6212 6196 128.966 108 0.021823 0.490802
193 16 6246 6230 129.002 136 0.356081 0.490863
194 16 6279 6263 129.014 132 0.553909 0.491168
195 15 6313 6298 129.07 140 0.292374 0.490949
196 16 6347 6331 129.085 132 0.212525 0.490656
197 16 6381 6365 129.12 136 0.287295 0.490472
198 16 6408 6392 129.013 108 0.223018 0.490753
199 16 6436 6420 128.928 112 0.240123 0.491064
2020-03-09 14:17:15.692483 min lat: 0.0198839 max lat: 2.29396 avg lat: 0.491105
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 6469 6453 128.943 132 0.118565 0.491105
201 16 6501 6485 128.938 128 1.22528 0.491256
202 16 6531 6515 128.893 120 0.337655 0.491108
203 16 6566 6550 128.948 140 1.63896 0.491307
204 16 6599 6583 128.963 132 0.076407 0.491135
205 16 6634 6618 129.016 140 0.0440429 0.490954
206 16 6668 6652 129.05 136 0.229665 0.490673
207 16 6702 6686 129.083 136 0.146742 0.490531
208 16 6735 6719 129.097 132 0.244986 0.490723
209 16 6763 6747 129.015 112 0.314523 0.490741
210 16 6795 6779 128.995 128 0.0547352 0.490936
211 15 6829 6814 129.046 140 0.142437 0.490677
212 16 6860 6844 129.003 120 0.220339 0.490963
213 16 6892 6876 128.998 128 1.49846 0.490942
214 15 6923 6908 128.993 128 1.37875 0.491289
215 16 6953 6937 128.932 116 0.440148 0.491463
216 16 6991 6975 129.039 152 0.0512537 0.490873
217 16 7025 7009 129.07 136 0.425699 0.490622
218 16 7058 7042 129.083 132 0.320678 0.490707
219 16 7090 7074 129.078 128 0.332081 0.490646
2020-03-09 14:17:35.726780 min lat: 0.0198839 max lat: 2.29396 avg lat: 0.490969
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 7119 7103 129.019 116 0.187146 0.490969
221 16 7153 7137 129.05 136 1.31869 0.491192
222 16 7182 7166 128.991 116 1.09072 0.491164
223 16 7213 7197 128.969 124 0.293644 0.491235
224 16 7244 7228 128.946 124 0.0474558 0.491304
225 16 7275 7259 128.924 124 1.72382 0.491409
226 16 7310 7294 128.97 140 0.293927 0.491241
227 16 7342 7326 128.966 128 0.475732 0.491443
228 16 7371 7355 128.909 116 0.234139 0.491569
229 16 7404 7388 128.922 132 0.387129 0.491578
230 16 7436 7420 128.918 128 1.29579 0.491705
231 16 7469 7453 128.931 132 0.0396863 0.49128
232 16 7500 7484 128.91 124 0.347787 0.491256
233 16 7532 7516 128.905 128 1.7789 0.491281
234 16 7569 7553 128.986 148 0.27894 0.491244
235 16 7598 7582 128.931 116 0.175157 0.49163
236 16 7628 7612 128.893 120 0.332206 0.491476
237 16 7662 7646 128.923 136 0.0741658 0.491593
238 16 7695 7679 128.935 132 0.237976 0.491317
239 16 7727 7711 128.931 128 0.291509 0.491187
2020-03-09 14:17:55.739590 min lat: 0.0198839 max lat: 2.29396 avg lat: 0.491305
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 7760 7744 128.944 132 0.177061 0.491305
241 16 7794 7778 128.973 136 0.0300137 0.491321
242 16 7826 7810 128.969 128 0.196213 0.49129
243 16 7854 7838 128.899 112 1.56495 0.491454
244 16 7887 7871 128.912 132 0.219027 0.49148
245 16 7919 7903 128.908 128 0.199531 0.49131
246 16 7951 7935 128.904 128 0.0337205 0.491566
247 16 7989 7973 128.997 152 1.6093 0.491123
248 16 8022 8006 129.009 132 0.18837 0.490774
249 16 8058 8042 129.069 144 0.275771 0.490829
250 16 8087 8071 129.017 116 0.0723554 0.491164
251 16 8120 8104 129.028 132 0.0524736 0.491132
252 16 8156 8140 129.088 144 0.150136 0.491018
253 16 8185 8169 129.036 116 0.156316 0.490976
254 16 8220 8204 129.079 140 0.177453 0.491009
255 16 8253 8237 129.09 132 0.313919 0.490764
256 16 8287 8271 129.117 136 1.60619 0.490697
257 16 8317 8301 129.081 120 0.163976 0.490834
258 16 8349 8333 129.077 128 1.58042 0.490968
259 16 8382 8366 129.088 132 1.49256 0.490914
2020-03-09 14:18:15.745674 min lat: 0.0198839 max lat: 2.29396 avg lat: 0.491011
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 15 8411 8396 129.053 120 1.49597 0.491011
261 16 8445 8429 129.063 132 0.311964 0.490963
262 16 8474 8458 129.013 116 1.77077 0.491087
263 16 8511 8495 129.085 148 0.0429818 0.490615
264 16 8546 8530 129.126 140 0.197586 0.490597
265 16 8581 8565 129.167 140 1.16235 0.490927
266 16 8611 8595 129.132 120 0.26186 0.490821
267 15 8644 8629 129.158 136 1.02237 0.491053
268 16 8672 8656 129.079 108 0.350566 0.491047
269 16 8703 8687 129.06 124 0.0451803 0.490987
270 16 8737 8721 129.085 136 0.0226707 0.490842
271 16 8766 8750 129.037 116 0.062004 0.491211
272 16 8795 8779 128.989 116 1.57246 0.491425
273 16 8826 8810 128.97 124 0.228278 0.491391
274 16 8853 8837 128.893 108 0.0687488 0.49166
275 16 8888 8872 128.933 140 0.0455489 0.491556
276 16 8919 8903 128.915 124 0.249098 0.491537
277 16 8951 8935 128.912 128 0.258679 0.491789
278 16 8982 8966 128.894 124 0.980672 0.491932
279 16 9010 8994 128.834 112 0.229116 0.492083
2020-03-09 14:18:35.769350 min lat: 0.0198839 max lat: 2.35194 avg lat: 0.491882
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 9041 9025 128.81 124 0.386641 0.491882
281 16 9077 9061 128.864 144 1.47453 0.491981
282 16 9113 9097 128.917 144 0.339502 0.491438
283 16 9145 9129 128.913 128 0.0959521 0.491708
284 16 9178 9162 128.924 132 0.224604 0.491489
285 16 9210 9194 128.92 128 1.20296 0.49182
286 16 9247 9231 128.987 148 0.32857 0.49156
287 16 9280 9264 128.997 132 0.199186 0.49137
288 16 9311 9295 128.98 124 1.17839 0.49138
289 16 9345 9329 129.003 136 0.862595 0.491521
290 16 9379 9363 129.027 136 0.300099 0.491356
291 16 9412 9396 129.037 132 0.161756 0.491219
292 16 9444 9428 129.032 128 0.13603 0.491136
293 16 9476 9460 129.029 128 1.38082 0.491174
294 16 9510 9494 129.052 136 0.0663548 0.491154
295 15 9547 9532 129.129 152 0.0491553 0.491014
296 16 9581 9565 129.139 132 0.352725 0.49085
297 16 9610 9594 129.095 116 0.137058 0.490972
298 16 9644 9628 129.118 136 0.1332 0.490998
299 16 9674 9658 129.087 120 1.18995 0.491188
2020-03-09 14:18:55.781087 min lat: 0.0198839 max lat: 2.35194 avg lat: 0.491059
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 9708 9692 129.11 136 0.0331784 0.491059
301 16 9743 9727 129.146 140 0.0285308 0.491034
302 16 9772 9756 129.103 116 0.222227 0.490774
303 16 9804 9788 129.098 128 1.70105 0.491047
304 16 9834 9818 129.068 120 1.44688 0.491285
305 16 9865 9849 129.052 124 0.19288 0.491161
306 16 9899 9883 129.074 136 1.53346 0.491208
307 16 9933 9917 129.097 136 1.3147 0.491168
308 16 9970 9954 129.158 148 0.0487696 0.49082
309 16 10006 9990 129.206 144 0.201482 0.490437
310 15 10036 10021 129.188 124 0.206617 0.490465
311 16 10070 10054 129.197 132 0.0232912 0.490727
312 16 10102 10086 129.193 128 1.45591 0.490649
313 16 10134 10118 129.19 128 0.2205 0.490793
314 16 10162 10146 129.135 112 1.28262 0.491086
315 16 10192 10176 129.106 120 0.408311 0.491056
316 16 10226 10210 129.127 136 0.239141 0.49082
317 16 10257 10241 129.111 124 0.21391 0.490973
318 16 10286 10270 129.07 116 0.378004 0.491026
319 16 10318 10302 129.066 128 0.241126 0.491264
2020-03-09 14:19:15.790044 min lat: 0.019786 max lat: 2.35194 avg lat: 0.491518
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 16 10346 10330 129.012 112 0.234121 0.491518
321 16 10381 10365 129.046 140 0.153162 0.491359
322 16 10410 10394 129.006 116 0.177621 0.491353
323 16 10446 10430 129.041 144 0.0287919 0.491359
324 16 10474 10458 128.988 112 1.60249 0.491507
325 16 10506 10490 128.985 128 1.59277 0.491495
326 16 10536 10520 128.957 120 0.206458 0.491606
327 16 10572 10556 129.003 144 0.17232 0.491646
328 16 10598 10582 128.927 104 0.217666 0.491672
329 16 10628 10612 128.899 120 0.0714257 0.491657
330 16 10663 10647 128.931 140 0.440115 0.491936
331 16 10694 10678 128.915 124 0.0709322 0.491827
332 16 10723 10707 128.876 116 0.0598779 0.491949
333 15 10757 10742 128.909 140 0.236215 0.491691
334 16 10790 10774 128.906 128 0.143116 0.491915
335 16 10820 10804 128.88 120 1.48811 0.491884
336 16 10853 10837 128.889 132 0.568778 0.491976
337 16 10887 10871 128.91 136 0.0806111 0.49183
338 16 10920 10904 128.919 132 1.89901 0.491806
339 16 10950 10934 128.892 120 0.431437 0.491919
2020-03-09 14:19:35.832713 min lat: 0.019786 max lat: 2.35194 avg lat: 0.491975
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 16 10982 10966 128.89 128 0.0308958 0.491975
341 16 11011 10995 128.851 116 1.6588 0.491907
342 16 11045 11029 128.872 136 0.359903 0.492043
343 16 11076 11060 128.858 124 0.142049 0.492059
344 16 11106 11090 128.831 120 0.325222 0.492177
345 16 11142 11126 128.875 144 0.401054 0.49203
346 16 11173 11157 128.861 124 0.0712791 0.492122
347 16 11201 11185 128.812 112 0.0359587 0.492192
348 16 11232 11216 128.798 124 0.263129 0.492676
349 16 11262 11246 128.772 120 0.0495504 0.49273
350 16 11290 11274 128.724 112 0.436102 0.492964
351 16 11325 11309 128.756 140 0.833385 0.492704
352 16 11355 11339 128.731 120 1.01503 0.492859
353 16 11384 11368 128.695 116 0.0685347 0.492875
354 16 11414 11398 128.67 120 0.148765 0.492831
355 16 11448 11432 128.691 136 0.0506394 0.492722
356 16 11475 11459 128.633 108 0.074135 0.492964
357 15 11509 11494 128.664 140 0.219638 0.492796
358 16 11543 11527 128.673 132 0.995272 0.493022
359 16 11573 11557 128.648 120 0.0206915 0.492867
2020-03-09 14:19:55.846824 min lat: 0.019786 max lat: 2.35194 avg lat: 0.492825
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 11608 11592 128.68 140 0.0879656 0.492825
361 16 11642 11626 128.7 136 1.44311 0.492758
362 16 11672 11656 128.676 120 0.373551 0.492935
363 16 11701 11685 128.641 116 0.267157 0.492848
364 16 11737 11721 128.679 144 0.215786 0.492787
365 16 11772 11756 128.71 140 0.264887 0.49256
366 16 11801 11785 128.675 116 1.71657 0.492852
367 16 11832 11816 128.663 124 1.44494 0.492947
368 16 11863 11847 128.65 124 1.50337 0.492934
369 16 11898 11882 128.681 140 1.58296 0.492794
370 16 11928 11912 128.657 120 1.75785 0.492807
371 16 11961 11945 128.666 132 0.178991 0.492926
372 16 11996 11980 128.696 140 0.0613502 0.492963
373 16 12029 12013 128.705 132 0.211431 0.492761
374 15 12057 12042 128.67 116 0.186892 0.492973
375 16 12094 12078 128.711 144 0.174732 0.492628
376 15 12127 12112 128.73 136 1.28581 0.49271
377 16 12158 12142 128.706 120 1.47145 0.492728
378 16 12189 12173 128.694 124 0.0205903 0.49269
379 16 12219 12203 128.671 120 0.069476 0.492771
2020-03-09 14:20:15.867479 min lat: 0.0197829 max lat: 2.35194 avg lat: 0.492937
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 12250 12234 128.658 124 0.0326588 0.492937
381 16 12285 12269 128.688 140 0.156159 0.492682
382 16 12315 12299 128.665 120 1.22456 0.492986
383 16 12344 12328 128.632 116 0.237924 0.493066
384 16 12379 12363 128.662 140 0.101755 0.492823
385 16 12413 12397 128.681 136 0.272619 0.492923
386 16 12445 12429 128.679 128 1.14942 0.492995
387 16 12476 12460 128.667 124 0.224306 0.49283
388 16 12511 12495 128.696 140 0.267396 0.492779
389 16 12549 12533 128.756 152 0.179397 0.492657
390 16 12575 12559 128.692 104 0.219564 0.492839
391 16 12605 12589 128.67 120 0.178358 0.492874
392 16 12645 12629 128.749 160 1.3368 0.492667
393 16 12677 12661 128.747 128 1.4419 0.4926
394 16 12709 12693 128.744 128 0.385929 0.492788
395 16 12741 12725 128.742 128 0.254265 0.492577
396 16 12776 12760 128.764 140 0.0521672 0.492214
397 16 12811 12795 128.792 140 2.39436 0.492282
398 16 12847 12831 128.822 144 0.713194 0.492362
399 16 12881 12865 128.84 136 0.376951 0.492356
2020-03-09 14:20:35.920429 min lat: 0.0197829 max lat: 2.39436 avg lat: 0.492544
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 12909 12893 128.798 112 0.25237 0.492544
401 16 12942 12926 128.806 132 0.31392 0.492272
402 16 12969 12953 128.754 108 0.299852 0.4925
403 16 13005 12989 128.792 144 0.313795 0.492555
404 16 13036 13020 128.78 124 0.315625 0.492574
405 16 13073 13057 128.827 148 0.416656 0.492385
406 16 13101 13085 128.786 112 0.222998 0.492428
407 16 13134 13118 128.794 132 0.244173 0.492491
408 15 13162 13147 128.762 116 0.0969783 0.492565
409 16 13201 13185 128.819 152 0.0720224 0.492474
410 16 13235 13219 128.836 136 1.16168 0.492538
411 16 13270 13254 128.863 140 0.036247 0.492327
412 16 13300 13284 128.841 120 0.271222 0.492486
413 16 13331 13315 128.829 124 1.11787 0.492434
414 16 13358 13342 128.779 108 0.250923 0.492527
415 16 13389 13373 128.767 124 0.0231837 0.492637
416 15 13422 13407 128.784 136 0.0498244 0.492682
417 16 13457 13441 128.801 136 0.0885117 0.492485
418 16 13490 13474 128.808 132 0.0763648 0.492563
419 16 13520 13504 128.787 120 0.212707 0.492394
2020-03-09 14:20:55.931522 min lat: 0.0197829 max lat: 2.39436 avg lat: 0.492491
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 13556 13540 128.823 144 0.0419348 0.492491
421 16 13587 13571 128.811 124 0.377949 0.492279
422 16 13619 13603 128.809 128 0.195854 0.492553
423 16 13646 13630 128.76 108 0.0835241 0.492499
424 16 13680 13664 128.777 136 0.21761 0.492531
425 16 13708 13692 128.737 112 0.244593 0.492776
426 16 13736 13720 128.698 112 0.287423 0.492787
427 16 13773 13757 128.743 148 0.0432397 0.492716
428 16 13802 13786 128.713 116 1.64474 0.492827
429 16 13835 13819 128.721 132 0.262212 0.492698
430 16 13869 13853 128.737 136 0.203097 0.492685
431 15 13897 13882 128.707 116 0.0520222 0.492917
432 16 13931 13915 128.715 132 0.175716 0.492847
433 16 13966 13950 128.74 140 0.285042 0.492689
434 15 13995 13980 128.72 120 0.658221 0.492814
435 15 14024 14009 128.691 116 0.178883 0.492958
436 16 14057 14041 128.689 128 0.196827 0.492869
437 16 14085 14069 128.65 112 0.165573 0.492992
438 15 14120 14105 128.685 144 0.0217068 0.493053
439 16 14151 14135 128.665 120 1.40709 0.49308
2020-03-09 14:21:15.946363 min lat: 0.0197829 max lat: 2.39436 avg lat: 0.493203
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 14179 14163 128.627 112 1.47629 0.493203
441 16 14215 14199 128.662 144 0.0853626 0.493081
442 16 14254 14238 128.724 156 0.221471 0.492772
443 16 14289 14273 128.749 140 0.230987 0.492739
444 16 14321 14305 128.747 128 0.297109 0.49275
445 16 14349 14333 128.71 112 1.09776 0.492916
446 15 14381 14366 128.716 132 0.12594 0.492769
447 16 14415 14399 128.723 132 0.146776 0.492748
448 16 14446 14430 128.712 124 0.352516 0.493012
449 16 14478 14462 128.71 128 0.249134 0.492915
450 16 14507 14491 128.682 116 0.0932466 0.492935
451 16 14539 14523 128.68 128 1.81653 0.49301
452 16 14577 14561 128.732 152 0.566595 0.492914
453 16 14611 14595 128.748 136 0.36238 0.492737
454 16 14643 14627 128.746 128 0.825547 0.492745
455 15 14674 14659 128.743 128 0.210759 0.492725
456 16 14710 14694 128.768 140 0.0225277 0.492751
457 16 14741 14725 128.758 124 0.259295 0.492682
458 16 14771 14755 128.738 120 0.156868 0.492717
459 16 14805 14789 128.754 136 1.39997 0.49273
2020-03-09 14:21:35.961346 min lat: 0.0197829 max lat: 2.39436 avg lat: 0.492733
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 14837 14821 128.752 128 0.0201961 0.492733
461 16 14868 14852 128.742 124 0.133854 0.492855
462 15 14904 14889 128.783 148 0.230342 0.492547
463 16 14933 14917 128.747 112 0.425118 0.492613
464 16 14967 14951 128.762 136 0.371976 0.492716
465 16 15004 14988 128.803 148 0.260281 0.492592
466 16 15030 15014 128.75 104 0.0424625 0.492609
467 16 15075 15059 128.86 180 0.431316 0.492154
468 16 15105 15089 128.841 120 0.243816 0.492406
469 16 15138 15122 128.847 132 0.326468 0.492313
470 16 15169 15153 128.837 124 0.0220026 0.492382
471 16 15197 15181 128.797 112 0.0761229 0.492454
472 15 15231 15216 128.821 140 1.53399 0.492607
473 16 15265 15249 128.827 132 0.248291 0.492598
474 16 15298 15282 128.834 132 0.0341461 0.49242
475 16 15330 15314 128.832 128 0.0204142 0.492312
476 16 15362 15346 128.83 128 0.356248 0.492415
477 16 15397 15381 128.854 140 0.191772 0.49233
478 15 15425 15410 128.826 116 0.211521 0.492388
479 16 15460 15444 128.841 136 0.244305 0.492445
2020-03-09 14:21:55.987290 min lat: 0.0197829 max lat: 2.39436 avg lat: 0.492472
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 15495 15479 128.864 140 0.542079 0.492472
481 16 15528 15512 128.87 132 0.432979 0.492447
482 15 15557 15542 128.851 120 1.27866 0.492487
483 16 15585 15569 128.808 108 0.452379 0.492445
484 16 15618 15602 128.815 132 0.0219412 0.492504
485 15 15647 15632 128.788 120 0.0973248 0.492639
486 16 15678 15662 128.77 120 0.02203 0.492745
487 16 15717 15701 128.825 156 0.0557803 0.492306
488 16 15750 15734 128.832 132 1.67333 0.492398
489 16 15783 15767 128.838 132 0.0436727 0.492468
490 16 15818 15802 128.86 140 0.179463 0.492364
491 15 15856 15841 128.915 156 0.466843 0.492076
492 16 15889 15873 128.913 128 0.155933 0.492155
493 16 15922 15906 128.919 132 0.268353 0.49211
494 16 15958 15942 128.95 144 0.297482 0.491754
495 15 15986 15971 128.923 116 0.0504102 0.491984
496 16 16022 16006 128.946 140 1.39849 0.492128
497 16 16054 16038 128.944 128 0.0908147 0.491885
498 16 16085 16069 128.933 124 0.084014 0.492046
499 16 16117 16101 128.931 128 0.444321 0.492112
2020-03-09 14:22:16.034211 min lat: 0.0197829 max lat: 2.39436 avg lat: 0.492122
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 16148 16132 128.921 124 1.21853 0.492122
501 16 16176 16160 128.887 112 1.73925 0.492174
502 16 16207 16191 128.877 124 0.277324 0.492387
503 16 16236 16220 128.851 116 0.238157 0.492364
504 16 16269 16253 128.857 132 0.187329 0.492342
505 16 16301 16285 128.856 128 0.266932 0.492337
506 16 16333 16317 128.854 128 1.90655 0.49232
507 16 16361 16345 128.82 112 0.0263439 0.492411
508 16 16396 16380 128.842 140 0.23099 0.492531
509 16 16425 16409 128.817 116 0.306136 0.492439
510 16 16461 16445 128.846 144 0.0483066 0.492355
511 16 16492 16476 128.837 124 0.340712 0.492339
512 16 16524 16508 128.835 128 0.446706 0.492331
513 16 16555 16539 128.826 124 0.306831 0.49248
514 16 16586 16570 128.816 124 0.398609 0.492589
515 16 16618 16602 128.815 128 0.23628 0.492496
516 16 16650 16634 128.813 128 0.0700872 0.492542
517 16 16684 16668 128.827 136 0.233433 0.492331
518 16 16715 16699 128.817 124 0.157196 0.492641
519 16 16746 16730 128.808 124 0.080323 0.4925
2020-03-09 14:22:36.044284 min lat: 0.0197829 max lat: 2.4451 avg lat: 0.492391
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 16782 16766 128.837 144 0.0908164 0.492391
521 16 16815 16799 128.843 132 0.476845 0.492301
522 16 16851 16835 128.872 144 0.0896802 0.492434
523 16 16884 16868 128.877 132 0.0449508 0.492421
524 16 16920 16904 128.906 144 0.783382 0.492327
525 15 16950 16935 128.896 124 0.586525 0.492388
526 16 16980 16964 128.871 116 0.0621543 0.492412
527 16 17012 16996 128.87 128 0.408836 0.492329
528 16 17051 17035 128.92 156 1.67982 0.492279
529 16 17084 17068 128.926 132 1.22913 0.492165
530 16 17119 17103 128.947 140 0.101223 0.492206
531 16 17155 17139 128.975 144 1.34578 0.491957
532 15 17191 17176 129.01 148 0.1107 0.491861
533 16 17225 17209 129.015 132 1.25785 0.491807
534 16 17255 17239 128.998 120 0.0653489 0.491755
535 15 17282 17267 128.966 112 0.208171 0.491928
536 16 17315 17299 128.964 128 0.313139 0.492094
537 16 17345 17329 128.948 120 0.735112 0.492071
538 16 17380 17364 128.966 140 0.021429 0.491999
539 16 17414 17398 128.979 136 0.406489 0.492057
2020-03-09 14:22:56.070701 min lat: 0.0197829 max lat: 2.4451 avg lat: 0.492151
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 17443 17427 128.955 116 1.00099 0.492151
541 16 17476 17460 128.961 132 0.971481 0.492127
542 16 17507 17491 128.952 124 0.68532 0.492186
543 16 17538 17522 128.942 124 0.220592 0.492132
544 16 17568 17552 128.926 120 0.14927 0.492168
545 16 17599 17583 128.917 124 0.231191 0.492159
546 16 17626 17610 128.878 108 1.94105 0.492308
547 16 17660 17644 128.891 136 0.0593207 0.492337
548 16 17688 17672 128.86 112 1.38285 0.492492
549 16 17716 17700 128.83 112 0.325558 0.492565
550 16 17748 17732 128.828 128 0.973484 0.492661
551 16 17784 17768 128.856 144 0.155631 0.492479
552 16 17816 17800 128.854 128 1.07623 0.49258
553 16 17852 17836 128.881 144 0.306735 0.492376
554 16 17884 17868 128.879 128 1.72593 0.492319
555 16 17916 17900 128.878 128 0.281777 0.492351
556 16 17943 17927 128.84 108 0.40143 0.49241
557 16 17975 17959 128.838 128 1.40423 0.492486
558 16 18007 17991 128.837 128 1.49951 0.492528
559 16 18038 18022 128.828 124 0.186343 0.492535
2020-03-09 14:23:16.081103 min lat: 0.0197829 max lat: 2.4451 avg lat: 0.492565
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 18071 18055 128.833 132 1.17569 0.492565
561 16 18103 18087 128.831 128 0.200221 0.492441
562 16 18135 18119 128.83 128 0.20541 0.492371
563 15 18168 18153 128.842 136 0.456806 0.49241
564 16 18204 18188 128.862 140 0.0649121 0.492384
565 16 18233 18217 128.839 116 0.0274667 0.492404
566 16 18264 18248 128.831 124 0.315095 0.492614
567 16 18297 18281 128.836 132 0.491125 0.492562
568 16 18327 18311 128.821 120 0.197906 0.492617
569 16 18355 18339 128.791 112 0.0711673 0.492672
570 16 18384 18368 128.769 116 0.188686 0.492716
571 16 18418 18402 128.781 136 0.267251 0.492787
572 16 18451 18435 128.787 132 0.180626 0.492769
573 16 18477 18461 128.743 104 0.453697 0.492867
574 16 18507 18491 128.728 120 0.462903 0.492904
575 16 18539 18523 128.727 128 1.1521 0.492964
576 16 18567 18551 128.698 112 1.13147 0.493135
577 16 18595 18579 128.669 112 0.288448 0.49305
578 16 18625 18609 128.654 120 0.0715439 0.493109
579 16 18657 18641 128.653 128 0.276054 0.493219
2020-03-09 14:23:36.086298 min lat: 0.0197829 max lat: 2.4451 avg lat: 0.49335
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 18682 18666 128.603 100 0.306074 0.49335
581 16 18716 18700 128.616 136 1.19553 0.493447
582 16 18748 18732 128.615 128 0.216555 0.493352
583 16 18780 18764 128.614 128 0.107101 0.493325
584 16 18813 18797 128.62 132 0.021941 0.493279
585 16 18849 18833 128.646 144 1.1934 0.49331
586 16 18885 18869 128.665 144 1.80438 0.4932
587 16 18916 18900 128.657 124 0.282091 0.493216
588 15 18949 18934 128.669 136 0.0688578 0.493132
589 16 18981 18965 128.661 124 1.17298 0.49327
590 16 19009 18993 128.633 112 0.104593 0.493249
591 15 19050 19035 128.699 168 0.191588 0.493082
592 16 19082 19066 128.691 124 1.17998 0.493128
593 16 19114 19098 128.689 128 0.0476302 0.493071
594 16 19143 19127 128.668 116 1.16177 0.493136
595 16 19174 19158 128.66 124 1.43278 0.493152
596 16 19213 19197 128.706 156 1.49273 0.493016
597 16 19244 19228 128.697 124 1.61806 0.492996
598 16 19276 19260 128.696 128 1.78827 0.492975
599 16 19308 19292 128.695 128 1.64029 0.493002
Total time run: 600.533
Total reads made: 19324
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 128.712
Average IOPS: 32
Stddev IOPS: 2.84268
Max IOPS: 45
Min IOPS: 25
Average Latency(s): 0.49342
Max latency(s): 2.4451
Min latency(s): 0.0197829
Total time run: 600.533
Total reads made: 19324
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 128.712
Average IOPS: 32
Stddev IOPS: 2.84268
Max IOPS: 45
Min IOPS: 25
Average Latency(s): 0.49342
Max latency(s): 2.4451
Min latency(s): 0.0197829
Random Read Test, pool with 256 PGs
podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 rand | tee --append $(date +%F)-rados-test-read-rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 13 13 0 0 0 - 0
1 15 44 29 115.763 116 0.189163 0.31611
2 16 77 61 121.858 128 0.0193401 0.430625
3 16 107 91 121.204 120 0.184531 0.450987
4 16 136 120 119.886 116 0.191766 0.4677
5 16 171 155 123.89 140 1.648 0.458556
6 16 205 189 125.887 136 0.0652734 0.451655
7 16 236 220 125.607 124 1.36441 0.471769
8 16 265 249 124.404 116 0.462344 0.481334
9 16 296 280 124.358 124 1.31412 0.482733
10 16 330 314 125.519 136 0.0497829 0.481064
11 16 365 349 126.833 140 1.39185 0.482003
12 16 398 382 127.255 132 0.226664 0.479182
13 15 427 412 126.692 120 0.711614 0.486657
14 15 460 445 127.068 132 0.23561 0.480579
15 16 490 474 126.298 116 0.130672 0.488069
16 16 523 507 126.653 132 0.00900812 0.481203
17 15 550 535 125.791 112 0.276632 0.486413
18 16 583 567 125.912 128 0.0482817 0.491239
19 15 616 601 126.442 136 1.51149 0.486239
2020-03-09 14:28:25.379343 min lat: 0.00759327 max lat: 1.92824 avg lat: 0.488058
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 648 632 126.319 124 0.0240633 0.488058
21 16 678 662 126.017 120 0.0675982 0.488332
22 16 710 694 126.107 128 1.53312 0.490658
23 15 741 726 126.188 128 0.345689 0.489675
24 16 771 755 125.763 116 0.194888 0.492858
25 16 801 785 125.532 120 0.377 0.493429
26 16 834 818 125.78 132 1.7753 0.496889
27 16 863 847 125.417 116 0.217836 0.498455
28 16 897 881 125.789 136 1.32996 0.497362
29 16 928 912 125.727 124 1.41115 0.496599
30 16 957 941 125.401 116 0.0553987 0.497336
31 16 990 974 125.613 132 0.0492972 0.498332
32 16 1019 1003 125.312 116 1.16908 0.499297
33 16 1050 1034 125.271 124 1.57998 0.497278
34 16 1084 1068 125.586 136 0.291149 0.497064
35 16 1113 1097 125.312 116 1.47583 0.499034
36 16 1145 1129 125.386 128 0.236787 0.498409
37 16 1177 1161 125.456 128 0.0507549 0.500018
38 16 1212 1196 125.838 140 0.220674 0.498035
39 16 1244 1228 125.893 128 0.265011 0.496612
2020-03-09 14:28:45.383869 min lat: 0.00758844 max lat: 1.95098 avg lat: 0.496939
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 1280 1264 126.345 144 0.0737707 0.496939
41 16 1309 1293 126.089 116 1.16005 0.497671
42 16 1339 1323 125.944 120 0.318852 0.497165
43 16 1371 1355 125.991 128 0.234976 0.497265
44 16 1397 1381 125.491 104 0.452323 0.501171
45 15 1430 1415 125.724 136 0.00916022 0.501471
46 16 1462 1446 125.686 124 0.334006 0.500406
47 16 1493 1477 125.65 124 0.662744 0.500872
48 16 1526 1510 125.782 132 0.206438 0.499631
49 16 1554 1538 125.5 112 0.308269 0.49914
50 15 1587 1572 125.626 136 0.224301 0.499158
51 16 1614 1598 125.202 104 0.210811 0.500811
52 16 1642 1626 124.948 112 0.47214 0.501813
53 16 1675 1659 125.08 132 1.73754 0.503802
54 16 1704 1688 124.912 116 0.324704 0.503105
55 16 1737 1721 125.04 132 0.161342 0.502503
56 16 1773 1757 125.378 144 0.115535 0.50217
57 16 1803 1787 125.283 120 0.0563292 0.501758
58 16 1835 1819 125.329 128 0.359336 0.500746
59 16 1865 1849 125.239 120 0.254863 0.501712
2020-03-09 14:29:05.421934 min lat: 0.00758844 max lat: 2.0412 avg lat: 0.502075
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 1898 1882 125.351 132 0.14297 0.502075
61 16 1930 1914 125.394 128 0.147922 0.502426
62 16 1965 1949 125.629 140 0.364127 0.500649
63 15 1994 1979 125.539 120 1.47404 0.50185
64 16 2026 2010 125.515 124 0.0561347 0.501803
65 16 2059 2043 125.614 132 0.307339 0.500391
66 16 2089 2073 125.529 120 0.172544 0.50154
67 16 2122 2106 125.625 132 1.40518 0.501262
68 16 2151 2135 125.484 116 1.64331 0.501343
69 15 2184 2169 125.635 136 0.261359 0.501214
70 16 2217 2201 125.669 128 0.203083 0.501397
71 15 2244 2229 125.476 112 0.0595244 0.502064
72 16 2277 2261 125.511 128 0.0934983 0.501999
73 16 2312 2296 125.709 140 0.0643922 0.50122
74 16 2341 2325 125.577 116 0.216984 0.502041
75 16 2368 2352 125.343 108 1.46016 0.503179
76 16 2401 2385 125.43 132 0.0425867 0.503542
77 16 2438 2422 125.723 148 0.0797482 0.501424
78 16 2471 2455 125.798 132 0.281652 0.500717
79 16 2508 2492 126.079 148 0.208958 0.500398
2020-03-09 14:29:25.428699 min lat: 0.00758844 max lat: 2.0412 avg lat: 0.500447
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
80 15 2540 2525 126.152 132 0.0655898 0.500447
81 16 2570 2554 126.026 116 0.567935 0.499762
82 16 2607 2591 126.294 148 0.362565 0.499377
83 16 2642 2626 126.459 140 0.196834 0.499217
84 16 2679 2663 126.715 148 0.238598 0.497743
85 16 2711 2695 126.729 128 0.243838 0.497581
86 16 2747 2731 126.929 144 0.443474 0.49703
87 16 2780 2764 126.987 132 0.252554 0.497319
88 16 2808 2792 126.817 112 0.334884 0.497985
89 16 2837 2821 126.695 116 1.26582 0.498803
90 16 2868 2852 126.665 124 0.16486 0.498447
91 16 2901 2885 126.724 132 1.47612 0.497824
92 16 2933 2917 126.737 128 0.183519 0.497762
93 16 2966 2950 126.794 132 0.0214082 0.498126
94 16 2997 2981 126.764 124 0.0473124 0.498186
95 16 3030 3014 126.818 132 0.269154 0.498132
96 16 3062 3046 126.83 128 0.0525916 0.497567
97 16 3096 3080 126.924 136 0.0901356 0.49752
98 16 3127 3111 126.893 124 0.188486 0.497984
99 16 3162 3146 127.025 140 0.177758 0.497015
2020-03-09 14:29:45.433613 min lat: 0.00758844 max lat: 2.0412 avg lat: 0.496782
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
100 16 3199 3183 127.235 148 0.0539025 0.496782
101 15 3230 3215 127.242 128 0.629777 0.497503
102 16 3266 3250 127.367 140 1.38567 0.49635
103 16 3300 3284 127.45 136 0.276637 0.495667
104 16 3332 3316 127.455 128 0.0709163 0.495696
105 16 3364 3348 127.459 128 0.0724538 0.495799
106 16 3393 3377 127.351 116 0.383282 0.496338
107 16 3423 3407 127.282 120 0.768926 0.496838
108 15 3460 3445 127.511 152 0.381757 0.495984
109 15 3492 3477 127.515 128 0.071551 0.495776
110 16 3526 3510 127.556 132 1.3167 0.495523
111 16 3557 3541 127.523 124 0.0736356 0.49555
112 15 3592 3577 127.67 144 0.0713146 0.494824
113 15 3627 3612 127.778 140 0.04953 0.495249
114 16 3657 3641 127.674 116 0.248503 0.494823
115 16 3692 3676 127.781 140 1.18351 0.494717
116 16 3724 3708 127.783 128 0.0240328 0.494033
117 16 3754 3738 127.716 120 0.0489401 0.493972
118 16 3789 3773 127.82 140 0.463131 0.494389
119 16 3819 3803 127.753 120 0.610839 0.495552
2020-03-09 14:30:05.440702 min lat: 0.00758844 max lat: 2.12855 avg lat: 0.495371
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
120 16 3848 3832 127.654 116 1.33663 0.495371
121 16 3886 3870 127.855 152 0.064901 0.49446
122 16 3920 3904 127.922 136 0.0451795 0.494092
123 16 3953 3937 127.954 132 0.291511 0.493635
124 16 3984 3968 127.922 124 0.657468 0.494118
125 16 4020 4004 128.05 144 0.0485406 0.493872
126 16 4054 4038 128.113 136 1.66854 0.493653
127 16 4081 4065 127.954 108 0.0473508 0.494324
128 16 4116 4100 128.048 140 0.811657 0.494353
129 16 4149 4133 128.079 132 0.257035 0.494438
130 16 4182 4166 128.108 132 0.0948986 0.494335
131 16 4216 4200 128.168 136 1.08268 0.493817
132 16 4247 4231 128.136 124 0.0484442 0.494168
133 15 4279 4264 128.163 132 0.146956 0.493825
134 16 4311 4295 128.132 124 1.58947 0.493647
135 16 4345 4329 128.189 136 0.0451388 0.493772
136 16 4378 4362 128.216 132 0.0936799 0.493416
137 16 4417 4401 128.419 156 0.169335 0.492621
138 16 4445 4429 128.3 112 1.59991 0.492925
139 16 4479 4463 128.355 136 0.277477 0.49307
2020-03-09 14:30:25.450530 min lat: 0.00758844 max lat: 2.12855 avg lat: 0.493065
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
140 16 4511 4495 128.352 128 0.319201 0.493065
141 16 4545 4529 128.406 136 1.4205 0.49272
142 16 4581 4565 128.515 144 0.151326 0.492073
143 16 4613 4597 128.511 128 0.218008 0.492406
144 16 4644 4628 128.48 124 0.280231 0.492305
145 16 4674 4658 128.421 120 1.73267 0.49242
146 16 4703 4687 128.336 116 0.338129 0.493134
147 15 4735 4720 128.36 132 1.76812 0.493022
148 16 4766 4750 128.302 120 0.277118 0.493387
149 16 4800 4784 128.353 136 1.40389 0.493366
150 16 4835 4819 128.43 140 0.231478 0.49267
151 16 4865 4849 128.374 120 0.214272 0.493014
152 16 4898 4882 128.398 132 1.38171 0.493616
153 16 4929 4913 128.369 124 0.0673812 0.493461
154 16 4960 4944 128.341 124 0.104399 0.493742
155 16 4994 4978 128.39 136 0.273172 0.492907
156 16 5027 5011 128.413 132 0.025074 0.492787
157 16 5067 5051 128.614 160 0.080997 0.491795
158 16 5100 5084 128.635 132 0.280364 0.491865
159 16 5133 5117 128.656 132 0.223638 0.492138
2020-03-09 14:30:45.457259 min lat: 0.00758844 max lat: 2.12855 avg lat: 0.492326
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
160 16 5163 5147 128.602 120 1.53826 0.492326
161 16 5194 5178 128.573 124 0.222583 0.492617
162 16 5228 5212 128.618 136 1.46397 0.492283
163 16 5260 5244 128.609 128 1.23506 0.492249
164 16 5288 5272 128.507 112 0.522441 0.492651
165 16 5319 5303 128.479 124 0.397334 0.492819
166 16 5352 5336 128.5 132 0.0707879 0.493225
167 16 5383 5367 128.472 124 0.0436325 0.492632
168 16 5421 5405 128.611 152 0.281206 0.492555
169 16 5454 5438 128.631 132 0.213111 0.492124
170 15 5491 5476 128.768 152 0.220927 0.491671
171 16 5524 5508 128.763 128 0.337313 0.491937
172 15 5559 5544 128.852 144 0.226524 0.491718
173 16 5596 5580 128.938 144 1.2731 0.491177
174 16 5627 5611 128.909 124 0.1746 0.491333
175 16 5657 5641 128.858 120 0.523888 0.491296
176 16 5689 5673 128.852 128 0.072858 0.491689
177 15 5724 5709 128.935 144 1.40637 0.491236
178 16 5754 5738 128.863 116 0.223111 0.491507
179 16 5790 5774 128.947 144 0.519273 0.491727
2020-03-09 14:31:05.479185 min lat: 0.00758844 max lat: 2.12855 avg lat: 0.491175
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
180 16 5826 5810 129.03 144 0.316725 0.491175
181 16 5857 5841 129 124 0.0655163 0.49096
182 15 5885 5870 128.928 116 0.306004 0.491671
183 15 5915 5900 128.879 120 0.669872 0.491709
184 16 5946 5930 128.831 120 1.35695 0.491863
185 16 5979 5963 128.848 132 0.205182 0.491501
186 16 6008 5992 128.779 116 0.0579184 0.491649
187 16 6043 6027 128.839 140 0.536809 0.491505
188 16 6074 6058 128.813 124 0.108285 0.491614
189 16 6101 6085 128.703 108 0.31382 0.492289
190 16 6136 6120 128.761 140 1.40723 0.492137
191 16 6166 6150 128.715 120 0.179843 0.492069
192 16 6195 6179 128.649 116 0.0518371 0.492127
193 16 6226 6210 128.624 124 0.0500438 0.492554
194 16 6254 6238 128.539 112 0.346221 0.492518
195 16 6286 6270 128.536 128 1.71537 0.492747
196 16 6318 6302 128.533 128 1.33163 0.49283
197 16 6350 6334 128.53 128 0.0451284 0.492875
198 16 6382 6366 128.527 128 0.330289 0.493035
199 16 6415 6399 128.545 132 1.13619 0.493026
2020-03-09 14:31:25.487928 min lat: 0.00758844 max lat: 2.12855 avg lat: 0.492909
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
200 16 6447 6431 128.542 128 0.0579771 0.492909
201 16 6481 6465 128.579 136 0.241814 0.492578
202 16 6512 6496 128.556 124 1.28887 0.493017
203 16 6544 6528 128.553 128 0.297991 0.492795
204 16 6576 6560 128.551 128 0.0553758 0.49279
205 15 6610 6595 128.606 140 1.30937 0.492809
206 16 6642 6626 128.584 124 0.196741 0.492283
207 16 6672 6656 128.542 120 0.0477203 0.492571
208 16 6703 6687 128.519 124 0.912837 0.493192
209 16 6735 6719 128.516 128 0.1108 0.492738
210 16 6769 6753 128.551 136 0.294788 0.492828
211 16 6802 6786 128.567 132 1.58597 0.492765
212 16 6835 6819 128.583 132 0.258177 0.492763
213 16 6865 6849 128.541 120 0.236762 0.493015
214 16 6897 6881 128.539 128 0.238013 0.493043
215 16 6928 6912 128.517 124 0.206292 0.493122
216 16 6958 6942 128.478 120 0.21966 0.493296
217 16 6990 6974 128.462 128 0.444428 0.49317
218 16 7023 7007 128.478 132 1.28373 0.493538
219 16 7059 7043 128.547 144 0.243892 0.493325
2020-03-09 14:31:45.523298 min lat: 0.00758844 max lat: 2.12855 avg lat: 0.49324
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
220 16 7089 7073 128.508 120 0.996789 0.49324
221 16 7123 7107 128.542 136 0.145334 0.493051
222 16 7150 7134 128.45 108 1.32096 0.493415
223 15 7179 7164 128.412 120 0.0839449 0.493748
224 16 7212 7196 128.409 128 0.181908 0.493767
225 16 7247 7231 128.46 140 0.228171 0.493494
226 16 7282 7266 128.511 140 0.0488206 0.492914
227 16 7315 7299 128.526 132 1.41182 0.493104
228 16 7348 7332 128.541 132 0.291632 0.492987
229 15 7386 7371 128.66 156 0.157679 0.492435
230 16 7421 7405 128.692 136 0.192474 0.492397
231 16 7448 7432 128.603 108 0.303068 0.492405
232 15 7480 7465 128.617 132 0.0521112 0.492661
233 16 7512 7496 128.597 124 0.385918 0.492894
234 16 7542 7526 128.56 120 0.0511737 0.492914
235 16 7572 7556 128.524 120 0.542226 0.493334
236 16 7600 7584 128.453 112 0.201575 0.493548
237 16 7630 7614 128.417 120 0.0561037 0.49356
238 16 7670 7654 128.55 160 0.0849755 0.492901
239 16 7704 7688 128.581 136 0.265047 0.492865
2020-03-09 14:32:05.531148 min lat: 0.00758844 max lat: 2.15434 avg lat: 0.492841
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
240 16 7740 7724 128.645 144 0.0520922 0.492841
241 15 7769 7754 128.609 120 0.302236 0.492862
242 16 7803 7787 128.623 132 0.0548091 0.492658
243 16 7838 7822 128.67 140 0.285622 0.492711
244 16 7870 7854 128.666 128 1.1918 0.492883
245 15 7904 7889 128.712 140 0.0848028 0.492829
246 16 7936 7920 128.693 124 0.0449712 0.492728
247 15 7968 7953 128.706 132 0.256293 0.492704
248 16 7999 7983 128.67 120 0.148891 0.492835
249 16 8030 8014 128.651 124 0.578604 0.492981
250 16 8066 8050 128.712 144 0.248972 0.492468
251 16 8102 8086 128.773 144 0.434066 0.492089
252 16 8130 8114 128.706 112 0.167202 0.492406
253 16 8164 8148 128.735 136 0.327447 0.49214
254 16 8195 8179 128.716 124 0.244601 0.492364
255 16 8234 8218 128.823 156 0.475849 0.492149
256 16 8266 8250 128.82 128 0.0758576 0.492001
257 16 8298 8282 128.816 128 1.75357 0.492137
258 15 8323 8308 128.72 104 1.10742 0.492735
259 16 8355 8339 128.702 124 0.131107 0.49267
2020-03-09 14:32:25.540825 min lat: 0.00758844 max lat: 2.15434 avg lat: 0.492416
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
260 16 8389 8373 128.729 136 0.00857981 0.492416
261 16 8420 8404 128.71 124 0.0837172 0.492602
262 16 8452 8436 128.708 128 1.51838 0.492711
263 16 8488 8472 128.766 144 0.0882103 0.492343
264 16 8518 8502 128.732 120 1.74093 0.492447
265 16 8551 8535 128.745 132 0.504377 0.492406
266 16 8587 8571 128.802 144 0.0597395 0.492269
267 16 8622 8606 128.844 140 0.149485 0.492174
268 16 8651 8635 128.795 116 1.37556 0.492117
269 15 8683 8668 128.807 132 0.420735 0.492147
270 16 8714 8698 128.774 120 1.17478 0.492491
271 16 8746 8730 128.771 128 0.0611217 0.49214
272 16 8785 8769 128.87 156 0.435239 0.49194
273 16 8819 8803 128.896 136 0.198297 0.49167
274 16 8848 8832 128.848 116 0.274941 0.492066
275 16 8882 8866 128.874 136 0.495659 0.491967
276 16 8910 8894 128.812 112 0.0707601 0.492035
277 16 8941 8925 128.794 124 0.499243 0.49212
278 16 8972 8956 128.775 124 0.595984 0.492319
279 15 9006 8991 128.815 140 0.142209 0.492165
2020-03-09 14:32:45.556957 min lat: 0.00758844 max lat: 2.15434 avg lat: 0.49211
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
280 16 9042 9026 128.855 140 0.456657 0.49211
281 16 9074 9058 128.851 128 0.151224 0.49202
282 15 9103 9088 128.82 120 0.0545886 0.492224
283 16 9138 9122 128.845 136 1.60324 0.491955
284 16 9169 9153 128.828 124 0.461395 0.492232
285 16 9200 9184 128.811 124 0.309175 0.492278
286 16 9232 9216 128.808 128 0.0684568 0.492433
287 16 9270 9254 128.889 152 1.33219 0.492114
288 16 9296 9280 128.802 104 0.232691 0.492141
289 16 9325 9309 128.757 116 0.462623 0.492671
290 16 9355 9339 128.727 120 0.223172 0.492626
291 15 9386 9371 128.724 128 0.304125 0.492512
292 16 9421 9405 128.749 136 0.51039 0.49245
293 16 9456 9440 128.788 140 1.36367 0.49226
294 16 9489 9473 128.798 132 0.291399 0.492226
295 15 9515 9500 128.728 108 0.178472 0.492375
296 16 9543 9527 128.657 108 1.855 0.492608
297 16 9577 9561 128.682 136 0.0490071 0.492966
298 16 9606 9590 128.639 116 0.160699 0.492809
299 16 9641 9625 128.677 140 0.113257 0.492842
2020-03-09 14:33:05.565139 min lat: 0.00758844 max lat: 2.22367 avg lat: 0.492681
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
300 16 9672 9656 128.661 124 0.19061 0.492681
301 15 9707 9692 128.712 144 0.290928 0.492389
302 16 9739 9723 128.696 124 0.202738 0.492598
303 16 9771 9755 128.694 128 0.745036 0.492747
304 16 9805 9789 128.718 136 0.446637 0.492782
305 16 9841 9825 128.767 144 0.250422 0.492416
306 15 9871 9856 128.751 124 0.0686151 0.492577
307 16 9904 9888 128.749 128 1.63885 0.492691
308 16 9930 9914 128.668 104 0.670912 0.492898
309 15 9965 9950 128.718 144 0.731271 0.49294
310 16 9997 9981 128.703 124 1.2368 0.492815
311 16 10029 10013 128.7 128 0.318304 0.492673
312 16 10061 10045 128.698 128 0.272861 0.492736
313 16 10096 10080 128.733 140 1.52497 0.492731
314 16 10124 10108 128.68 112 0.640861 0.492957
315 16 10155 10139 128.665 124 0.779685 0.49307
316 16 10185 10169 128.637 120 1.02686 0.492947
317 16 10217 10201 128.635 128 0.19747 0.492794
318 16 10247 10231 128.607 120 0.157774 0.492928
319 16 10275 10259 128.555 112 1.45464 0.493131
2020-03-09 14:33:25.576584 min lat: 0.00758844 max lat: 2.22367 avg lat: 0.493064
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
320 15 10309 10294 128.591 140 0.343314 0.493064
321 16 10343 10327 128.601 132 0.197438 0.493072
322 16 10375 10359 128.598 128 0.0525443 0.493091
323 16 10409 10393 128.621 136 0.242701 0.493092
324 16 10437 10421 128.57 112 1.41396 0.493195
325 16 10470 10454 128.58 132 0.225383 0.493195
326 16 10501 10485 128.566 124 0.248241 0.493161
327 16 10534 10518 128.577 132 0.0774712 0.493306
328 15 10566 10551 128.587 132 1.41793 0.493123
329 15 10593 10578 128.524 108 0.259335 0.493244
330 16 10628 10612 128.547 136 0.0442344 0.493329
331 16 10663 10647 128.581 140 0.3253 0.493408
332 16 10696 10680 128.592 132 0.893511 0.493377
333 16 10728 10712 128.59 128 0.311234 0.493074
334 16 10764 10748 128.636 144 0.262498 0.493077
335 16 10797 10781 128.646 132 1.40241 0.492944
336 16 10830 10814 128.656 132 0.044292 0.492942
337 16 10864 10848 128.677 136 0.168146 0.492757
338 16 10897 10881 128.687 132 1.3778 0.492866
339 16 10934 10918 128.744 148 0.319363 0.492484
2020-03-09 14:33:45.584674 min lat: 0.00758844 max lat: 2.22367 avg lat: 0.492421
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
340 15 10971 10956 128.811 152 0.447902 0.492421
341 16 11000 10984 128.762 112 0.488588 0.492526
342 16 11032 11016 128.76 128 0.185412 0.492655
343 16 11060 11044 128.711 112 0.192753 0.492687
344 16 11090 11074 128.685 120 0.187793 0.492708
345 16 11121 11105 128.672 124 1.61635 0.492923
346 15 11153 11138 128.669 132 0.0609645 0.492882
347 16 11181 11165 128.609 108 1.56338 0.493049
348 15 11210 11195 128.584 120 0.241957 0.493386
349 16 11248 11232 128.64 148 0.195023 0.49309
350 16 11280 11264 128.638 128 1.49536 0.493015
351 16 11309 11293 128.602 116 0.00787264 0.493074
352 16 11339 11323 128.577 120 0.458127 0.493152
353 16 11372 11356 128.587 132 0.134739 0.493165
354 16 11405 11389 128.596 132 0.00822177 0.493143
355 16 11435 11419 128.572 120 0.850725 0.493414
356 16 11466 11450 128.559 124 0.0163893 0.493191
357 16 11497 11481 128.546 124 0.00973857 0.493324
358 15 11533 11518 128.6 148 0.0929121 0.493285
359 16 11567 11551 128.61 132 0.0446927 0.493259
2020-03-09 14:34:05.623724 min lat: 0.00758844 max lat: 2.22367 avg lat: 0.493115
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
360 16 11601 11585 128.63 136 0.185377 0.493115
361 16 11633 11617 128.629 128 0.101306 0.493093
362 16 11665 11649 128.627 128 0.319652 0.492944
363 15 11698 11683 128.647 136 0.167573 0.493022
364 16 11728 11712 128.612 116 1.35791 0.49309
365 16 11758 11742 128.588 120 0.446025 0.493355
366 16 11789 11773 128.576 124 0.249789 0.493407
367 16 11819 11803 128.552 120 0.253794 0.493391
368 15 11849 11834 128.54 124 1.52666 0.493368
369 16 11885 11869 128.571 140 0.200598 0.493246
370 15 11912 11897 128.526 112 0.297605 0.493556
371 16 11946 11930 128.535 132 0.338924 0.493423
372 15 11981 11966 128.576 144 0.0431001 0.493212
373 16 12016 12000 128.596 136 0.0616273 0.493273
374 16 12050 12034 128.615 136 0.0261335 0.493009
375 16 12079 12063 128.582 116 0.0195697 0.493146
376 16 12113 12097 128.601 136 0.547183 0.493402
377 16 12147 12131 128.621 136 0.0578911 0.493176
378 15 12183 12168 128.672 148 1.88786 0.49296
379 16 12215 12199 128.66 124 0.194814 0.492892
2020-03-09 14:34:25.630542 min lat: 0.00758844 max lat: 2.22367 avg lat: 0.492896
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
380 16 12250 12234 128.69 140 0.290504 0.492896
381 16 12283 12267 128.698 132 0.555411 0.49277
382 16 12316 12300 128.707 132 0.167732 0.492893
383 16 12352 12336 128.747 144 0.63584 0.492801
384 15 12381 12366 128.724 120 0.74141 0.492988
385 16 12414 12398 128.722 128 0.744432 0.492995
386 16 12445 12429 128.71 124 0.233459 0.492936
387 15 12483 12468 128.78 156 0.348882 0.492793
388 15 12524 12509 128.87 164 0.0533723 0.492405
389 16 12557 12541 128.868 128 0.200685 0.492314
390 16 12589 12573 128.865 128 0.058201 0.492312
391 15 12622 12607 128.884 136 1.5035 0.492203
392 16 12657 12641 128.902 136 0.237846 0.491977
393 16 12688 12672 128.889 124 0.262442 0.492089
394 16 12716 12700 128.846 112 0.304671 0.492372
395 15 12745 12730 128.824 120 1.51254 0.492375
396 16 12780 12764 128.842 136 0.278281 0.492278
397 16 12813 12797 128.85 132 0.503318 0.492341
398 16 12846 12830 128.858 132 0.584168 0.492396
399 16 12875 12859 128.826 116 0.555365 0.492404
2020-03-09 14:34:45.636586 min lat: 0.00758844 max lat: 2.22367 avg lat: 0.492408
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
400 16 12910 12894 128.853 140 1.11721 0.492408
401 16 12938 12922 128.811 112 0.324473 0.49247
402 16 12971 12955 128.819 132 1.15171 0.492567
403 16 13004 12988 128.827 132 0.28324 0.492369
404 15 13035 13020 128.825 128 0.568556 0.492572
405 16 13068 13052 128.823 128 0.285947 0.492328
406 16 13096 13080 128.781 112 0.983314 0.492578
407 16 13133 13117 128.828 148 1.34305 0.49241
408 16 13162 13146 128.797 116 1.21148 0.492506
409 16 13194 13178 128.795 128 1.42986 0.492525
410 16 13223 13207 128.764 116 0.0734785 0.492599
411 16 13254 13238 128.752 124 0.233044 0.492378
412 16 13286 13270 128.75 128 0.410644 0.492725
413 16 13319 13303 128.758 132 0.453446 0.492756
414 16 13349 13333 128.737 120 1.5458 0.492846
415 16 13383 13367 128.754 136 0.0106207 0.492789
416 16 13415 13399 128.752 128 0.395381 0.492615
417 16 13447 13431 128.75 128 0.2048 0.49283
418 16 13480 13464 128.758 132 1.3471 0.492711
419 16 13507 13491 128.709 108 0.0397955 0.492894
2020-03-09 14:35:05.639344 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.49295
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
420 16 13540 13524 128.716 132 0.0507944 0.49295
421 16 13572 13556 128.715 128 0.282947 0.492984
422 16 13606 13590 128.732 136 0.706111 0.493061
423 16 13636 13620 128.711 120 1.06421 0.492954
424 16 13668 13652 128.709 128 0.914274 0.493021
425 16 13704 13688 128.745 144 0.608122 0.492901
426 16 13735 13719 128.734 124 0.0564004 0.492908
427 15 13770 13755 128.77 144 0.0526887 0.492775
428 16 13806 13790 128.796 140 0.0443094 0.492592
429 16 13835 13819 128.766 116 0.0501112 0.492613
430 16 13865 13849 128.746 120 0.22598 0.492801
431 16 13895 13879 128.725 120 0.486399 0.492949
432 16 13925 13909 128.705 120 0.0852251 0.492927
433 16 13955 13939 128.685 120 0.0745961 0.492949
434 16 13989 13973 128.702 136 1.02644 0.493052
435 16 14025 14009 128.737 144 1.14398 0.492908
436 16 14055 14039 128.716 120 0.350549 0.492762
437 16 14088 14072 128.724 132 1.73861 0.492887
438 16 14119 14103 128.713 124 0.164738 0.492694
439 16 14154 14138 128.739 140 2.16674 0.492729
2020-03-09 14:35:25.644849 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.492933
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
440 16 14183 14167 128.71 116 1.15626 0.492933
441 16 14216 14200 128.717 132 1.39708 0.492857
442 16 14249 14233 128.724 132 0.0871006 0.492847
443 16 14279 14263 128.704 120 1.40747 0.492984
444 16 14317 14301 128.757 152 0.351914 0.49274
445 16 14351 14335 128.773 136 0.139878 0.492688
446 16 14377 14361 128.717 104 0.373503 0.492805
447 16 14407 14391 128.698 120 0.496401 0.492977
448 16 14438 14422 128.687 124 1.3126 0.493038
449 16 14467 14451 128.659 116 0.550495 0.493221
450 16 14500 14484 128.666 132 0.10734 0.493166
451 16 14535 14519 128.691 140 0.0491615 0.492978
452 16 14570 14554 128.716 140 0.050293 0.492956
453 16 14602 14586 128.714 128 0.0475525 0.492785
454 16 14631 14615 128.686 116 0.270657 0.492873
455 16 14663 14647 128.685 128 0.0872759 0.493021
456 15 14699 14684 128.727 148 1.01948 0.492978
457 16 14730 14714 128.707 120 1.19433 0.493034
458 16 14762 14746 128.706 128 0.209505 0.492863
459 16 14796 14780 128.721 136 0.00978147 0.492855
2020-03-09 14:35:45.652884 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.492738
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
460 16 14827 14811 128.711 124 0.209213 0.492738
461 16 14865 14849 128.762 152 0.0246178 0.492769
462 15 14894 14879 128.743 120 0.186257 0.492873
463 16 14929 14913 128.758 136 0.813413 0.49293
464 16 14961 14945 128.757 128 0.0718973 0.492902
465 16 14988 14972 128.712 108 0.700856 0.493065
466 15 15015 15000 128.676 112 1.45437 0.493088
467 16 15046 15030 128.657 120 0.982401 0.493312
468 16 15077 15061 128.647 124 1.32528 0.493185
469 16 15111 15095 128.663 136 0.13288 0.493218
470 16 15140 15124 128.636 116 1.41088 0.493255
471 16 15175 15159 128.66 140 0.24409 0.492975
472 15 15207 15192 128.667 132 0.581044 0.493001
473 16 15237 15221 128.64 116 0.0664894 0.493291
474 16 15270 15254 128.647 132 0.0535102 0.493187
475 16 15299 15283 128.62 116 1.53691 0.493409
476 16 15330 15314 128.611 124 1.35426 0.493402
477 16 15361 15345 128.601 124 0.110866 0.493399
478 16 15393 15377 128.6 128 0.405185 0.493416
479 16 15424 15408 128.59 124 0.226533 0.493389
2020-03-09 14:36:05.657299 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.493266
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
480 16 15459 15443 128.614 140 0.0733489 0.493266
481 16 15494 15478 128.637 140 0.0616342 0.493254
482 16 15522 15506 128.603 112 1.21014 0.493424
483 16 15551 15535 128.576 116 0.13167 0.493514
484 16 15583 15567 128.575 128 1.16737 0.493633
485 16 15612 15596 128.549 116 0.0367693 0.493533
486 16 15645 15629 128.556 132 0.397224 0.493586
487 16 15677 15661 128.555 128 0.281867 0.493638
488 16 15707 15691 128.537 120 0.216604 0.49357
489 16 15735 15719 128.503 112 0.343392 0.493664
490 16 15769 15753 128.519 136 0.213094 0.493635
491 16 15798 15782 128.493 116 0.184509 0.49382
492 16 15828 15812 128.476 120 1.36858 0.493852
493 16 15862 15846 128.491 136 0.399842 0.493788
494 16 15896 15880 128.506 136 0.00783099 0.493718
495 16 15926 15910 128.489 120 0.761711 0.493817
496 16 15962 15946 128.52 144 0.0600508 0.493766
497 16 15995 15979 128.527 132 0.162762 0.49367
498 16 16027 16011 128.526 128 0.293957 0.493739
499 16 16056 16040 128.501 116 0.0545122 0.493855
2020-03-09 14:36:25.663755 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.493762
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
500 16 16090 16074 128.516 136 0.0539079 0.493762
501 16 16128 16112 128.562 152 0.211005 0.493564
502 16 16160 16144 128.561 128 0.187399 0.493432
503 16 16192 16176 128.559 128 1.75977 0.493535
504 15 16227 16212 128.589 144 1.41888 0.493603
505 16 16267 16251 128.644 156 0.519815 0.493384
506 15 16298 16283 128.642 128 1.10476 0.493385
507 16 16333 16317 128.656 136 0.387107 0.493177
508 16 16367 16351 128.671 136 0.0528087 0.493226
509 16 16400 16384 128.677 132 0.0533276 0.493184
510 16 16436 16420 128.707 144 1.41282 0.492935
511 16 16471 16455 128.729 140 0.144499 0.492955
512 16 16508 16492 128.767 148 0.231932 0.492727
513 16 16545 16529 128.804 148 1.56851 0.492625
514 15 16583 16568 128.857 156 0.107234 0.492319
515 16 16618 16602 128.871 136 1.93003 0.492464
516 16 16649 16633 128.861 124 0.0800015 0.492454
517 16 16680 16664 128.852 124 0.50851 0.492514
518 15 16714 16699 128.873 140 0.249528 0.492439
519 16 16747 16731 128.872 128 0.126583 0.492352
2020-03-09 14:36:45.674097 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.492419
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
520 16 16780 16764 128.878 132 0.148523 0.492419
521 16 16815 16799 128.899 140 0.903131 0.492416
522 16 16844 16828 128.874 116 0.611226 0.49234
523 16 16880 16864 128.903 144 0.278245 0.492306
524 16 16914 16898 128.917 136 1.21513 0.492365
525 16 16945 16929 128.907 124 1.07566 0.492361
526 16 16975 16959 128.89 120 0.257593 0.492315
527 16 17004 16988 128.866 116 0.664407 0.492495
528 16 17039 17023 128.887 140 1.09811 0.492434
529 16 17068 17052 128.862 116 0.217041 0.492387
530 16 17098 17082 128.846 120 1.42617 0.492447
531 16 17132 17116 128.859 136 1.67557 0.492498
532 16 17161 17145 128.835 116 0.272223 0.492629
533 16 17192 17176 128.826 124 0.470807 0.492679
534 16 17230 17214 128.869 152 0.800189 0.492543
535 16 17259 17243 128.845 116 0.268247 0.492526
536 16 17286 17270 128.806 108 1.59423 0.492582
537 16 17319 17303 128.812 132 1.37513 0.492643
538 16 17353 17337 128.825 136 1.32731 0.492684
539 16 17386 17370 128.831 132 0.55438 0.492697
2020-03-09 14:37:05.677000 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.492739
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
540 16 17416 17400 128.815 120 0.00819962 0.492739
541 16 17447 17431 128.806 124 0.0714803 0.492737
542 16 17484 17468 128.841 148 0.570417 0.492588
543 16 17512 17496 128.81 112 0.618657 0.492652
544 16 17547 17531 128.83 140 1.34152 0.492587
545 16 17577 17561 128.814 120 0.315035 0.492582
546 16 17607 17591 128.798 120 0.410163 0.49261
547 16 17639 17623 128.797 128 0.303394 0.492792
548 16 17669 17653 128.781 120 0.444412 0.49282
549 16 17706 17690 128.815 148 1.64699 0.492705
550 16 17734 17718 128.785 112 0.0850279 0.4928
551 16 17764 17748 128.769 120 1.23936 0.492867
552 16 17797 17781 128.775 132 1.22765 0.492883
553 16 17827 17811 128.759 120 1.04583 0.492945
554 16 17857 17841 128.743 120 0.0581155 0.493009
555 16 17892 17876 128.763 140 0.529912 0.492957
556 16 17922 17906 128.747 120 0.270666 0.492965
557 16 17956 17940 128.76 136 0.516379 0.492999
558 16 17984 17968 128.73 112 0.496621 0.493072
559 16 18015 17999 128.721 124 0.0324908 0.493108
2020-03-09 14:37:25.685870 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.493198
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
560 16 18043 18027 128.691 112 0.251608 0.493198
561 16 18077 18061 128.704 136 0.214448 0.493023
562 16 18106 18090 128.681 116 0.431902 0.493103
563 16 18141 18125 128.701 140 0.487035 0.493054
564 16 18179 18163 128.742 152 1.40696 0.492956
565 16 18212 18196 128.748 132 0.216091 0.492718
566 16 18243 18227 128.739 124 0.212926 0.492741
567 16 18272 18256 128.717 116 1.85285 0.492963
568 16 18305 18289 128.722 132 0.56152 0.492974
569 16 18341 18325 128.749 144 1.37575 0.492928
570 16 18373 18357 128.747 128 0.223864 0.492992
571 16 18403 18387 128.732 120 1.36241 0.492997
572 16 18435 18419 128.731 128 1.63608 0.492984
573 15 18466 18451 128.729 128 0.0785299 0.493063
574 16 18499 18483 128.722 128 0.01434 0.492989
575 15 18534 18519 128.749 144 0.0794387 0.493002
576 16 18565 18549 128.733 120 1.299 0.492965
577 16 18601 18585 128.76 144 1.43417 0.492892
578 16 18628 18612 128.723 108 0.17641 0.493021
579 16 18659 18643 128.714 124 0.0147537 0.493039
2020-03-09 14:37:45.727174 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.492994
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
580 16 18693 18677 128.727 136 1.36927 0.492994
581 15 18726 18711 128.739 136 0.0761666 0.492962
582 16 18756 18740 128.717 116 0.0884404 0.49296
583 16 18787 18771 128.709 124 1.685 0.493085
584 16 18817 18801 128.694 120 1.6518 0.493044
585 15 18854 18839 128.734 152 0.0676967 0.49298
586 16 18883 18867 128.705 112 0.22075 0.492987
587 16 18915 18899 128.704 128 0.467729 0.493062
588 16 18955 18939 128.757 160 0.340188 0.492902
589 16 18990 18974 128.776 140 0.0728635 0.492743
590 16 19029 19013 128.822 156 0.0705404 0.492615
591 16 19060 19044 128.813 124 0.175871 0.492696
592 16 19092 19076 128.812 128 0.218337 0.492612
593 16 19125 19109 128.817 132 0.0757147 0.492674
594 16 19156 19140 128.809 124 1.07111 0.492724
595 16 19195 19179 128.854 156 0.481125 0.492541
596 16 19225 19209 128.84 120 0.337741 0.49251
597 16 19256 19240 128.831 124 0.0444039 0.492556
598 16 19286 19270 128.817 120 1.44237 0.492705
599 16 19318 19302 128.815 128 0.321825 0.492744
2020-03-09 14:38:05.738163 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.49274
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
600 16 19337 19321 128.727 76 0.0763224 0.49274
Total time run: 600.554
Total reads made: 19337
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 128.794
Average IOPS: 32
Stddev IOPS: 2.82388
Max IOPS: 41
Min IOPS: 19
Average Latency(s): 0.493164
Max latency(s): 2.31737
Min latency(s): 0.00758844
Total time run: 600.554
Total reads made: 19337
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 128.794
Average IOPS: 32
Stddev IOPS: 2.82388
Max IOPS: 41
Min IOPS: 19
Average Latency(s): 0.493164
Max latency(s): 2.31737
Min latency(s): 0.00758844
Cleanup, pool with 256 PGs and remove it
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 rados -p testbench cleanup
Removed 20649 objects
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph osd pool delete testbench testbench --yes-i-really-really-mean-it
Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool
Error: non zero exit code: 1: OCI runtime error
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph tell mon.* injectargs --mon-allow-pool-delete=true
mon.f5-422-01: injectargs:mon_allow_pool_delete = 'true'
mon.f5-422-02: injectargs:mon_allow_pool_delete = 'true'
mon.f5-422-03: injectargs:mon_allow_pool_delete = 'true'
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph osd pool delete testbench testbench --yes-i-really-really-mean-it
pool 'testbench' removed
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph tell mon.* injectargs --mon-allow-pool-delete=false
mon.f5-422-01: injectargs:mon_allow_pool_delete = 'false'
mon.f5-422-02: injectargs:mon_allow_pool_delete = 'false'
mon.f5-422-03: injectargs:mon_allow_pool_delete = 'false'
libvirt and Ceph RBD Performance Baseline
Before I start fiddling any tunables, I want to have a baseline also from the RBD consumer side.
Prepare Pool
I created a replica 3 pool with 256 PGs with the Application set to rbd.
Prepare libvirt
Since one can use libvirt with Ceph RBD, I created 2 vols on the libvirt side. Note that I set up libvirt with a pool.
root@hypervisor ~ # virsh vol-list --pool Ceph-HouseNet-libvirt-pool
Name Path
------------------------------------------------------------------------------
20200309-fio-testing-1TiB libvirt-pool/20200309-fio-testing-1TiB
Prepare Test VM
Attached the 500 GiB RBD (20200309-banana) to an existing VM as a virtio block device, formatted it with XFS, mounted it and ran fio.
[root@centos-stream ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 1G 0 disk
└─vda1 252:1 0 1023M 0 part /boot
vdb 252:16 0 30G 0 disk
└─vdb1 252:17 0 30G 0 part
├─VG_cs-LV_root 253:0 0 8G 0 lvm /
├─VG_cs-LV_swap 253:1 0 3.1G 0 lvm [SWAP]
├─VG_cs-LV_home 253:2 0 1G 0 lvm /home
└─VG_cs-LV_var_log 253:3 0 1G 0 lvm /var/log
vdc 252:32 0 1T 0 disk
[root@centos-stream ~]# mkfs.xfs /dev/vdc
meta-data=/dev/vdc isize=512 agcount=4, agsize=67108864 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=268435456, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=131072, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@centos-stream ~]# mount /dev/vdc /mnt/
[root@centos-stream ~]# df -h /mnt/
Filesystem Size Used Avail Use% Mounted on
/dev/vdc 1.0T 7.2G 1017G 1% /mnt
[root@centos-stream ~]# mkdir /mnt/fiotest/
Sequential Read, 4k Blocksize
READ: bw=170MiB/siops: avg=43.6k
fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4k
readtest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
readtest: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=184MiB/s,w=0KiB/s][r=47.1k,w=0 IOPS][eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=3140: Mon Mar 9 18:20:21 2020
read: IOPS=43.6k, BW=170MiB/s (179MB/s)(160GiB/962069msec)
clat (nsec): min=1050, max=590644k, avg=21330.22, stdev=839795.46
lat (nsec): min=1390, max=590645k, avg=21688.15, stdev=839795.50
clat percentiles (nsec):
| 1.00th=[ 1112], 5.00th=[ 1144], 10.00th=[ 1160],
| 20.00th=[ 1176], 30.00th=[ 1224], 40.00th=[ 1240],
| 50.00th=[ 1288], 60.00th=[ 1528], 70.00th=[ 3728],
| 80.00th=[ 3856], 90.00th=[ 4080], 95.00th=[ 4320],
| 99.00th=[ 5472], 99.50th=[ 10432], 99.90th=[ 3424256],
| 99.95th=[15007744], 99.99th=[34865152]
bw ( KiB/s): min= 8192, max=249856, per=99.98%, avg=174358.81, stdev=35849.34, samples=1924
iops : min= 2048, max=62464, avg=43589.69, stdev=8962.33, samples=1924
lat (usec) : 2=64.16%, 4=22.50%, 10=12.81%, 20=0.37%, 50=0.03%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.03%, 10=0.02%, 20=0.03%, 50=0.03%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=3.36%, sys=16.00%, ctx=51046, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=41943040,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=170MiB/s (179MB/s), 170MiB/s-170MiB/s (179MB/s-179MB/s), io=160GiB (172GB), run=962069-962069msec
Disk stats (read/write):
vdc: ios=166533/7, merge=0/0, ticks=9340283/505, in_queue=9257463, util=19.25%
Sequential Read, 4m Blocksize
READ: bw=172MiB/siops: avg=43
fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4m
readtest: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [R(1)][99.9%][r=160MiB/s,w=0KiB/s][r=40,w=0 IOPS][eta 00m:01s]
readtest: (groupid=0, jobs=1): err= 0: pid=3347: Mon Mar 9 18:41:54 2020
read: IOPS=43, BW=172MiB/s (181MB/s)(160GiB/949850msec)
clat (usec): min=407, max=595494, avg=23178.72, stdev=29947.47
lat (usec): min=408, max=595494, avg=23179.67, stdev=29947.41
clat percentiles (usec):
| 1.00th=[ 449], 5.00th=[ 537], 10.00th=[ 685], 20.00th=[ 1614],
| 30.00th=[ 2638], 40.00th=[ 5800], 50.00th=[ 12780], 60.00th=[ 20579],
| 70.00th=[ 31589], 80.00th=[ 41681], 90.00th=[ 58983], 95.00th=[ 74974],
| 99.00th=[114820], 99.50th=[139461], 99.90th=[316670], 99.95th=[383779],
| 99.99th=[463471]
bw ( KiB/s): min=24576, max=294912, per=99.98%, avg=176603.21, stdev=34480.37, samples=1899
iops : min= 6, max= 72, avg=43.09, stdev= 8.41, samples=1899
lat (usec) : 500=3.89%, 750=7.06%, 1000=1.85%
lat (msec) : 2=12.08%, 4=11.78%, 10=9.67%, 20=13.01%, 50=26.31%
lat (msec) : 100=12.57%, 250=1.61%, 500=0.18%, 750=0.01%
cpu : usr=0.05%, sys=7.76%, ctx=41548, majf=0, minf=524
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=40960,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=172MiB/s (181MB/s), 172MiB/s-172MiB/s (181MB/s-181MB/s), io=160GiB (172GB), run=949850-949850msec
Disk stats (read/write):
vdc: ios=166571/0, merge=18/0, ticks=9225180/0, in_queue=9141512, util=19.43%
Sequential Write, 4k Blocksize, 1st run
WRITE: bw=113MiB/siops: avg=28.8k
fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
writetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
writetest: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=111MiB/s][r=0,w=28.4k IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=3500: Mon Mar 9 19:07:39 2020
write: IOPS=28.8k, BW=113MiB/s (118MB/s)(160GiB/1454565msec)
clat (usec): min=2, max=208014, avg=33.00, stdev=612.11
lat (usec): min=2, max=208015, avg=33.38, stdev=612.11
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3],
| 30.00th=[ 3], 40.00th=[ 3], 50.00th=[ 3], 60.00th=[ 3],
| 70.00th=[ 4], 80.00th=[ 6], 90.00th=[ 6], 95.00th=[ 7],
| 99.00th=[ 13], 99.50th=[ 19], 99.90th=[ 7898], 99.95th=[ 7963],
| 99.99th=[11863]
bw ( KiB/s): min= 16, max=922760, per=99.98%, avg=115323.40, stdev=32338.49, samples=2909
iops : min= 4, max=230690, avg=28830.84, stdev=8084.64, samples=2909
lat (usec) : 4=76.54%, 10=22.10%, 20=0.89%, 50=0.08%, 100=0.01%
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.34%, 20=0.03%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=2.58%, sys=12.15%, ctx=158764, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,41943040,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=113MiB/s (118MB/s), 113MiB/s-113MiB/s (118MB/s-118MB/s), io=160GiB (172GB), run=1454565-1454565msec
Disk stats (read/write):
vdc: ios=0/168080, merge=0/7, ticks=0/275742333, in_queue=275658313, util=1.66%
Sequential Write, 4k Blocksize, 2nd run
WRITE: bw=140MiB/siops: avg=35.8k
fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
writetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=140MiB/s][r=0,w=35.7k IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=3855: Mon Mar 9 19:43:04 2020
write: IOPS=35.8k, BW=140MiB/s (147MB/s)(160GiB/1170655msec)
clat (usec): min=2, max=35949, avg=26.17, stdev=512.53
lat (usec): min=2, max=35949, avg=26.55, stdev=512.53
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3],
| 30.00th=[ 3], 40.00th=[ 3], 50.00th=[ 3], 60.00th=[ 3],
| 70.00th=[ 4], 80.00th=[ 6], 90.00th=[ 6], 95.00th=[ 7],
| 99.00th=[ 12], 99.50th=[ 17], 99.90th=[11600], 99.95th=[13435],
| 99.99th=[15270]
bw ( KiB/s): min=42064, max=944664, per=100.00%, avg=143469.78, stdev=27523.62, samples=2338
iops : min=10516, max=236166, avg=35867.42, stdev=6880.91, samples=2338
lat (usec) : 4=70.21%, 10=28.56%, 20=0.91%, 50=0.11%, 100=0.01%
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.09%, 20=0.12%, 50=0.01%
cpu : usr=3.24%, sys=16.08%, ctx=88932, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,41943040,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=140MiB/s (147MB/s), 140MiB/s-140MiB/s (147MB/s-147MB/s), io=160GiB (172GB), run=1170655-1170655msec
Disk stats (read/write):
vdc: ios=0/165401, merge=0/2, ticks=0/221847866, in_queue=221764979, util=2.00%
Sequential Write, 4m Blocksize, 1st run
WRITE: bw=138MiB/siops: avg=34
fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
writetest: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=152MiB/s][r=0,w=38 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=4084: Mon Mar 9 20:06:52 2020
write: IOPS=34, BW=138MiB/s (144MB/s)(160GiB/1191017msec)
clat (usec): min=1400, max=57630, avg=28903.19, stdev=6483.51
lat (usec): min=1522, max=57763, avg=29031.47, stdev=6482.47
clat percentiles (usec):
| 1.00th=[11338], 5.00th=[16450], 10.00th=[17695], 20.00th=[25035],
| 30.00th=[27132], 40.00th=[27657], 50.00th=[30016], 60.00th=[31589],
| 70.00th=[33424], 80.00th=[33817], 90.00th=[35390], 95.00th=[35914],
| 99.00th=[39584], 99.50th=[39584], 99.90th=[43779], 99.95th=[46400],
| 99.99th=[51119]
bw ( KiB/s): min=114459, max=1294336, per=100.00%, avg=141025.75, stdev=34335.39, samples=2379
iops : min= 27, max= 316, avg=34.39, stdev= 8.39, samples=2379
lat (msec) : 2=0.19%, 4=0.72%, 10=0.08%, 20=12.65%, 50=86.35%
lat (msec) : 100=0.01%
cpu : usr=0.36%, sys=7.24%, ctx=100250, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,40960,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=138MiB/s (144MB/s), 138MiB/s-138MiB/s (144MB/s-144MB/s), io=160GiB (172GB), run=1191017-1191017msec
Disk stats (read/write):
vdc: ios=0/164711, merge=0/2, ticks=0/225940838, in_queue=225858476, util=1.88%
Sequential Write, 4m Blocksize, 2nd run
WRITE: bw=135MiB/siops: avg=33
fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
writetest: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=132MiB/s][r=0,w=33 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=4308: Mon Mar 9 20:32:40 2020
write: IOPS=33, BW=135MiB/s (142MB/s)(160GiB/1213647msec)
clat (usec): min=1404, max=109269, avg=29455.52, stdev=5986.18
lat (usec): min=1547, max=109383, avg=29584.63, stdev=5985.02
clat percentiles (usec):
| 1.00th=[13435], 5.00th=[17695], 10.00th=[23987], 20.00th=[26870],
| 30.00th=[27132], 40.00th=[27132], 50.00th=[29754], 60.00th=[31589],
| 70.00th=[33424], 80.00th=[35390], 90.00th=[35390], 95.00th=[35914],
| 99.00th=[39584], 99.50th=[39584], 99.90th=[44827], 99.95th=[68682],
| 99.99th=[91751]
bw ( KiB/s): min=49152, max=1449984, per=100.00%, avg=138391.61, stdev=38375.99, samples=2424
iops : min= 12, max= 354, avg=33.75, stdev= 9.37, samples=2424
lat (msec) : 2=0.47%, 4=0.42%, 10=0.03%, 20=5.69%, 50=93.32%
lat (msec) : 100=0.06%, 250=0.01%
cpu : usr=0.35%, sys=7.21%, ctx=119939, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,40960,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=135MiB/s (142MB/s), 135MiB/s-135MiB/s (142MB/s-142MB/s), io=160GiB (172GB), run=1213647-1213647msec
Disk stats (read/write):
vdc: ios=0/164742, merge=0/2, ticks=0/230335859, in_queue=230253272, util=1.88%
Sequential, 80% Read, 20% Write, 4k Blocksize, 1st run
READ: bw=133MiB/siops: avg=34.0kWRITE: bw=33.2MiB/siops: avg=8503
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/
--bs=4k
read80write20test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
read80write20test: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [M(1)][100.0%][r=144MiB/s,w=36.4MiB/s][r=36.9k,w=9325 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=5025: Mon Mar 9 21:39:08 2020
read: IOPS=34.0k, BW=133MiB/s (139MB/s)(128GiB/986447msec)
clat (nsec): min=1030, max=583009k, avg=26272.54, stdev=1289481.27
lat (nsec): min=1370, max=583009k, avg=26627.44, stdev=1289481.31
clat percentiles (nsec):
| 1.00th=[ 1128], 5.00th=[ 1160], 10.00th=[ 1176],
| 20.00th=[ 1208], 30.00th=[ 1240], 40.00th=[ 1288],
| 50.00th=[ 1368], 60.00th=[ 1576], 70.00th=[ 3760],
| 80.00th=[ 3952], 90.00th=[ 4192], 95.00th=[ 4448],
| 99.00th=[ 5984], 99.50th=[ 11072], 99.90th=[ 2441216],
| 99.95th=[14876672], 99.99th=[49020928]
bw ( KiB/s): min= 4096, max=245760, per=100.00%, avg=136100.29, stdev=55704.13, samples=1971
iops : min= 1024, max=61440, avg=34025.06, stdev=13926.03, samples=1971
write: IOPS=8503, BW=33.2MiB/s (34.8MB/s)(31.0GiB/986447msec)
clat (nsec): min=1790, max=12100k, avg=4191.77, stdev=15119.81
lat (usec): min=2, max=12100, avg= 4.57, stdev=15.13
clat percentiles (nsec):
| 1.00th=[ 2008], 5.00th=[ 2320], 10.00th=[ 2448], 20.00th=[ 2544],
| 30.00th=[ 2608], 40.00th=[ 2768], 50.00th=[ 2992], 60.00th=[ 3696],
| 70.00th=[ 5472], 80.00th=[ 5920], 90.00th=[ 6560], 95.00th=[ 7584],
| 99.00th=[12736], 99.50th=[16512], 99.90th=[25728], 99.95th=[30080],
| 99.99th=[41216]
bw ( KiB/s): min= 880, max=63016, per=100.00%, avg=34021.30, stdev=13952.04, samples=1971
iops : min= 220, max=15754, avg=8505.31, stdev=3488.01, samples=1971
lat (usec) : 2=52.62%, 4=26.23%, 10=20.35%, 20=0.59%, 50=0.10%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=0.02%, 20=0.02%, 50=0.02%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=3.57%, sys=16.91%, ctx=40056, majf=0, minf=15
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=33555281,8387759,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=133MiB/s (139MB/s), 133MiB/s-133MiB/s (139MB/s-139MB/s), io=128GiB (137GB), run=986447-986447msec
WRITE: bw=33.2MiB/s (34.8MB/s), 33.2MiB/s-33.2MiB/s (34.8MB/s-34.8MB/s), io=31.0GiB (34.4GB), run=986447-986447msec
Disk stats (read/write):
vdc: ios=132917/33058, merge=0/11, ticks=9237406/9508567, in_queue=18662956, util=15.87%
Sequential, 80% Read, 20% Write, 4k Blocksize, 2nd run
READ: bw=131MiB/siops: avg=33.5kWRITE: bw=32.7MiB/siops: avg=8372
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k
read80write20test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=89.0MiB/s,w=22.6MiB/s][r=22.8k,w=5798 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=5195: Mon Mar 9 21:58:36 2020
read: IOPS=33.5k, BW=131MiB/s (137MB/s)(128GiB/1001856msec)
clat (nsec): min=1040, max=504078k, avg=26745.04, stdev=1301910.00
lat (nsec): min=1380, max=504078k, avg=27099.54, stdev=1301913.08
clat percentiles (nsec):
| 1.00th=[ 1112], 5.00th=[ 1144], 10.00th=[ 1176],
| 20.00th=[ 1208], 30.00th=[ 1240], 40.00th=[ 1272],
| 50.00th=[ 1336], 60.00th=[ 1432], 70.00th=[ 1720],
| 80.00th=[ 3888], 90.00th=[ 4192], 95.00th=[ 4448],
| 99.00th=[ 5984], 99.50th=[ 10176], 99.90th=[ 2637824],
| 99.95th=[15269888], 99.99th=[51118080]
bw ( KiB/s): min= 4096, max=245760, per=100.00%, avg=134179.48, stdev=55187.04, samples=2000
iops : min= 1024, max=61440, avg=33544.87, stdev=13796.76, samples=2000
write: IOPS=8372, BW=32.7MiB/s (34.3MB/s)(31.0GiB/1001856msec)
clat (nsec): min=1790, max=12108k, avg=3939.87, stdev=8882.46
lat (usec): min=2, max=12108, avg= 4.33, stdev= 9.56
clat percentiles (nsec):
| 1.00th=[ 1976], 5.00th=[ 2320], 10.00th=[ 2416], 20.00th=[ 2512],
| 30.00th=[ 2576], 40.00th=[ 2672], 50.00th=[ 2832], 60.00th=[ 3216],
| 70.00th=[ 4128], 80.00th=[ 5792], 90.00th=[ 6432], 95.00th=[ 7520],
| 99.00th=[12608], 99.50th=[16512], 99.90th=[24960], 99.95th=[29312],
| 99.99th=[39680]
bw ( KiB/s): min= 992, max=63360, per=100.00%, avg=33540.82, stdev=13817.76, samples=2000
iops : min= 248, max=15840, avg=8385.20, stdev=3454.44, samples=2000
lat (usec) : 2=58.41%, 4=23.03%, 10=17.80%, 20=0.56%, 50=0.09%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=0.02%, 20=0.02%, 50=0.02%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=3.53%, sys=15.99%, ctx=39306, majf=0, minf=15
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=33555281,8387759,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=128GiB (137GB), run=1001856-1001856msec
WRITE: bw=32.7MiB/s (34.3MB/s), 32.7MiB/s-32.7MiB/s (34.3MB/s-34.3MB/s), io=31.0GiB (34.4GB), run=1001856-1001856msec
Disk stats (read/write):
vdc: ios=132838/33023, merge=0/12, ticks=9344607/9568626, in_queue=18830124, util=15.68%
Sequential, 80% Read, 20% Write, 4m Blocksize, 1st run
READ: bw=129MiB/siops: avg=32WRITE: bw=32.7MiB/siops: avg=8
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
read80write20test: (g=0): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=176MiB/s,w=56.1MiB/s][r=44,w=14 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=5355: Mon Mar 9 22:16:49 2020
read: IOPS=32, BW=129MiB/s (135MB/s)(128GiB/1012922msec)
clat (usec): min=405, max=646248, avg=30349.54, stdev=50502.76
lat (usec): min=405, max=646249, avg=30350.45, stdev=50502.71
clat percentiles (usec):
| 1.00th=[ 449], 5.00th=[ 562], 10.00th=[ 734], 20.00th=[ 1565],
| 30.00th=[ 2311], 40.00th=[ 4621], 50.00th=[ 12518], 60.00th=[ 22152],
| 70.00th=[ 34341], 80.00th=[ 47973], 90.00th=[ 73925], 95.00th=[106431],
| 99.00th=[270533], 99.50th=[350225], 99.90th=[450888], 99.95th=[484443],
| 99.99th=[557843]
bw ( KiB/s): min= 8175, max=245760, per=100.00%, avg=132405.29, stdev=56654.42, samples=2021
iops : min= 1, max= 60, avg=32.30, stdev=13.83, samples=2021
write: IOPS=8, BW=32.7MiB/s (34.3MB/s)(32.3GiB/1012922msec)
clat (usec): min=838, max=13117, avg=2202.18, stdev=882.87
lat (usec): min=883, max=13231, avg=2292.22, stdev=886.68
clat percentiles (usec):
| 1.00th=[ 1106], 5.00th=[ 1303], 10.00th=[ 1352], 20.00th=[ 1434],
| 30.00th=[ 1500], 40.00th=[ 1647], 50.00th=[ 1893], 60.00th=[ 2245],
| 70.00th=[ 2606], 80.00th=[ 2999], 90.00th=[ 3556], 95.00th=[ 3884],
| 99.00th=[ 4490], 99.50th=[ 4686], 99.90th=[ 5145], 99.95th=[ 5604],
| 99.99th=[13173]
bw ( KiB/s): min= 8078, max=139264, per=100.00%, avg=36216.64, stdev=22038.23, samples=1870
iops : min= 1, max= 34, avg= 8.82, stdev= 5.38, samples=1870
lat (usec) : 500=2.16%, 750=5.95%, 1000=1.86%
lat (msec) : 2=22.01%, 4=18.53%, 10=7.24%, 20=8.78%, 50=18.39%
lat (msec) : 100=10.62%, 250=3.49%, 500=0.94%, 750=0.03%
cpu : usr=0.12%, sys=7.75%, ctx=32560, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=32688,8272,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=128GiB (137GB), run=1012922-1012922msec
WRITE: bw=32.7MiB/s (34.3MB/s), 32.7MiB/s-32.7MiB/s (34.3MB/s-34.3MB/s), io=32.3GiB (34.7GB), run=1012922-1012922msec
Disk stats (read/write):
vdc: ios=132738/33417, merge=15/0, ticks=9532728/11600214, in_queue=21049821, util=15.38%
Sequential, 80% Read, 20% Write, 4m Blocksize, 2nd run
READ: bw=128MiB/siops: avg=32WRITE: bw=32.5MiB/siops: avg=8
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
read80write20test: (g=0): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=144MiB/s,w=44.0MiB/s][r=36,w=11 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=5539: Mon Mar 9 22:36:54 2020
read: IOPS=32, BW=128MiB/s (135MB/s)(128GiB/1017899msec)
clat (usec): min=415, max=722288, avg=30504.65, stdev=51268.90
lat (usec): min=416, max=722289, avg=30505.49, stdev=51268.86
clat percentiles (usec):
| 1.00th=[ 453], 5.00th=[ 553], 10.00th=[ 758], 20.00th=[ 1582],
| 30.00th=[ 2311], 40.00th=[ 4424], 50.00th=[ 12649], 60.00th=[ 22414],
| 70.00th=[ 34341], 80.00th=[ 47973], 90.00th=[ 74974], 95.00th=[106431],
| 99.00th=[270533], 99.50th=[350225], 99.90th=[480248], 99.95th=[517997],
| 99.99th=[616563]
bw ( KiB/s): min= 8192, max=245760, per=100.00%, avg=131905.37, stdev=56989.64, samples=2029
iops : min= 2, max= 60, avg=32.18, stdev=13.92, samples=2029
write: IOPS=8, BW=32.5MiB/s (34.1MB/s)(32.3GiB/1017899msec)
clat (usec): min=827, max=20250, avg=2193.92, stdev=953.01
lat (usec): min=869, max=20452, avg=2282.39, stdev=956.96
clat percentiles (usec):
| 1.00th=[ 898], 5.00th=[ 1090], 10.00th=[ 1336], 20.00th=[ 1418],
| 30.00th=[ 1483], 40.00th=[ 1647], 50.00th=[ 1893], 60.00th=[ 2245],
| 70.00th=[ 2606], 80.00th=[ 2999], 90.00th=[ 3556], 95.00th=[ 3949],
| 99.00th=[ 4555], 99.50th=[ 4686], 99.90th=[ 6128], 99.95th=[10290],
| 99.99th=[20317]
bw ( KiB/s): min= 8143, max=155648, per=100.00%, avg=36108.79, stdev=22228.68, samples=1876
iops : min= 1, max= 38, avg= 8.78, stdev= 5.43, samples=1876
lat (usec) : 500=2.29%, 750=5.60%, 1000=2.45%
lat (msec) : 2=21.33%, 4=18.91%, 10=7.06%, 20=8.78%, 50=18.43%
lat (msec) : 100=10.68%, 250=3.52%, 500=0.91%, 750=0.06%
cpu : usr=0.10%, sys=7.70%, ctx=32782, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=32688,8272,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=128MiB/s (135MB/s), 128MiB/s-128MiB/s (135MB/s-135MB/s), io=128GiB (137GB), run=1017899-1017899msec
WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=32.3GiB (34.7GB), run=1017899-1017899msec
Disk stats (read/write):
vdc: ios=132648/33381, merge=15/0, ticks=9582613/11738047, in_queue=21237573, util=15.26%
sequential write, 2300 bytes, fsync
Normally this test is not part of my baseline but a colleague inquired, so pasting here.
WRITE: bw=232KiB/siops: avg=103
fio --rw=write --ioengine=sync --fdatasync=1 --size=22m --bs=2300 --name=etcdtest --directory=/mnt/fiotest/
[root@centos-stream ~]# fio --rw=write --ioengine=sync --fdatasync=1 --size=22m --bs=2300 --name=etcdtest --directory=/mnt/fiotest/
etcdtest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.7
Starting 1 process
etcdtest: Laying out IO file (1 file / 22MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=242KiB/s][r=0,w=108 IOPS][eta 00m:00s]
etcdtest: (groupid=0, jobs=1): err= 0: pid=24595: Wed Mar 11 09:56:35 2020
write: IOPS=103, BW=232KiB/s (237kB/s)(21.0MiB/97165msec)
clat (usec): min=6, max=175, avg=15.50, stdev= 6.16
lat (usec): min=7, max=191, avg=16.18, stdev= 6.22
clat percentiles (nsec):
| 1.00th=[ 8256], 5.00th=[ 8896], 10.00th=[ 9152], 20.00th=[ 9408],
| 30.00th=[ 9792], 40.00th=[10304], 50.00th=[17536], 60.00th=[19072],
| 70.00th=[19840], 80.00th=[20352], 90.00th=[21376], 95.00th=[22656],
| 99.00th=[31360], 99.50th=[33536], 99.90th=[43264], 99.95th=[47360],
| 99.99th=[93696]
bw ( KiB/s): min= 53, max= 260, per=100.00%, avg=231.48, stdev=21.29, samples=194
iops : min= 24, max= 116, avg=103.21, stdev= 9.46, samples=194
lat (usec) : 10=35.28%, 20=38.59%, 50=26.08%, 100=0.04%, 250=0.01%
fsync/fdatasync/sync_file_range:
sync (msec): min=3, max=204, avg= 9.67, stdev= 6.68
sync percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6],
| 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 11], 60.00th=[ 11],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 13], 95.00th=[ 16],
| 99.00th=[ 36], 99.50th=[ 40], 99.90th=[ 66], 99.95th=[ 94],
| 99.99th=[ 203]
cpu : usr=0.06%, sys=0.45%, ctx=35706, majf=0, minf=12
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,10029,0,0 short=10029,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=232KiB/s (237kB/s), 232KiB/s-232KiB/s (237kB/s-237kB/s), io=21.0MiB (23.1MB), run=97165-97165msec
Disk stats (read/write):
vdc: ios=0/25676, merge=0/8, ticks=0/97623, in_queue=89570, util=13.10%
sequential write, 2300 bytes, NO fsync
Since it seems sily to me to call fsync after 2300 bytes, same test without that option to have a comparison.
WRITE: bw=108MiB/siops: avg=49.4k
fio --rw=write --ioengine=sync --size=160g --bs=2300 --name=etcdtest --directory=/mnt/fiotest/
[root@centos-stream ~]# fio --rw=write --ioengine=sync --size=160g --bs=2300 --name=etcdtest --directory=/mnt/fiotest/
etcdtest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.7
Starting 1 process
etcdtest: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=113MiB/s][r=0,w=51.4k IOPS][eta 00m:00s]
etcdtest: (groupid=0, jobs=1): err= 0: pid=24660: Wed Mar 11 10:24:51 2020
write: IOPS=49.4k, BW=108MiB/s (114MB/s)(160GiB/1512486msec)
clat (nsec): min=1560, max=207919k, avg=18588.48, stdev=418983.46
lat (nsec): min=1910, max=207919k, avg=18957.71, stdev=418983.68
clat percentiles (nsec):
| 1.00th=[ 1624], 5.00th=[ 1640], 10.00th=[ 1656],
| 20.00th=[ 1672], 30.00th=[ 1704], 40.00th=[ 1768],
| 50.00th=[ 2800], 60.00th=[ 2928], 70.00th=[ 3216],
| 80.00th=[ 5088], 90.00th=[ 5920], 95.00th=[ 6496],
| 99.00th=[ 10176], 99.50th=[ 14784], 99.90th=[ 6979584],
| 99.95th=[ 7766016], 99.99th=[11862016]
bw ( KiB/s): min= 247, max=405042, per=99.97%, avg=110893.31, stdev=28246.21, samples=3024
iops : min= 110, max=180332, avg=49371.82, stdev=12575.70, samples=3024
lat (usec) : 2=42.13%, 4=35.25%, 10=21.60%, 20=0.75%, 50=0.07%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.15%, 20=0.04%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=4.16%, sys=19.94%, ctx=143227, majf=1, minf=19
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,74695083,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=108MiB/s (114MB/s), 108MiB/s-108MiB/s (114MB/s-114MB/s), io=160GiB (172GB), run=1512486-1512486msec
Disk stats (read/write):
vdc: ios=0/169867, merge=0/8, ticks=0/286486813, in_queue=286401832, util=1.66%