f1: (g=0): rw=write, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 ... f1: (g=0): rw=write, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 f2: (g=1): rw=randwrite, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 ... f2: (g=1): rw=randwrite, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 f3: (g=2): rw=read, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 ... f3: (g=2): rw=read, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 f4: (g=3): rw=randread, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 ... f4: (g=3): rw=randread, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 Starting 16 processes f1: Laying out IO file(s) (1 file(s) / 1024MiB) f1: Laying out IO file(s) (1 file(s) / 1024MiB) f1: Laying out IO file(s) (1 file(s) / 1024MiB) f1: Laying out IO file(s) (1 file(s) / 1024MiB) f2: Laying out IO file(s) (1 file(s) / 1024MiB) f2: Laying out IO file(s) (1 file(s) / 1024MiB) f2: Laying out IO file(s) (1 file(s) / 1024MiB) f2: Laying out IO file(s) (1 file(s) / 1024MiB) f3: Laying out IO file(s) (1 file(s) / 1024MiB) f3: Laying out IO file(s) (1 file(s) / 1024MiB) f3: Laying out IO file(s) (1 file(s) / 1024MiB) f3: Laying out IO file(s) (1 file(s) / 1024MiB) f4: Laying out IO file(s) (1 file(s) / 1024MiB) f4: Laying out IO file(s) (1 file(s) / 1024MiB) f4: Laying out IO file(s) (1 file(s) / 1024MiB) f4: Laying out IO file(s) (1 file(s) / 1024MiB) f1: (groupid=0, jobs=1): err= 0: pid=6452 write: io=1,024MiB, bw=16,042KiB/s, iops=1,002, runt= 65364msec clat (usec): min=52, max=3,240K, avg=993.79, stdev=35805.77 bw (KiB/s) : min= 0, max=136096, per=41.12%, avg=23595.52, stdev=36428.88 cpu : usr=0.40%, sys=9.50%, ctx=3617, majf=0, minf=232 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=55.81%, 250=43.81%, 500=0.12%, 750=0.01%, 1000=0.02% lat (msec): 2=0.05%, 4=0.03%, 10=0.04%, 20=0.01%, 50=0.01% lat (msec): 100=0.01%, 250=0.01%, 500=0.03%, 750=0.04%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.01% f1: (groupid=0, jobs=1): err= 0: pid=6453 write: io=1,024MiB, bw=14,346KiB/s, iops=896, runt= 73090msec clat (usec): min=48, max=6,845K, avg=1111.17, stdev=44003.99 bw (KiB/s) : min= 0, max=151648, per=39.82%, avg=22846.00, stdev=37354.13 cpu : usr=0.38%, sys=8.29%, ctx=3529, majf=0, minf=230 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 50=0.06%, 100=64.88%, 250=34.68%, 500=0.10%, 750=0.02% lat (usec): 1000=0.01% lat (msec): 2=0.03%, 4=0.03%, 10=0.05%, 20=0.02%, 50=0.01% lat (msec): 100=0.01%, 250=0.01%, 500=0.04%, 750=0.04%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.01% f1: (groupid=0, jobs=1): err= 0: pid=6454 write: io=1,024MiB, bw=14,344KiB/s, iops=896, runt= 73102msec clat (usec): min=48, max=5,085K, avg=1111.62, stdev=41712.34 bw (KiB/s) : min= 0, max=135616, per=39.22%, avg=22501.57, stdev=34013.63 cpu : usr=0.38%, sys=8.49%, ctx=4686, majf=0, minf=251 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 50=0.05%, 100=56.17%, 250=43.30%, 500=0.11%, 750=0.04% lat (usec): 1000=0.02% lat (msec): 2=0.07%, 4=0.06%, 10=0.05%, 20=0.01%, 50=0.01% lat (msec): 250=0.01%, 500=0.04%, 750=0.04%, 1000=0.01%, 2000=0.01% lat (msec): >=2000=0.01% f1: (groupid=0, jobs=1): err= 0: pid=6455 write: io=1,024MiB, bw=14,371KiB/s, iops=898, runt= 72963msec clat (usec): min=51, max=5,755K, avg=1109.85, stdev=40850.77 bw (KiB/s) : min= 0, max=145600, per=41.23%, avg=23655.96, stdev=37437.12 cpu : usr=0.38%, sys=8.45%, ctx=2860, majf=0, minf=234 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=58.69%, 250=40.95%, 500=0.07%, 750=0.01%, 1000=0.01% lat (msec): 2=0.03%, 4=0.04%, 10=0.04%, 20=0.02%, 50=0.01% lat (msec): 100=0.01%, 250=0.01%, 500=0.05%, 750=0.04%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.01% f2: (groupid=1, jobs=1): err= 0: pid=6674 write: io=1,024MiB, bw=878KiB/s, iops=54, runt=1193413msec clat (usec): min=48, max=22,294K, avg=18206.03, stdev=492359.75 bw (KiB/s) : min= 0, max=159616, per=173.84%, avg=6108.63, stdev=28463.64 cpu : usr=0.02%, sys=0.48%, ctx=3309, majf=0, minf=1048 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 50=0.05%, 100=79.40%, 250=20.12%, 500=0.04%, 750=0.02% lat (usec): 1000=0.04% lat (msec): 2=0.08%, 4=0.02%, 10=0.03%, 20=0.02%, 50=0.01% lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01% lat (msec): >=2000=0.16% f2: (groupid=1, jobs=1): err= 0: pid=6675 write: io=1,024MiB, bw=888KiB/s, iops=55, runt=1179857msec clat (usec): min=44, max=22,294K, avg=17999.00, stdev=488687.67 bw (KiB/s) : min= 0, max=163424, per=182.21%, avg=6402.74, stdev=29368.78 cpu : usr=0.02%, sys=0.48%, ctx=2719, majf=0, minf=1011 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 50=0.63%, 100=84.00%, 250=14.85%, 500=0.12%, 750=0.02% lat (usec): 1000=0.04% lat (msec): 2=0.07%, 4=0.03%, 10=0.03%, 20=0.02%, 50=0.02% lat (msec): 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01%, >=2000=0.15% f2: (groupid=1, jobs=1): err= 0: pid=6676 write: io=1,024MiB, bw=945KiB/s, iops=59, runt=1108784msec clat (usec): min=47, max=22,280K, avg=16914.69, stdev=469277.43 bw (KiB/s) : min= 0, max=162112, per=195.05%, avg=6853.94, stdev=29699.20 cpu : usr=0.02%, sys=0.52%, ctx=3686, majf=0, minf=984 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 50=0.33%, 100=80.91%, 250=18.43%, 500=0.04%, 750=0.01% lat (usec): 1000=0.01% lat (msec): 2=0.04%, 4=0.02%, 10=0.03%, 20=0.01%, 50=0.01% lat (msec): 100=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01% lat (msec): >=2000=0.15% f2: (groupid=1, jobs=1): err= 0: pid=6677 write: io=1,024MiB, bw=888KiB/s, iops=55, runt=1179912msec clat (usec): min=46, max=21,641K, avg=18000.17, stdev=492332.25 bw (KiB/s) : min= 0, max=166592, per=188.87%, avg=6636.88, stdev=30179.97 cpu : usr=0.02%, sys=0.48%, ctx=2983, majf=0, minf=1044 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 50=1.32%, 100=84.79%, 250=13.39%, 500=0.05%, 750=0.03% lat (usec): 1000=0.05% lat (msec): 2=0.11%, 4=0.02%, 10=0.03%, 20=0.02%, 50=0.02% lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01% lat (msec): >=2000=0.14% f3: (groupid=2, jobs=1): err= 0: pid=8851 read : io=1,024MiB, bw=17,574KiB/s, iops=1,098, runt= 59663msec clat (usec): min=7, max=772K, avg=907.60, stdev=10183.77 bw (KiB/s) : min= 218, max=29460, per=26.93%, avg=18913.00, stdev=10644.33 cpu : usr=0.37%, sys=1.68%, ctx=4129, majf=0, minf=41 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 10=49.81%, 20=43.50%, 50=0.36%, 100=0.04%, 250=0.02% lat (usec): 500=0.01%, 750=0.01%, 1000=0.01% lat (msec): 2=0.13%, 4=0.43%, 10=2.65%, 20=2.74%, 50=0.18% lat (msec): 100=0.02%, 250=0.05%, 500=0.04%, 750=0.01%, 1000=0.01% f3: (groupid=2, jobs=1): err= 0: pid=8852 read : io=1,024MiB, bw=17,582KiB/s, iops=1,098, runt= 59637msec clat (usec): min=7, max=726K, avg=906.86, stdev=9920.25 bw (KiB/s) : min= 217, max=28500, per=26.57%, avg=18654.04, stdev=10703.89 cpu : usr=0.41%, sys=1.69%, ctx=4120, majf=0, minf=41 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 10=50.26%, 20=43.09%, 50=0.33%, 100=0.05%, 250=0.02% lat (usec): 500=0.01%, 750=0.01%, 1000=0.01% lat (msec): 2=0.14%, 4=0.38%, 10=2.68%, 20=2.74%, 50=0.16% lat (msec): 100=0.02%, 250=0.07%, 500=0.03%, 750=0.01% f3: (groupid=2, jobs=1): err= 0: pid=8853 read : io=1,024MiB, bw=17,554KiB/s, iops=1,097, runt= 59731msec clat (usec): min=7, max=814K, avg=908.54, stdev=10282.60 bw (KiB/s) : min= 196, max=31045, per=26.68%, avg=18733.18, stdev=10578.19 cpu : usr=0.34%, sys=1.66%, ctx=4101, majf=0, minf=41 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 10=49.38%, 20=43.98%, 50=0.33%, 100=0.05%, 250=0.03% lat (usec): 500=0.01%, 750=0.01%, 1000=0.01% lat (msec): 2=0.14%, 4=0.39%, 10=2.64%, 20=2.76%, 50=0.18% lat (msec): 100=0.02%, 250=0.05%, 500=0.04%, 750=0.01%, 1000=0.01% f3: (groupid=2, jobs=1): err= 0: pid=8854 read : io=1,024MiB, bw=17,575KiB/s, iops=1,098, runt= 59662msec clat (usec): min=7, max=844K, avg=907.59, stdev=9924.62 bw (KiB/s) : min= 188, max=29577, per=26.86%, avg=18860.74, stdev=10599.74 cpu : usr=0.42%, sys=1.65%, ctx=4128, majf=0, minf=41 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 10=49.88%, 20=43.45%, 50=0.33%, 100=0.05%, 250=0.02% lat (usec): 500=0.01%, 1000=0.01% lat (msec): 2=0.13%, 4=0.42%, 10=2.60%, 20=2.79%, 50=0.16% lat (msec): 100=0.03%, 250=0.06%, 500=0.03%, 750=0.01%, 1000=0.01% f4: (groupid=3, jobs=1): err= 0: pid=9025 read : io=1,024MiB, bw=922KiB/s, iops=57, runt=1136586msec clat (msec): min=2, max=197, avg=17.33, stdev=10.40 bw (KiB/s) : min= 559, max= 1721, per=25.06%, avg=923.10, stdev=117.71 cpu : usr=0.03%, sys=0.39%, ctx=262413, majf=0, minf=710 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (msec): 4=0.83%, 10=25.51%, 20=41.48%, 50=31.17%, 100=1.00% lat (msec): 250=0.01% f4: (groupid=3, jobs=1): err= 0: pid=9026 read : io=1,024MiB, bw=958KiB/s, iops=59, runt=1094511msec clat (msec): min=2, max=146, avg=16.69, stdev=10.12 bw (KiB/s) : min= 531, max= 1752, per=26.02%, avg=958.55, stdev=110.23 cpu : usr=0.03%, sys=0.40%, ctx=262407, majf=0, minf=904 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (msec): 4=0.88%, 10=27.62%, 20=41.83%, 50=28.76%, 100=0.91% lat (msec): 250=0.01% f4: (groupid=3, jobs=1): err= 0: pid=9027 read : io=1,024MiB, bw=952KiB/s, iops=59, runt=1101331msec clat (msec): min=2, max=132, avg=16.80, stdev=10.17 bw (KiB/s) : min= 611, max= 1692, per=25.86%, avg=952.69, stdev=111.95 cpu : usr=0.04%, sys=0.40%, ctx=262390, majf=0, minf=909 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (msec): 4=0.88%, 10=27.19%, 20=42.11%, 50=28.86%, 100=0.94% lat (msec): 250=0.01% f4: (groupid=3, jobs=1): err= 0: pid=9028 read : io=1,024MiB, bw=921KiB/s, iops=57, runt=1138397msec clat (msec): min=2, max=129, avg=17.36, stdev=10.42 bw (KiB/s) : min= 439, max= 2144, per=25.02%, avg=921.66, stdev=129.02 cpu : usr=0.03%, sys=0.38%, ctx=262402, majf=0, minf=940 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (msec): 4=0.97%, 10=25.31%, 20=41.44%, 50=31.27%, 100=1.01% lat (msec): 250=0.01% Run status group 0 (all jobs): WRITE: io=4,096MiB, aggrb=57,376KiB/s, minb=14,344KiB/s, maxb=16,042KiB/s, mint=65364msec, maxt=73102msec Run status group 1 (all jobs): WRITE: io=4,096MiB, aggrb=3,514KiB/s, minb=878KiB/s, maxb=945KiB/s, mint=1108784msec, maxt=1193413msec Run status group 2 (all jobs): READ: io=4,096MiB, aggrb=70,220KiB/s, minb=17,555KiB/s, maxb=17,582KiB/s, mint=59637msec, maxt=59731msec Run status group 3 (all jobs): READ: io=4,096MiB, aggrb=3,684KiB/s, minb=921KiB/s, maxb=958KiB/s, mint=1094511msec, maxt=1138397msec Disk stats (read/write): sda: ios=1066065/385820, merge=3/3880955, ticks=4958500/384709896, in_queue=389718038, util=99.29% 4.95user 103.15system 54:35.24elapsed 3%CPU (0avgtext+0avgdata 0maxresident)k 16777256inputs+33599320outputs (38major+24316minor)pagefaults 0swaps