f1: (g=0): rw=write, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 ... f1: (g=0): rw=write, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 f2: (g=1): rw=randwrite, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 ... f2: (g=1): rw=randwrite, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 f3: (g=2): rw=read, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 ... f3: (g=2): rw=read, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 f4: (g=3): rw=randread, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 ... f4: (g=3): rw=randread, bs=16K-16K/16K-16K, ioengine=sync, iodepth=4 Starting 16 processes f1: Laying out IO file(s) (1 file(s) / 1024MiB) f1: Laying out IO file(s) (1 file(s) / 1024MiB) f1: Laying out IO file(s) (1 file(s) / 1024MiB) f1: Laying out IO file(s) (1 file(s) / 1024MiB) f2: Laying out IO file(s) (1 file(s) / 1024MiB) f2: Laying out IO file(s) (1 file(s) / 1024MiB) f2: Laying out IO file(s) (1 file(s) / 1024MiB) f2: Laying out IO file(s) (1 file(s) / 1024MiB) f3: Laying out IO file(s) (1 file(s) / 1024MiB) f3: Laying out IO file(s) (1 file(s) / 1024MiB) f3: Laying out IO file(s) (1 file(s) / 1024MiB) f3: Laying out IO file(s) (1 file(s) / 1024MiB) f4: Laying out IO file(s) (1 file(s) / 1024MiB) f4: Laying out IO file(s) (1 file(s) / 1024MiB) f4: Laying out IO file(s) (1 file(s) / 1024MiB) f4: Laying out IO file(s) (1 file(s) / 1024MiB) f1: (groupid=0, jobs=1): err= 0: pid=6465 write: io=1,024MiB, bw=15,915KiB/s, iops=994, runt= 65886msec clat (usec): min=51, max=3,333K, avg=1001.55, stdev=36661.91 bw (KiB/s) : min= 0, max=148192, per=45.22%, avg=25614.75, stdev=30711.43 cpu : usr=0.39%, sys=9.60%, ctx=4196, majf=0, minf=219 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=62.81%, 250=36.74%, 500=0.10%, 750=0.02%, 1000=0.01% lat (msec): 2=0.05%, 4=0.06%, 10=0.06%, 20=0.01%, 50=0.01% lat (msec): 100=0.01%, 250=0.02%, 500=0.05%, 750=0.02%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.01% f1: (groupid=0, jobs=1): err= 0: pid=6466 write: io=1,024MiB, bw=14,162KiB/s, iops=885, runt= 74041msec clat (usec): min=51, max=5,334K, avg=1125.56, stdev=40650.38 bw (KiB/s) : min= 0, max=98144, per=35.66%, avg=20201.07, stdev=21420.44 cpu : usr=0.34%, sys=8.26%, ctx=3466, majf=0, minf=250 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=66.23%, 250=32.78%, 500=0.22%, 750=0.07%, 1000=0.06% lat (msec): 2=0.16%, 4=0.14%, 10=0.14%, 20=0.04%, 50=0.01% lat (msec): 100=0.01%, 250=0.02%, 500=0.05%, 750=0.04%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.01% f1: (groupid=0, jobs=1): err= 0: pid=6467 write: io=1,024MiB, bw=14,190KiB/s, iops=886, runt= 73892msec clat (usec): min=51, max=5,374K, avg=1124.04, stdev=40481.85 bw (KiB/s) : min= 0, max=134400, per=38.38%, avg=21740.92, stdev=25601.14 cpu : usr=0.39%, sys=8.44%, ctx=4038, majf=0, minf=237 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=62.24%, 250=36.94%, 500=0.19%, 750=0.07%, 1000=0.04% lat (msec): 2=0.11%, 4=0.12%, 10=0.11%, 20=0.02%, 50=0.01% lat (msec): 100=0.01%, 250=0.02%, 500=0.06%, 750=0.03%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.01% f1: (groupid=0, jobs=1): err= 0: pid=6468 write: io=1,024MiB, bw=14,583KiB/s, iops=911, runt= 71903msec clat (usec): min=51, max=5,273K, avg=1093.52, stdev=41013.98 bw (KiB/s) : min= 0, max=110112, per=39.64%, avg=22453.36, stdev=24897.59 cpu : usr=0.40%, sys=8.60%, ctx=3644, majf=0, minf=226 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=62.62%, 250=36.64%, 500=0.19%, 750=0.04%, 1000=0.04% lat (msec): 2=0.11%, 4=0.11%, 10=0.09%, 20=0.02%, 50=0.01% lat (msec): 100=0.01%, 250=0.02%, 500=0.05%, 750=0.04%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.01% f2: (groupid=1, jobs=1): err= 0: pid=6689 write: io=1,024MiB, bw=879KiB/s, iops=54, runt=1192565msec clat (usec): min=50, max=28,980K, avg=18193.02, stdev=470690.60 bw (KiB/s) : min= 0, max=125760, per=103.49%, avg=3593.16, stdev=13646.83 cpu : usr=0.02%, sys=0.48%, ctx=2960, majf=0, minf=1196 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=84.51%, 250=14.97%, 500=0.05%, 750=0.03%, 1000=0.03% lat (msec): 2=0.08%, 4=0.02%, 10=0.05%, 20=0.02%, 50=0.01% lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.16% f2: (groupid=1, jobs=1): err= 0: pid=6690 write: io=1,024MiB, bw=980KiB/s, iops=61, runt=1069974msec clat (usec): min=51, max=28,754K, avg=16322.60, stdev=464684.31 bw (KiB/s) : min= 0, max=131680, per=166.66%, avg=5786.45, stdev=18638.92 cpu : usr=0.02%, sys=0.53%, ctx=2906, majf=0, minf=1044 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=84.95%, 250=14.58%, 500=0.06%, 750=0.04%, 1000=0.03% lat (msec): 2=0.08%, 4=0.02%, 10=0.02%, 20=0.02%, 50=0.02% lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.14% f2: (groupid=1, jobs=1): err= 0: pid=6691 write: io=1,024MiB, bw=1,026KiB/s, iops=64, runt=1021633msec clat (usec): min=48, max=27,719K, avg=15584.10, stdev=436543.32 bw (KiB/s) : min= 0, max=129440, per=200.18%, avg=6950.42, stdev=21996.65 cpu : usr=0.03%, sys=0.53%, ctx=2721, majf=0, minf=963 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 50=0.01%, 100=86.27%, 250=13.33%, 500=0.03%, 750=0.02% lat (usec): 1000=0.03% lat (msec): 2=0.04%, 4=0.03%, 10=0.02%, 20=0.01%, 50=0.02% lat (msec): 100=0.01%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.14% f2: (groupid=1, jobs=1): err= 0: pid=6692 write: io=1,024MiB, bw=868KiB/s, iops=54, runt=1207773msec clat (usec): min=51, max=26,607K, avg=18424.06, stdev=492387.17 bw (KiB/s) : min= 0, max=113344, per=124.66%, avg=4328.12, stdev=14107.27 cpu : usr=0.02%, sys=0.48%, ctx=3078, majf=0, minf=1213 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=81.78%, 250=17.66%, 500=0.06%, 750=0.03%, 1000=0.04% lat (msec): 2=0.10%, 4=0.02%, 10=0.04%, 20=0.02%, 50=0.02% lat (msec): 100=0.01%, 250=0.01%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.16% f3: (groupid=2, jobs=1): err= 0: pid=8881 read : io=1,024MiB, bw=14,808KiB/s, iops=925, runt= 70810msec clat (usec): min=7, max=575K, avg=1077.73, stdev=9562.22 bw (KiB/s) : min= 48, max=32637, per=26.30%, avg=15577.22, stdev=11538.01 cpu : usr=0.33%, sys=1.55%, ctx=4123, majf=0, minf=43 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 10=53.05%, 20=40.32%, 50=0.30%, 100=0.01%, 250=0.04% lat (usec): 500=0.01%, 750=0.01%, 1000=0.01% lat (msec): 2=0.18%, 4=0.44%, 10=2.47%, 20=2.58%, 50=0.23% lat (msec): 100=0.16%, 250=0.19%, 500=0.02%, 750=0.01% f3: (groupid=2, jobs=1): err= 0: pid=8882 read : io=1,024MiB, bw=14,839KiB/s, iops=927, runt= 70660msec clat (usec): min=7, max=526K, avg=1075.25, stdev=9258.82 bw (KiB/s) : min= 62, max=32063, per=26.48%, avg=15686.44, stdev=11593.57 cpu : usr=0.32%, sys=1.49%, ctx=4107, majf=0, minf=43 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 10=52.56%, 20=40.83%, 50=0.29%, 100=0.01%, 250=0.06% lat (usec): 500=0.01%, 750=0.01%, 1000=0.01% lat (msec): 2=0.13%, 4=0.41%, 10=2.58%, 20=2.47%, 50=0.28% lat (msec): 100=0.15%, 250=0.18%, 500=0.02%, 750=0.01% f3: (groupid=2, jobs=1): err= 0: pid=8883 read : io=1,024MiB, bw=14,899KiB/s, iops=931, runt= 70378msec clat (usec): min=7, max=551K, avg=1070.65, stdev=9263.27 bw (KiB/s) : min= 69, max=31107, per=26.35%, avg=15610.42, stdev=11794.70 cpu : usr=0.32%, sys=1.50%, ctx=4108, majf=0, minf=43 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 10=51.49%, 20=41.95%, 50=0.26%, 100=0.01%, 250=0.04% lat (usec): 500=0.01%, 750=0.01%, 1000=0.01% lat (msec): 2=0.17%, 4=0.47%, 10=2.48%, 20=2.49%, 50=0.25% lat (msec): 100=0.16%, 250=0.18%, 500=0.02%, 750=0.01% f3: (groupid=2, jobs=1): err= 0: pid=8884 read : io=1,024MiB, bw=14,820KiB/s, iops=926, runt= 70751msec clat (usec): min=7, max=509K, avg=1076.52, stdev=9138.06 bw (KiB/s) : min= 217, max=31072, per=26.03%, avg=15415.52, stdev=11315.14 cpu : usr=0.30%, sys=1.43%, ctx=4120, majf=0, minf=43 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 10=53.33%, 20=40.09%, 50=0.25%, 100=0.02%, 250=0.05% lat (usec): 500=0.01%, 1000=0.01% lat (msec): 2=0.14%, 4=0.42%, 10=2.46%, 20=2.57%, 50=0.29% lat (msec): 100=0.16%, 250=0.19%, 500=0.02%, 750=0.01% f4: (groupid=3, jobs=1): err= 0: pid=9063 read : io=1,024MiB, bw=926KiB/s, iops=57, runt=1131916msec clat (msec): min=2, max=150, avg=17.26, stdev=10.35 bw (KiB/s) : min= 552, max= 1622, per=25.15%, avg=927.10, stdev=115.24 cpu : usr=0.03%, sys=0.39%, ctx=262232, majf=0, minf=709 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (msec): 4=0.87%, 10=25.57%, 20=41.76%, 50=30.73%, 100=1.06% lat (msec): 250=0.01% f4: (groupid=3, jobs=1): err= 0: pid=9064 read : io=1,024MiB, bw=953KiB/s, iops=59, runt=1099545msec clat (msec): min=2, max=156, avg=16.77, stdev=10.21 bw (KiB/s) : min= 597, max= 1540, per=25.88%, avg=954.36, stdev=106.99 cpu : usr=0.03%, sys=0.40%, ctx=262227, majf=0, minf=908 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (msec): 4=0.88%, 10=27.38%, 20=41.97%, 50=28.79%, 100=0.98% lat (msec): 250=0.01% f4: (groupid=3, jobs=1): err= 0: pid=9065 read : io=1,024MiB, bw=950KiB/s, iops=59, runt=1103705msec clat (msec): min=2, max=153, avg=16.83, stdev=10.17 bw (KiB/s) : min= 507, max= 1763, per=25.78%, avg=950.60, stdev=111.03 cpu : usr=0.03%, sys=0.40%, ctx=262244, majf=0, minf=911 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (msec): 4=0.89%, 10=27.10%, 20=41.82%, 50=29.29%, 100=0.90% lat (msec): 250=0.01% f4: (groupid=3, jobs=1): err= 0: pid=9066 read : io=1,024MiB, bw=921KiB/s, iops=57, runt=1137579msec clat (msec): min=2, max=175, avg=17.35, stdev=10.41 bw (KiB/s) : min= 504, max= 2035, per=25.01%, avg=922.03, stdev=128.03 cpu : usr=0.03%, sys=0.39%, ctx=262255, majf=0, minf=940 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (msec): 4=0.98%, 10=25.25%, 20=41.53%, 50=31.18%, 100=1.05% lat (msec): 250=0.01% Run status group 0 (all jobs): WRITE: io=4,096MiB, aggrb=56,649KiB/s, minb=14,162KiB/s, maxb=15,915KiB/s, mint=65886msec, maxt=74041msec Run status group 1 (all jobs): WRITE: io=4,096MiB, aggrb=3,472KiB/s, minb=868KiB/s, maxb=1,026KiB/s, mint=1021633msec, maxt=1207773msec Run status group 2 (all jobs): READ: io=4,096MiB, aggrb=59,233KiB/s, minb=14,808KiB/s, maxb=14,899KiB/s, mint=70378msec, maxt=70810msec Run status group 3 (all jobs): READ: io=4,096MiB, aggrb=3,687KiB/s, minb=921KiB/s, maxb=953KiB/s, mint=1099545msec, maxt=1137579msec Disk stats (read/write): sda: ios=1066221/388934, merge=0/3877739, ticks=5092402/396631856, in_queue=401755349, util=99.26% 4.84user 105.09system 55:04.74elapsed 3%CPU (0avgtext+0avgdata 0maxresident)k 16778496inputs+33601336outputs (38major+24744minor)pagefaults 0swaps