f1: (g=0): rw=write, bs=16K-16K/16K-16K, ioengine=sync, iodepth=1 ... f1: (g=0): rw=write, bs=16K-16K/16K-16K, ioengine=sync, iodepth=1 f2: (g=1): rw=randwrite, bs=16K-16K/16K-16K, ioengine=sync, iodepth=1 ... f2: (g=1): rw=randwrite, bs=16K-16K/16K-16K, ioengine=sync, iodepth=1 f3: (g=2): rw=read, bs=16K-16K/16K-16K, ioengine=sync, iodepth=1 ... f3: (g=2): rw=read, bs=16K-16K/16K-16K, ioengine=sync, iodepth=1 f4: (g=3): rw=randread, bs=16K-16K/16K-16K, ioengine=sync, iodepth=1 ... f4: (g=3): rw=randread, bs=16K-16K/16K-16K, ioengine=sync, iodepth=1 Starting 16 processes f1: Laying out IO file(s) (1 file(s) / 1024MiB) f1: Laying out IO file(s) (1 file(s) / 1024MiB) f1: Laying out IO file(s) (1 file(s) / 1024MiB) f1: Laying out IO file(s) (1 file(s) / 1024MiB) f2: Laying out IO file(s) (1 file(s) / 1024MiB) f2: Laying out IO file(s) (1 file(s) / 1024MiB) f2: Laying out IO file(s) (1 file(s) / 1024MiB) f2: Laying out IO file(s) (1 file(s) / 1024MiB) f3: Laying out IO file(s) (1 file(s) / 1024MiB) f3: Laying out IO file(s) (1 file(s) / 1024MiB) f3: Laying out IO file(s) (1 file(s) / 1024MiB) f3: Laying out IO file(s) (1 file(s) / 1024MiB) f4: Laying out IO file(s) (1 file(s) / 1024MiB) f4: Laying out IO file(s) (1 file(s) / 1024MiB) f4: Laying out IO file(s) (1 file(s) / 1024MiB) f4: Laying out IO file(s) (1 file(s) / 1024MiB) f1: (groupid=0, jobs=1): err= 0: pid=6448 write: io=1,024MiB, bw=13,781KiB/s, iops=861, runt= 76085msec clat (usec): min=50, max=1,689K, avg=1141.23, stdev=26713.63 bw (KiB/s) : min= 0, max=119456, per=29.22%, avg=16010.48, stdev=25296.03 cpu : usr=0.33%, sys=8.36%, ctx=5313, majf=0, minf=128 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=58.01%, 250=41.66%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec): 2=0.04%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec): 100=0.01%, 250=0.01%, 500=0.07%, 750=0.07%, 1000=0.02% lat (msec): 2000=0.01% f1: (groupid=0, jobs=1): err= 0: pid=6449 write: io=1,024MiB, bw=13,786KiB/s, iops=861, runt= 76057msec clat (usec): min=51, max=1,418K, avg=1117.92, stdev=26625.21 bw (KiB/s) : min= 0, max=128160, per=29.90%, avg=16384.99, stdev=27576.43 cpu : usr=0.29%, sys=8.49%, ctx=3563, majf=0, minf=126 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=54.80%, 250=44.91%, 500=0.03%, 750=0.01%, 1000=0.02% lat (msec): 2=0.03%, 4=0.02%, 10=0.02%, 20=0.01%, 50=0.01% lat (msec): 250=0.01%, 500=0.07%, 750=0.06%, 1000=0.02%, 2000=0.01% f1: (groupid=0, jobs=1): err= 0: pid=6450 write: io=1,024MiB, bw=13,697KiB/s, iops=856, runt= 76550msec clat (usec): min=51, max=2,172K, avg=1130.62, stdev=27029.17 bw (KiB/s) : min= 0, max=117536, per=28.70%, avg=15725.25, stdev=24377.22 cpu : usr=0.32%, sys=8.70%, ctx=6106, majf=0, minf=130 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=56.73%, 250=42.98%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec): 2=0.02%, 4=0.01%, 10=0.02%, 20=0.01%, 50=0.01% lat (msec): 100=0.01%, 250=0.02%, 500=0.07%, 750=0.07%, 1000=0.02% lat (msec): 2000=0.01%, >=2000=0.01% f1: (groupid=0, jobs=1): err= 0: pid=6451 write: io=1,024MiB, bw=13,773KiB/s, iops=860, runt= 76132msec clat (usec): min=51, max=2,288K, avg=1133.81, stdev=27312.75 bw (KiB/s) : min= 0, max=126656, per=29.55%, avg=16189.54, stdev=26756.03 cpu : usr=0.34%, sys=8.53%, ctx=3529, majf=0, minf=126 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 100=55.07%, 250=44.54%, 500=0.09%, 750=0.02%, 1000=0.01% lat (msec): 2=0.04%, 4=0.03%, 10=0.02%, 20=0.01%, 50=0.01% lat (msec): 500=0.07%, 750=0.07%, 1000=0.02%, 2000=0.01%, >=2000=0.01% f2: (groupid=1, jobs=1): err= 0: pid=6529 write: io=1,024MiB, bw=1,250KiB/s, iops=78, runt=838698msec clat (usec): min=46, max=16,640K, avg=12552.09, stdev=306744.93 bw (KiB/s) : min= 0, max=11906, per=39.75%, avg=1987.66, stdev=2422.81 cpu : usr=0.03%, sys=0.59%, ctx=4109, majf=0, minf=505 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 50=0.03%, 100=98.16%, 250=1.38%, 500=0.02%, 750=0.03% lat (usec): 1000=0.07% lat (msec): 2=0.03%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02% lat (msec): 2000=0.02%, >=2000=0.19% f2: (groupid=1, jobs=1): err= 0: pid=6530 write: io=1,024MiB, bw=1,250KiB/s, iops=78, runt=838679msec clat (usec): min=48, max=17,745K, avg=12641.13, stdev=300115.58 bw (KiB/s) : min= 0, max=18797, per=40.29%, avg=2014.62, stdev=2865.29 cpu : usr=0.03%, sys=0.59%, ctx=4076, majf=0, minf=517 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 50=0.03%, 100=98.51%, 250=1.05%, 500=0.02%, 750=0.03% lat (usec): 1000=0.05% lat (msec): 2=0.02%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (msec): 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.02%, 2000=0.04% lat (msec): >=2000=0.20% f2: (groupid=1, jobs=1): err= 0: pid=6531 write: io=1,024MiB, bw=1,250KiB/s, iops=78, runt=838719msec clat (usec): min=48, max=18,325K, avg=12646.10, stdev=315573.51 bw (KiB/s) : min= 0, max=24117, per=42.33%, avg=2116.58, stdev=3027.40 cpu : usr=0.03%, sys=0.59%, ctx=4106, majf=0, minf=500 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 50=0.01%, 100=98.15%, 250=1.42%, 500=0.03%, 750=0.04% lat (usec): 1000=0.07% lat (msec): 2=0.02%, 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01% lat (msec): 500=0.01%, 750=0.02%, 1000=0.01%, 2000=0.02%, >=2000=0.19% f2: (groupid=1, jobs=1): err= 0: pid=6532 write: io=1,024MiB, bw=1,270KiB/s, iops=79, runt=825507msec clat (usec): min=49, max=19,115K, avg=12203.89, stdev=276887.49 bw (KiB/s) : min= 0, max=21720, per=37.86%, avg=1893.08, stdev=2600.84 cpu : usr=0.03%, sys=0.60%, ctx=4072, majf=0, minf=540 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=0/65536, short=0/0 lat (usec): 50=0.01%, 100=99.09%, 250=0.48%, 500=0.02%, 750=0.03% lat (usec): 1000=0.06% lat (msec): 2=0.02%, 10=0.01%, 20=0.01%, 50=0.01%, 250=0.01% lat (msec): 500=0.01%, 750=0.02%, 1000=0.01%, 2000=0.04%, >=2000=0.21% f3: (groupid=2, jobs=1): err= 0: pid=7632 read : io=1,024MiB, bw=5,389KiB/s, iops=336, runt=194574msec clat (usec): min=7, max=577K, avg=2966.11, stdev=20636.64 bw (KiB/s) : min= 64, max= 9936, per=25.79%, avg=5535.31, stdev=2091.16 cpu : usr=0.13%, sys=0.56%, ctx=3106, majf=0, minf=68 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 10=49.69%, 20=42.33%, 50=0.44%, 100=0.40%, 250=2.50% lat (usec): 500=0.02%, 750=0.02%, 1000=0.08% lat (msec): 2=0.18%, 4=0.21%, 10=0.47%, 20=0.65%, 50=0.90% lat (msec): 100=1.14%, 250=0.85%, 500=0.12%, 750=0.01% f3: (groupid=2, jobs=1): err= 0: pid=7633 read : io=1,024MiB, bw=5,382KiB/s, iops=336, runt=194827msec clat (usec): min=7, max=513K, avg=2970.00, stdev=20438.79 bw (KiB/s) : min= 64, max=10698, per=25.51%, avg=5475.08, stdev=2070.39 cpu : usr=0.11%, sys=0.55%, ctx=3083, majf=0, minf=68 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 10=48.47%, 20=43.54%, 50=0.46%, 100=0.40%, 250=2.52% lat (usec): 500=0.01%, 750=0.02%, 1000=0.11% lat (msec): 2=0.13%, 4=0.19%, 10=0.42%, 20=0.67%, 50=0.93% lat (msec): 100=1.14%, 250=0.85%, 500=0.12%, 750=0.01% f3: (groupid=2, jobs=1): err= 0: pid=7634 read : io=1,024MiB, bw=5,413KiB/s, iops=338, runt=193707msec clat (usec): min=6, max=574K, avg=2952.29, stdev=20310.18 bw (KiB/s) : min= 77, max=10459, per=25.73%, avg=5521.93, stdev=2028.20 cpu : usr=0.12%, sys=0.57%, ctx=3033, majf=0, minf=68 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 10=50.62%, 20=41.40%, 50=0.47%, 100=0.46%, 250=2.51% lat (usec): 500=0.01%, 750=0.02%, 1000=0.10% lat (msec): 2=0.12%, 4=0.16%, 10=0.43%, 20=0.63%, 50=0.93% lat (msec): 100=1.15%, 250=0.85%, 500=0.11%, 750=0.01% f3: (groupid=2, jobs=1): err= 0: pid=7635 read : io=1,024MiB, bw=5,364KiB/s, iops=335, runt=195456msec clat (usec): min=7, max=572K, avg=2970.97, stdev=20590.75 bw (KiB/s) : min= 64, max=19538, per=25.43%, avg=5457.39, stdev=2142.26 cpu : usr=0.12%, sys=0.56%, ctx=3148, majf=0, minf=68 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 10=48.46%, 20=43.54%, 50=0.44%, 100=0.36%, 250=2.52% lat (usec): 500=0.01%, 750=0.04%, 1000=0.12% lat (msec): 2=0.16%, 4=0.23%, 10=0.49%, 20=0.63%, 50=0.91% lat (msec): 100=1.09%, 250=0.88%, 500=0.11%, 750=0.01% f4: (groupid=3, jobs=1): err= 0: pid=7922 read : io=1,024MiB, bw=416KiB/s, iops=26, runt=2519712msec clat (usec): min=556, max=1,856K, avg=38416.34, stdev=44800.99 bw (KiB/s) : min= 8, max= 1045, per=25.79%, avg=428.05, stdev=103.08 cpu : usr=0.01%, sys=0.13%, ctx=262390, majf=0, minf=2046 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 750=0.03%, 1000=0.06% lat (msec): 2=0.03%, 4=0.02%, 10=0.06%, 20=2.80%, 50=90.78% lat (msec): 100=4.88%, 250=0.62%, 500=0.36%, 750=0.33%, 1000=0.02% lat (msec): 2000=0.02% f4: (groupid=3, jobs=1): err= 0: pid=7923 read : io=1,024MiB, bw=416KiB/s, iops=26, runt=2516562msec clat (usec): min=556, max=1,758K, avg=38362.80, stdev=44899.10 bw (KiB/s) : min= 9, max= 775, per=25.81%, avg=428.47, stdev=102.57 cpu : usr=0.01%, sys=0.14%, ctx=262406, majf=0, minf=2043 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 750=0.03%, 1000=0.05% lat (msec): 2=0.02%, 4=0.01%, 10=0.06%, 20=2.89%, 50=90.79% lat (msec): 100=4.83%, 250=0.61%, 500=0.34%, 750=0.34%, 1000=0.02% lat (msec): 2000=0.02% f4: (groupid=3, jobs=1): err= 0: pid=7924 read : io=1,024MiB, bw=417KiB/s, iops=26, runt=2511175msec clat (usec): min=557, max=1,693K, avg=38298.54, stdev=44981.33 bw (KiB/s) : min= 9, max= 695, per=25.90%, avg=429.86, stdev=102.20 cpu : usr=0.01%, sys=0.14%, ctx=262412, majf=0, minf=2035 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 750=0.05%, 1000=0.12% lat (msec): 2=0.04%, 4=0.01%, 10=0.04%, 20=2.70%, 50=91.09% lat (msec): 100=4.62%, 250=0.62%, 500=0.34%, 750=0.35%, 1000=0.02% lat (msec): 2000=0.01% f4: (groupid=3, jobs=1): err= 0: pid=7925 read : io=1,024MiB, bw=415KiB/s, iops=25, runt=2525218msec clat (usec): min=559, max=1,544K, avg=38501.86, stdev=44833.72 bw (KiB/s) : min= 10, max= 2661, per=25.73%, avg=427.20, stdev=128.15 cpu : usr=0.01%, sys=0.14%, ctx=262423, majf=0, minf=2048 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=65536/0, short=0/0 lat (usec): 750=0.08%, 1000=0.08% lat (msec): 2=0.19%, 4=0.22%, 10=0.18%, 20=3.07%, 50=89.65% lat (msec): 100=5.18%, 250=0.62%, 500=0.36%, 750=0.34%, 1000=0.01% lat (msec): 2000=0.02% Run status group 0 (all jobs): WRITE: io=4,096MiB, aggrb=54,791KiB/s, minb=13,697KiB/s, maxb=13,786KiB/s, mint=76057msec, maxt=76550msec Run status group 1 (all jobs): WRITE: io=4,096MiB, aggrb=5,000KiB/s, minb=1,250KiB/s, maxb=1,270KiB/s, mint=825507msec, maxt=838719msec Run status group 2 (all jobs): READ: io=4,096MiB, aggrb=21,459KiB/s, minb=5,364KiB/s, maxb=5,413KiB/s, mint=193707msec, maxt=195456msec Run status group 3 (all jobs): READ: io=4,096MiB, aggrb=1,660KiB/s, minb=415KiB/s, maxb=417KiB/s, mint=2511175msec, maxt=2525218msec Disk stats (read/write): cciss!c0d0: ios=1100469/584955, merge=1089/3723613, ticks=14041916/205582811, in_queue=219624359, util=99.37% 4.87user 97.97system 1:03:25elapsed 2%CPU (0avgtext+0avgdata 0maxresident)k 16777528inputs+33610640outputs (38major+28821minor)pagefaults 0swaps