[lustre-devel] IOR MPIRUN read low performance and values are not consistent
muhammad uzair jawaid
muhammaduzairjawaid at gmail.com
Wed Oct 9 08:44:28 PDT 2024
Hi,
I have 2 question
1) I run tests using fio and mpirun. The write speed for both are same but
the read speed of mpirun is very low as compared to fio. My fio test gave
around 8GB whereas my IOR test gave only 1.5GB. I see on top utility
mpirun read is only using 1 cpu whereas fio uses 12 cpus.
2) why the result in mpirun/fio read and write are not constant? I have
isolated CPUS on numa 1 and my network interfaces are also on numa 1. On
every run I see a different value which can deviate upto 4 GB .
I am attaching my test in a text file.
My setup is 2 lustre servers having 3 OSS and 1 MGS/MDS and 1 lustre
client. I am using rdma rocev2. 1 NIC card 100Gbps Connectx5 on each
server.
Thanks for the help.
Best regards
Muhammad Uzair Jawaid
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20241009/bcf4d846/attachment-0001.htm>
-------------- next part --------------
root at ec-g-094-kh:/home/uzair# mpirun --allow-run-as-root --rankfile /mnt/lustre/rankfile.txt -np 12 /usr/local/bin/ior -F -t 128M -b 2G -k -r -w -o /mnt/lustre/testfile -s 3 --posix.odirect -M 10g
IOR-4.1.0+dev: MPI Coordinated Test of Parallel I/O
Began : Wed Oct 9 15:26:39 2024
Command line : /usr/local/bin/ior -F -t 128M -b 2G -k -r -w -o /mnt/lustre/testfile -s 3 --posix.odirect -M 10g
Machine : Linux ec-g-094-kh
TestID : 0
StartTime : Wed Oct 9 15:26:39 2024
Path : /mnt/lustre/testfile.00000000
FS : 22.2 TiB Used FS: 0.6% Inodes: 44.7 Mi Used Inodes: 0.0%
Options:
api : POSIX
apiVersion :
test filename : /mnt/lustre/testfile
access : file-per-process
type : independent
segments : 3
ordering in a file : sequential
ordering inter file : no tasks offsets
nodes : 1
tasks : 12
clients per node : 12
memoryPerNode : 25.17 GiB
memoryBuffer : CPU
dataAccess : CPU
GPUDirect : 0
repetitions : 1
xfersize : 128 MiB
blocksize : 2 GiB
aggregate filesize : 72 GiB
Results:
WARNING: The file "/mnt/lustre/testfile.00000010" exists already and will be deleted
WARNING: The file "/mnt/lustre/testfile.00000004" exists already and will be deleted
WARNING: The file "/mnt/lustre/testfile.00000006" exists already and will be deleted
WARNING: The file "/mnt/lustre/testfile.00000000" exists already and will be deleted
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
WARNING: The file "/mnt/lustre/testfile.00000002" exists already and will be deleted
WARNING: The file "/mnt/lustre/testfile.00000007" exists already and will be deleted
WARNING: The file "/mnt/lustre/testfile.00000005" exists already and will be deleted
WARNING: The file "/mnt/lustre/testfile.00000008" exists already and will be deleted
WARNING: The file "/mnt/lustre/testfile.00000009" exists already and will be deleted
WARNING: The file "/mnt/lustre/testfile.00000011" exists already and will be deleted
WARNING: The file "/mnt/lustre/testfile.00000001" exists already and will be deleted
WARNING: The file "/mnt/lustre/testfile.00000003" exists already and will be deleted
write 10001 78.14 0.460550 2097152 131072 0.004889 7.37 0.556516 7.37 0
read 1511.52 11.81 3.05 2097152 131072 0.000603 48.78 6.56 48.78 0
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum
write 10000.92 10000.92 10000.92 0.00 78.13 78.13 78.13 0.00 7.37212 NA NA 0 12 12 1 1 0 1 0 0 3 2147483648 134217728 73728.0 POSIX 0
read 1511.52 1511.52 1511.52 0.00 11.81 11.81 11.81 0.00 48.77747 NA NA 0 12 12 1 1 0 1 0 0 3 2147483648 134217728 73728.0 POSIX 0
Finished : Wed Oct 9 15:27:37 2024
root at ec-g-094-kh:/home/uzair# /home/uzair/fio/fio --bs=256M --numjobs=12 --iodepth=128 --loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --filename=/
mnt/lustre/testfile1 --name=read-phase --rw=read --size=10GB --cpus_allowed=12-23 --cpus_allowed_policy=split
read-phase: (g=0): rw=read, bs=(R) 256MiB-256MiB, (W) 256MiB-256MiB, (T) 256MiB-256MiB, ioengine=libaio, iodepth=128
...
fio-3.37-78-gc63e
Starting 12 processes
Jobs: 9 (f=0): [f(8),_(3),f(1)][-.-%][eta 00m:00s]
read-phase: (groupid=0, jobs=1): err= 0: pid=54091: Wed Oct 9 15:32:50 2024
read: IOPS=2, BW=602MiB/s (632MB/s)(10.0GiB/16996msec)
slat (msec): min=84, max=598, avg=191.08, stdev=96.61
clat (msec): min=9349, max=16769, avg=13614.56, stdev=2341.90
lat (msec): min=9665, max=16993, avg=13805.64, stdev=2296.36
clat percentiles (msec):
| 1.00th=[ 9329], 5.00th=[ 9731], 10.00th=[10134], 20.00th=[11073],
| 30.00th=[11879], 40.00th=[12684], 50.00th=[13624], 60.00th=[14966],
| 70.00th=[15503], 80.00th=[16040], 90.00th=[16308], 95.00th=[16576],
| 99.00th=[16711], 99.50th=[16711], 99.90th=[16711], 99.95th=[16711],
| 99.99th=[16711]
bw ( MiB/s): min=20480, max=20480, per=100.00%, avg=20480.00, stdev= 0.00, samples=1
iops : min= 80, max= 80, avg=80.00, stdev= 0.00, samples=1
lat (msec) : >=2000=100.00%
cpu : usr=0.00%, sys=22.74%, ctx=18226, majf=0, minf=2621460
IO depths : 1=2.5%, 2=5.0%, 4=10.0%, 8=20.0%, 16=40.0%, 32=22.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
issued rwts: total=40,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
read-phase: (groupid=0, jobs=1): err= 0: pid=54092: Wed Oct 9 15:32:50 2024
read: IOPS=2, BW=592MiB/s (621MB/s)(10.0GiB/17290msec)
slat (msec): min=102, max=1261, avg=208.63, stdev=183.61
clat (msec): min=8940, max=16023, avg=12901.42, stdev=2200.83
lat (msec): min=9177, max=17285, avg=13110.05, stdev=2211.22
clat percentiles (msec):
| 1.00th=[ 8926], 5.00th=[ 9194], 10.00th=[ 9597], 20.00th=[10537],
| 30.00th=[11342], 40.00th=[12147], 50.00th=[13087], 60.00th=[14026],
| 70.00th=[14563], 80.00th=[14966], 90.00th=[15637], 95.00th=[15771],
| 99.00th=[16040], 99.50th=[16040], 99.90th=[16040], 99.95th=[16040],
| 99.99th=[16040]
bw ( MiB/s): min=20480, max=20480, per=100.00%, avg=20480.00, stdev= 0.00, samples=1
iops : min= 80, max= 80, avg=80.00, stdev= 0.00, samples=1
lat (msec) : >=2000=100.00%
cpu : usr=0.01%, sys=21.97%, ctx=8759, majf=0, minf=2621460
IO depths : 1=2.5%, 2=5.0%, 4=10.0%, 8=20.0%, 16=40.0%, 32=22.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
issued rwts: total=40,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
read-phase: (groupid=0, jobs=1): err= 0: pid=54093: Wed Oct 9 15:32:50 2024
read: IOPS=2, BW=597MiB/s (626MB/s)(10.0GiB/17165msec)
slat (msec): min=90, max=668, avg=199.34, stdev=116.21
clat (msec): min=9187, max=16492, avg=13358.94, stdev=2308.91
lat (msec): min=9449, max=17161, avg=13558.27, stdev=2283.59
clat percentiles (msec):
| 1.00th=[ 9194], 5.00th=[ 9463], 10.00th=[10000], 20.00th=[10805],
| 30.00th=[11610], 40.00th=[12550], 50.00th=[13355], 60.00th=[14697],
| 70.00th=[15234], 80.00th=[15637], 90.00th=[16174], 95.00th=[16308],
| 99.00th=[16442], 99.50th=[16442], 99.90th=[16442], 99.95th=[16442],
| 99.99th=[16442]
bw ( MiB/s): min=20480, max=20480, per=100.00%, avg=20480.00, stdev= 0.00, samples=1
iops : min= 80, max= 80, avg=80.00, stdev= 0.00, samples=1
lat (msec) : >=2000=100.00%
cpu : usr=0.01%, sys=22.74%, ctx=4646, majf=0, minf=2621455
IO depths : 1=2.5%, 2=5.0%, 4=10.0%, 8=20.0%, 16=40.0%, 32=22.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
issued rwts: total=40,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
read-phase: (groupid=0, jobs=1): err= 0: pid=54094: Wed Oct 9 15:32:50 2024
read: IOPS=2, BW=596MiB/s (625MB/s)(10.0GiB/17191msec)
slat (msec): min=94, max=773, avg=201.26, stdev=125.36
clat (msec): min=9136, max=16413, avg=13265.36, stdev=2291.35
lat (msec): min=9393, max=17187, avg=13466.62, stdev=2272.87
clat percentiles (msec):
| 1.00th=[ 9194], 5.00th=[ 9329], 10.00th=[ 9866], 20.00th=[10805],
| 30.00th=[11610], 40.00th=[12416], 50.00th=[13355], 60.00th=[14563],
| 70.00th=[15100], 80.00th=[15503], 90.00th=[16040], 95.00th=[16174],
| 99.00th=[16442], 99.50th=[16442], 99.90th=[16442], 99.95th=[16442],
| 99.99th=[16442]
bw ( MiB/s): min=20480, max=20480, per=100.00%, avg=20480.00, stdev= 0.00, samples=1
iops : min= 80, max= 80, avg=80.00, stdev= 0.00, samples=1
lat (msec) : >=2000=100.00%
cpu : usr=0.00%, sys=22.41%, ctx=4684, majf=0, minf=2621456
IO depths : 1=2.5%, 2=5.0%, 4=10.0%, 8=20.0%, 16=40.0%, 32=22.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
issued rwts: total=40,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
read-phase: (groupid=0, jobs=1): err= 0: pid=54095: Wed Oct 9 15:32:50 2024
read: IOPS=2, BW=593MiB/s (622MB/s)(10.0GiB/17267msec)
slat (msec): min=103, max=1132, avg=206.81, stdev=171.82
clat (msec): min=8990, max=16130, avg=13041.15, stdev=2243.44
lat (msec): min=9235, max=17263, avg=13247.96, stdev=2241.79
clat percentiles (msec):
| 1.00th=[ 8926], 5.00th=[ 9194], 10.00th=[ 9731], 20.00th=[10537],
| 30.00th=[11342], 40.00th=[12281], 50.00th=[13087], 60.00th=[14429],
| 70.00th=[14832], 80.00th=[15234], 90.00th=[15637], 95.00th=[15905],
| 99.00th=[16174], 99.50th=[16174], 99.90th=[16174], 99.95th=[16174],
| 99.99th=[16174]
bw ( MiB/s): min=20480, max=20480, per=100.00%, avg=20480.00, stdev= 0.00, samples=1
iops : min= 80, max= 80, avg=80.00, stdev= 0.00, samples=1
lat (msec) : >=2000=100.00%
cpu : usr=0.00%, sys=22.17%, ctx=69, majf=0, minf=2621457
IO depths : 1=2.5%, 2=5.0%, 4=10.0%, 8=20.0%, 16=40.0%, 32=22.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
issued rwts: total=40,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
read-phase: (groupid=0, jobs=1): err= 0: pid=54096: Wed Oct 9 15:32:50 2024
read: IOPS=2, BW=595MiB/s (624MB/s)(10.0GiB/17215msec)
slat (msec): min=96, max=888, avg=203.18, stdev=139.19
clat (msec): min=9083, max=16322, avg=13188.54, stdev=2274.40
lat (msec): min=9338, max=17210, avg=13391.72, stdev=2261.26
clat percentiles (msec):
| 1.00th=[ 9060], 5.00th=[ 9329], 10.00th=[ 9866], 20.00th=[10671],
| 30.00th=[11476], 40.00th=[12416], 50.00th=[13221], 60.00th=[14563],
| 70.00th=[14966], 80.00th=[15503], 90.00th=[15905], 95.00th=[16174],
| 99.00th=[16308], 99.50th=[16308], 99.90th=[16308], 99.95th=[16308],
| 99.99th=[16308]
bw ( MiB/s): min=20480, max=20480, per=100.00%, avg=20480.00, stdev= 0.00, samples=1
iops : min= 80, max= 80, avg=80.00, stdev= 0.00, samples=1
lat (msec) : >=2000=100.00%
cpu : usr=0.01%, sys=22.22%, ctx=63, majf=0, minf=2621458
IO depths : 1=2.5%, 2=5.0%, 4=10.0%, 8=20.0%, 16=40.0%, 32=22.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
issued rwts: total=40,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
read-phase: (groupid=0, jobs=1): err= 0: pid=54097: Wed Oct 9 15:32:50 2024
read: IOPS=2, BW=592MiB/s (620MB/s)(10.0GiB/17306msec)
slat (msec): min=101, max=1392, avg=210.44, stdev=203.59
clat (msec): min=8884, max=15909, avg=12823.63, stdev=2189.23
lat (msec): min=9120, max=17302, avg=13034.06, stdev=2205.38
clat percentiles (msec):
| 1.00th=[ 8926], 5.00th=[ 9060], 10.00th=[ 9597], 20.00th=[10402],
| 30.00th=[11208], 40.00th=[12013], 50.00th=[12953], 60.00th=[14026],
| 70.00th=[14429], 80.00th=[14966], 90.00th=[15503], 95.00th=[15637],
| 99.00th=[15905], 99.50th=[15905], 99.90th=[15905], 99.95th=[15905],
| 99.99th=[15905]
bw ( MiB/s): min=20480, max=20480, per=100.00%, avg=20480.00, stdev= 0.00, samples=1
iops : min= 80, max= 80, avg=80.00, stdev= 0.00, samples=1
lat (msec) : >=2000=100.00%
cpu : usr=0.00%, sys=21.45%, ctx=75, majf=0, minf=2621458
IO depths : 1=2.5%, 2=5.0%, 4=10.0%, 8=20.0%, 16=40.0%, 32=22.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
issued rwts: total=40,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
read-phase: (groupid=0, jobs=1): err= 0: pid=54098: Wed Oct 9 15:32:50 2024
read: IOPS=2, BW=597MiB/s (626MB/s)(10.0GiB/17139msec)
slat (msec): min=90, max=565, avg=197.41, stdev=106.95
clat (msec): min=9238, max=16580, avg=13443.95, stdev=2319.91
lat (msec): min=9532, max=17135, avg=13641.36, stdev=2288.63
clat percentiles (msec):
| 1.00th=[ 9194], 5.00th=[ 9597], 10.00th=[10000], 20.00th=[10939],
| 30.00th=[11745], 40.00th=[12550], 50.00th=[13489], 60.00th=[14697],
| 70.00th=[15368], 80.00th=[15771], 90.00th=[16174], 95.00th=[16442],
| 99.00th=[16576], 99.50th=[16576], 99.90th=[16576], 99.95th=[16576],
| 99.99th=[16576]
bw ( MiB/s): min=20480, max=20480, per=100.00%, avg=20480.00, stdev= 0.00, samples=1
iops : min= 80, max= 80, avg=80.00, stdev= 0.00, samples=1
lat (msec) : >=2000=100.00%
cpu : usr=0.00%, sys=22.59%, ctx=4682, majf=0, minf=2621458
IO depths : 1=2.5%, 2=5.0%, 4=10.0%, 8=20.0%, 16=40.0%, 32=22.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
issued rwts: total=40,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
read-phase: (groupid=0, jobs=1): err= 0: pid=54099: Wed Oct 9 15:32:50 2024
read: IOPS=2, BW=601MiB/s (630MB/s)(10.0GiB/17042msec)
slat (msec): min=89, max=588, avg=193.24, stdev=96.16
clat (msec): min=9308, max=16711, avg=13562.68, stdev=2336.23
lat (msec): min=9621, max=17038, avg=13755.92, stdev=2294.66
clat percentiles (msec):
| 1.00th=[ 9329], 5.00th=[ 9597], 10.00th=[10134], 20.00th=[11073],
| 30.00th=[11879], 40.00th=[12684], 50.00th=[13624], 60.00th=[14832],
| 70.00th=[15503], 80.00th=[15905], 90.00th=[16308], 95.00th=[16442],
| 99.00th=[16711], 99.50th=[16711], 99.90th=[16711], 99.95th=[16711],
| 99.99th=[16711]
bw ( MiB/s): min=20480, max=20480, per=100.00%, avg=20480.00, stdev= 0.00, samples=1
iops : min= 80, max= 80, avg=80.00, stdev= 0.00, samples=1
lat (msec) : >=2000=100.00%
cpu : usr=0.00%, sys=22.99%, ctx=65, majf=0, minf=2621458
IO depths : 1=2.5%, 2=5.0%, 4=10.0%, 8=20.0%, 16=40.0%, 32=22.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
issued rwts: total=40,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
read-phase: (groupid=0, jobs=1): err= 0: pid=54100: Wed Oct 9 15:32:50 2024
read: IOPS=2, BW=599MiB/s (628MB/s)(10.0GiB/17091msec)
slat (msec): min=89, max=578, avg=195.30, stdev=101.09
clat (msec): min=9274, max=16645, avg=13507.12, stdev=2331.54
lat (msec): min=9574, max=17087, avg=13702.42, stdev=2294.85
clat percentiles (msec):
| 1.00th=[ 9329], 5.00th=[ 9597], 10.00th=[10134], 20.00th=[10939],
| 30.00th=[11745], 40.00th=[12684], 50.00th=[13489], 60.00th=[14832],
| 70.00th=[15368], 80.00th=[15905], 90.00th=[16308], 95.00th=[16442],
| 99.00th=[16711], 99.50th=[16711], 99.90th=[16711], 99.95th=[16711],
| 99.99th=[16711]
bw ( MiB/s): min=20480, max=20480, per=100.00%, avg=20480.00, stdev= 0.00, samples=1
iops : min= 80, max= 80, avg=80.00, stdev= 0.00, samples=1
lat (msec) : >=2000=100.00%
cpu : usr=0.01%, sys=22.35%, ctx=4323, majf=0, minf=2621457
IO depths : 1=2.5%, 2=5.0%, 4=10.0%, 8=20.0%, 16=40.0%, 32=22.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
issued rwts: total=40,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
read-phase: (groupid=0, jobs=1): err= 0: pid=54101: Wed Oct 9 15:32:50 2024
read: IOPS=2, BW=656MiB/s (688MB/s)(10.0GiB/15617msec)
slat (msec): min=80, max=132, avg=104.16, stdev=12.87
clat (msec): min=11448, max=15519, avg=13543.82, stdev=1252.17
lat (msec): min=11546, max=15615, avg=13647.99, stdev=1246.73
clat percentiles (msec):
| 1.00th=[11476], 5.00th=[11610], 10.00th=[11745], 20.00th=[12147],
| 30.00th=[12550], 40.00th=[13087], 50.00th=[13489], 60.00th=[13892],
| 70.00th=[14429], 80.00th=[14832], 90.00th=[15234], 95.00th=[15368],
| 99.00th=[15503], 99.50th=[15503], 99.90th=[15503], 99.95th=[15503],
| 99.99th=[15503]
bw ( MiB/s): min=20480, max=20480, per=100.00%, avg=20480.00, stdev= 0.00, samples=1
iops : min= 80, max= 80, avg=80.00, stdev= 0.00, samples=1
lat (msec) : >=2000=100.00%
cpu : usr=0.00%, sys=24.47%, ctx=4227, majf=0, minf=2621455
IO depths : 1=2.5%, 2=5.0%, 4=10.0%, 8=20.0%, 16=40.0%, 32=22.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
issued rwts: total=40,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
read-phase: (groupid=0, jobs=1): err= 0: pid=54102: Wed Oct 9 15:32:50 2024
read: IOPS=2, BW=594MiB/s (623MB/s)(10.0GiB/17240msec)
slat (msec): min=102, max=1011, avg=205.06, stdev=155.10
clat (msec): min=9033, max=16224, avg=13111.11, stdev=2254.96
lat (msec): min=9285, max=17236, avg=13316.17, stdev=2247.61
clat percentiles (msec):
| 1.00th=[ 9060], 5.00th=[ 9329], 10.00th=[ 9731], 20.00th=[10671],
| 30.00th=[11476], 40.00th=[12281], 50.00th=[13221], 60.00th=[14429],
| 70.00th=[14832], 80.00th=[15368], 90.00th=[15771], 95.00th=[16040],
| 99.00th=[16174], 99.50th=[16174], 99.90th=[16174], 99.95th=[16174],
| 99.99th=[16174]
bw ( MiB/s): min=20480, max=20480, per=100.00%, avg=20480.00, stdev= 0.00, samples=1
iops : min= 80, max= 80, avg=80.00, stdev= 0.00, samples=1
lat (msec) : >=2000=100.00%
cpu : usr=0.01%, sys=22.13%, ctx=66, majf=0, minf=2621456
IO depths : 1=2.5%, 2=5.0%, 4=10.0%, 8=20.0%, 16=40.0%, 32=22.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
issued rwts: total=40,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
READ: bw=7100MiB/s (7445MB/s), 592MiB/s-656MiB/s (620MB/s-688MB/s), io=120GiB (129GB), run=15617-17306msec
root at ec-g-094-kh:/home/uzair#
More information about the lustre-devel
mailing list