<HTML><HEAD></HEAD>
<BODY dir=ltr>
<DIV dir=ltr>
<DIV style="FONT-FAMILY: 'Calibri'; COLOR: #000000; FONT-SIZE: 12pt">
<DIV>Hi,</DIV>
<DIV> </DIV>
<DIV>I am experiencing poor direct-IO performance using Lustre 2.1.5 (latest
stable) on CentOS 6.3.</DIV>
<DIV> </DIV>
<DIV>Two OSS servers connect to the same MD3200 (daisy chained by 4
MD1200).</DIV>
<DIV>5 disks (from each MD) form a RAID-5 virtual disk as an OST.</DIV>
<DIV>8 OSTs are created in the file system.</DIV>
<DIV> </DIV>
<DIV>RAID segment size is 256K, stripe size is 1MB.</DIV>
<DIV> </DIV>
<DIV>4 clients connect to the OSS servers by 10GigEthernet.</DIV>
<DIV>Network performace between servers and clients is normal. 1GB/s throughput
is obtained in netperf and lnet self test.</DIV>
<DIV> </DIV>
<DIV>4 clients are running iozone to write 4G files.</DIV>
<DIV> </DIV>
<DIV>prompt$ RSH=ssh iozone -i 0 -I -M -C -w -r 1m -t 4 -s 4g -+m
/root/iozone_clients</DIV>
<DIV>....</DIV>
<DIV> O_DIRECT feature enabled</DIV>
<DIV> </DIV>
<DIV> Machine = Linux
cluster.iseis.cuhk.edu.hk 2.6.32-279.14.1.el6.x86_64 #1 SMP Tu Setting
no_unlink</DIV>
<DIV> Setting no_unlink</DIV>
<DIV> Record Size 1024 KB</DIV>
<DIV> File size set to 4194304
KB</DIV>
<DIV> Network distribution mode
enabled.</DIV>
<DIV> Command line used: iozone -i 0
-I -M -C -w -r 1m -t 4 -s 4g -+m /root/iozone_clients</DIV>
<DIV> Output is in Kbytes/sec</DIV>
<DIV> Time Resolution = 0.000001
seconds.</DIV>
<DIV> Processor cache size set to 1024
Kbytes.</DIV>
<DIV> Processor cache line size set to
32 bytes.</DIV>
<DIV> File stride size set to 17 *
record size.</DIV>
<DIV> Throughput test with 4
processes</DIV>
<DIV> Each process writes a 4194304
Kbyte file in 1024 Kbyte records</DIV>
<DIV> </DIV>
<DIV> Test running:</DIV>
<DIV> Children see throughput
for 4 initial writers = 238195.08 KB/sec</DIV>
<DIV> Min throughput per
process
= 58905.94 KB/sec</DIV>
<DIV> Max throughput per
process
= 60983.77 KB/sec</DIV>
<DIV> Avg throughput per
process
= 59548.77 KB/sec</DIV>
<DIV> Min
xfer
= 4051968.00 KB</DIV>
<DIV> Child[0] xfer count = 4194304.00
KB, Throughput = 60983.77 KB/sec</DIV>
<DIV> Child[1] xfer count = 4066304.00
KB, Throughput = 59111.19 KB/sec</DIV>
<DIV> Child[2] xfer count = 4071424.00
KB, Throughput = 59194.18 KB/sec</DIV>
<DIV> Child[3] xfer count = 4051968.00
KB, Throughput = 58905.94 KB/sec</DIV>
<DIV> </DIV>
<DIV>Aggregate throughout 238 MB/s is obtained. </DIV>
<DIV> </DIV>
<DIV>There is only about 30MB/s throughput (238 / 8) for each OST. (seen at Dell
Storage Maanger Performance Monitor)</DIV>
<DIV>I think it is considered poor, as one OST has 4 effective disks within a
RAID-5 volume.</DIV>
<DIV> </DIV>
<DIV>Why the direct io performance is so slow? Thanks in advance.</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV></DIV></DIV></BODY></HTML>