[Lustre-discuss] LUSTRE Bad Read Performance

[Sergius ssi. Siczek - sysGen GmbH]

Hello,

I've got a strange problem with lustre read perforamnce.

Following setup:

8 Core (100GB RAM) combined MGS/OSS with two OST targets server, with Infiniband

2x RAID6  (11x 1 TB HDD -> 8TB usable on each RAID). (/dev/sdc and /dev/sdd)

/dev/sdc1 = 1024GB
/dev/sdd1 = 1024GB

1x small HDD for journal and MDS. (/dev/sdb)

/dev/sdb1 = 10 GB journal
/dev/sdb2 = 10GB journal
/dev/sdb3 = 50 GB MDS

Procedure:

mkfs.lustre --reformat --fsname lustre --mdt --mgs /dev/sdb3

mkfs.lustre --fsname=lustre --reformat --ost --mgsnode 192.168.0.1 at o2ib --mkfsoptions="-E stride=256 -E stripe-width=9 -J device=/dev/sdb1" /dev/sdc1
mkfs.lustre --fsname=lustre --reformat --ost --mgsnode 192.168.0.1 at o2ib --mkfsoptions="-E stride=256 -E stripe-width=9 -J device=/dev/sdb2" /dev/sdd1


Client machine:

24 core (50 GB RAM) with infiniband.

Mount :
mount -t lustre 192.168.0.1 at o2ib:/lustre /mnt/client/<mailto:192.168.0.1 at o2ib:/lustre%20/mnt/client/>

After that I ran some simple throughput tests. Write performance (5 threads) is about 1072 MB/s (which is great), but read performance is very bad ~ 510 MB/s.

What could be wrong about this setup? Why write performance is so good, but read so bad?

Of course I tested the hardware itself, made a filesystem on the raids (without lustre) and ran the same tests, ~1,3GB/s write 1,7GB/s read (800 GB file)

The infiniband is also okay, used the same cards/cable for some HPC tests just hours before, everything good. Nevertheless used the server as a client two (just to bypass the infiniband side as a potential failure). Nearly the same results (write good, read bad). It's like there is a wall at ~510 MB/s read.


Any ideas or suggestions ? Its really urgent. Thank you a lot!


2.6.27.39+lustre1.8.3+0.credativ.lenny.4
Lustre version: 1.8.3-20100410022943-PRISTINE-2.6.27.39+lustre1.8.3+0.credativ.lenny.4









sysGen GmbH

Sergius Siczek

Am Hallacker 48 · 28327 Bremen

Tel (0421) 4 09 66-32 · Fax (0421) 4 09 66-33

ssiczek at sysgen.de<mailto:ssiczek at sysgen.de> · www.sysgen.de<http://www.sysgen.de/>

Geschäftsführung: Gabriele Nikisch, Anke Heinßen · Eingetragen beim Amtsgericht Walsrode · HRB 121943

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100811/c2d41432/attachment.htm>