[Lustre-discuss] bursty instead of even write performance

Jure Pečar pegasus at nerv.eu.org
Thu Jul 14 07:29:50 PDT 2011


Hello all,

This is my first meet with decently sized lustre storage. 

I have a system to play with here, consisting of 10 HP DL320 with 12x750GB drives each, attached to SmartArray controllers. Machines are few years old and possibly underpowered, equipped with xeon 3060 and 2GB of RAM each. Each OSS is capable of about 120MB/s write and 240MB/s read. MDS is equally weak by today's standards, a DL140 with xeon 5110 and only a gig of RAM. OSSs are connected via bonded GigE to a switch that has 10GigE connection to other switches, through which clients connect.

I managed to get lustre 2.0.58 to show signs of life on this system, using SL6 and about a week of trial-and-error compiling. It was sporiadic to mount and actually start working, but about every third attempt produced a working lustre fs that clients could mount.
Since mngmt tools were mostly useless, I decided to go back to 1.8 and chose 1.8.6-wc, which worked out-of-the-box on CentOS5. Then I went and did some tests with performance and here is my first question.

My clients are two racks of IBM machines (84 of them). I'm getting about 700MB/s write and 1.5GB/s read combined, which feels great for my first lustre. However I'm noticing some strange patterns when looking at ganglia graphs.

graph.gif shows combined write speed when each node was simply writing a large file using dd. Performance slowly drops as the disks get full, something I'm used to. But nodes.gif shows write speed of each node, which shows things I'm not used to - long periods of no activity, then sudden bursts, then again nothing. I would assume each client to have a steady and even write activity, if only at 8-9 MB/s, but that's not what I see.

So, my question: Is what I'm observing an expected situation? Or am I right that I should be seeing more balanced write activity from each node?

Since all of the lustre settings are at their defaults, what should I look into to see if I can tune anything? 

Thanks for pointers,


-- 

Jure Pečar
http://jure.pecar.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graph.gif
Type: image/gif
Size: 122005 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20110714/be906df7/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nodes.gif
Type: image/gif
Size: 33979 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20110714/be906df7/attachment-0001.gif>


More information about the lustre-discuss mailing list