[Lustre-discuss] tuning for performance

[Andreas Dilger]

On 2012-04-03, at 7:04 AM, Francois Chassaing wrote:
> Hello list,
> I'm trying to tune my lustreFS so that it'll perform better, because I'm sure I have a lot of low-level stuff that needs to be optimzed.
> My (poor) choice was to have a pair of MDS backed by a (supposely fast Equallogic) iSCSI array, I also have three OSSes serving one OST each.
> Those OST are a set of 6 NL SAS drives on RAID10 behind a Perc6/i (those Dell cards I'm sure a lot know about, this one has BBU & 256Mb RAM). system for those OST are on a separate mirrored pair of small disks
> lnet sits on DDR IB (mellanox memory-less cards). Lustre version is 1.8.7-wc1
> 
> I want to use more sensible settings for the RAID volume :
> First of all I would change the chunk/stripe/whatever-name-you-like size for the volume to try to align with 1Mb/# of drives.
> If I had four drives RAID10, i would have chosen 512k, since I've 6, I'm more leaning towards 256 (means 1.33 writes per disk for 1Mb 'block' which would most probably perform better than the 5.33 writes per disk using the 64k default setting).

What you really should be looking at is to do component-level analysis using the Lustre iokit to test the performance of each part of your IO stack.  First the RAID level, next the OST level, and finally the client performance.

> Does that makes sense to any of you, or is this pointless ?
> Also, given the Raid card, do you think that it will be better to really align to 1Mb by using a 5-drive RAID5 (so sparing one drive) or a RAID6 volume ?
> then I could use the 256k stripe size AND be aligned, but I'll have to pay the cost for parity-computation and write(s)...

I suspect you will get better performance and reliability with RAID-6 4+2 than with RAID-10 3+3, but this is easily tested with sgpdd-survey from the iokit.

Note also that most deployments have multiple OSTs per OSS, since 6 drives is typically not enough to saturate a decent server, and you get more performance with less overhead.  You could probably put 4 OSTs into each OSS reasonably.  At least it is something to consider for the future if you expand your filesystem.


Cheers, Andreas
--
Andreas Dilger                       Whamcloud, Inc.
Principal Lustre Engineer            http://www.whamcloud.com/