[Lustre-discuss] mkfs options/tuning for RAID based OSTs

Edward Walter ewalter at cs.cmu.edu
Tue Oct 19 18:00:02 PDT 2010


Hi Dennis,

That seems to validate how I'm interpreting the parameters. We have 10 data disks and 2 parity disks per array so it looks like we need to be at 64 KB or less.

I'm guessing I'll just need to run some tests to see how performance changes as I adjust the segment size. 

Thanks,

-Ed


On  Oct 19, 2010, at 5:56 PM, Dennis Nelson <dnelson at sgi.com> wrote:

> Segment size should be 128.
> 
> 128 KB * 8 data drives = 1 MB.
> 
> 
> On 10/19/10 3:42 PM, "Edward Walter" <ewalter at cs.cmu.edu> wrote:
> 
>> Hello All,
>> 
>> We're doing a fresh Lustre 1.8.4 install using Sun StorageTek 2540
>> arrays for our OST targets.  We've configured these as RAID6 with no
>> spares which means we have the equivalent of 10 data disks and 2 parity
>> disks in play on each OST.
>> 
>> We configured the "Segment Size" on these arrays at 512 KB.  I believe
>> this is equivalent to the "chunk size" in the Lustre operations manual
>> (section 10.1.1).  Based on the formulae in the manual: in order to have
>> my stripe width fall below 1MB; I need to reconfigure my "Segment Size"
>> like this:
>> 
>> Segment Size <= 1024KB/(12-2) = 102.4 KB
>> so 16KB, 32KB or 64KB are optimal values
>> Does this seem right?
>> 
>> Do I really need to do this (reinitialize the arrays/volumes) to get my
>> Segment Size below 1MB?  What impact will/won't this have on performance?
>> 
>> When I format the OST filesystem; I need to provide options for both
>> stripe and stride.  The manual indicates that the units for these values
>> are 4096-byte (4KB) blocks.  Given that, I should use something like:
>> 
>> -E stride= (one of)
>>   16KB/4KB = 4
>>   32KB/4KB = 8
>>   64KB/4KB = 16
>> 
>> stripe= (one of)
>>   16KB*10/4KB = 40
>>   32KB*10/4KB = 80
>>   64KB*10/4KB = 160
>> 
>> so for example I would issue the following:
>> mkfs.lustre --mountfsoptions="stripe=160" --mkfsoptions="-E stride=16 -m
>> 1" ...
>> 
>> Is it better for to opt for the higher values or lower values here?
>> 
>> Also, does anyone have recommendations for "aligning" the filesystem so
>> that the fs blocks align with the RAID chunks?  We've done things like
>> this for SSD drives.  We'd normally give Lustre the entire RAID device
>> (without partitions) so this hasn't been an issue in the past.  For this
>> installation though; we're creating multiple volumes (for size/space
>> reasons) so partitioning is a necessary evil now.
>> 
>> Thanks for any feedback!
>> 
>> -Ed Walter
>> Carnegie Mellon University
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
> 




More information about the lustre-discuss mailing list