[Lustre-discuss] clarification on mkfs.lustre options

[Andreas Dilger]

On 2010-07-30, at 00:42, Sebastian Gutierrez wrote:
> The enclosure I am using supports 15 disks.   We have initially ordered 6 disks + hostpare.  I think my recommendation is going to be a 6 disk raid 6.  Which will give us a 4 data disk raid set plus 2 parity disks.  Later we will have the option to create another 6 disk raid6 or expand the current raid set to a 10 disk raid 6 then move the journal to a 4 disk raid 1/0 and have 1 disk hot spare.   
> 
> The current Raid 6 will have a 128k chunksize (lustre terminology) 
> 
> this gives us:
> 
> <stripe_width> = <chunksize> * (<disks> - <parity_disks>) <=1MB
> 512K <= 128k*4
> 
> <chunksize> <= 1024kB/4; either 256k, 128k, 64k 
> 256k = <	= 1024k/4k
> 
> <chunk_blocks> = <chunksize(decided above)> / 4k
> 32 = 256/4 
> 
> 128k = 512k / 4k
> <stripe_width_blocks> = <stripe_width> / 4k   
> 
> Therefore 
> My mkfs options should be 
> --mkfsoptions="-E stripe=128 -E stride=32" /dev/sdb

If you are planning on expanding this at the RAID6 level to be an 8+2 configuration, you should specify "-E stripe=256,stride=64".  Note that you cannot specify mulitple separate "-E" options to mke2fs, it would only use the last one specified.

> purchase 6 more disks
> This will allow for a new raid 6 that is aligned the same way with 3 hotspares.

> or (I need clarification if this understanding is correct)
> 
> purchase 10 more disks
> Expand the current raid 6 to a larger 10 disk/raid 6 with a 4 disk raid 1/0 for a external journal plus a hot spare.  

Using a 4-disk RAID-10 external journal is unlikely to give you any extra performance, since journal IO is nearly sequential (though sometimes small block writes if there are few clients and you are not using async journal).

Also, 16TB LUN support is only available with ext4, so if you have 2TB drives you need to make sure to download the right ldiskfs package.

Depending on the hardware options on your RAID, it may be that you need a separate hot spare for each LUN, in which case having 2 hot spares makes sense.  Otherwise, you can probably use only 13 or 14 drives.

> Is my understanding of the documentation accurate? 
> Do both of these options seem like potential upgrade options?

Either of them seem reasonable.

If the hardware allows in-place RAID reshaping then it is possible.  I'd always recommend to make a backup before doing this, because one never knows what might happen if this operation is interrupted for some reason.

Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.