[Lustre-discuss] Not sure how we should configure our RAID arrays (HW limitation)

[Kevin Van Maren]

On May 4, 2012, at 2:53 PM, Frank Riley wrote:

> Hello,
> 
> We are using Nexsan E18s for our storage systems, and we are in the process of setting them up for Lustre. Each E18 has 18 disks total (max'ed out) in them. According to the Lustre docs, I want to have a stripe width of 1MB. Unfortunately, these E18s have a max stripe size of 128K. As I see it, for RAID6 this leaves us two options:
> 
> 1)  One array 16+2 with a stripe size of 64K for a stripe width of 1MB. I'm hesitant with this option because of the increased chance that we could have more than 2 disks fail.
> 
> 2) Do two arrays 7+2 with a stripe size of 128K for a stripe width of 896K. I'd then modify the max_pages_per_rpc tunable to match the 896K. I'm not sure what to do with the flex_bg filesystem option since it has to be a power of 2.

Note that you need to set the stripe size to match 896K, as otherwise you will send 896 and 128KB to each OST.  Additional tuning of the mkfs options is also necessary so that the file system understands the layout (see -E in the Lustre manual), as otherwise all the block allocations will start mid-stripe.  This is not ideal for applications that expect Po2 sizes to be optimal.

> 
> What is the better option here? Or is there an option I'm missing? I've pretty much ruled out RAID5 arrays at 8+1 due to data loss risk, and RAID1+0 wastes too much disk for our use.

8+1 is the best option from a Lustre performance standpoint.  You should get better performance from 2 7+2 arrays than with a 16+2 simply because you can have twice the number of independent IOs.

How about doing 3 4+2 RAIDs?  12 usable disks, instead of 14 or 16, but still better than 8 with RAID1.  Doing 4*128KB, resulting in 2 full-stripe writes for each 1MB IO is not that bad.

Kevin


This e-mail message, its contents and any attachments to it are confidential to the intended recipient, and may contain information that is privileged and/or exempt from disclosure under applicable law. If you are not the intended recipient, please immediately notify the sender and destroy the original e-mail message and any attachments (and any copies that may have been made) from your system or otherwise. Any unauthorized use, copying, disclosure or distribution of this information is strictly prohibited.  Email addresses that end with a ?-c? identify the sender as a Fusion-io contractor.