[Lustre-discuss] software raid

[Mark Hahn]

choosing sw vs hw raid depends entirely on your systems, pocketbook, taste.
I think there are two edge cases which are pretty unambiguous:

 	- modern systems have obscene CPU power and memory bandwidth,
 	at least compared to disks.  and even compared to the embedded
 	cpu in raid cards.  this means that software raid is very fast
 	and attractive for at least moderate numbers of disks.  because
 	disks are so incredibly cheap, it's almost a shame not to use
 	the 6+ sata ports present on every motherboard, for instance.

 	- if you need to minimize CPU and memory-bandwidth overheads,
 	or address very large numbers of disks, you want as much hardware
 	assist as you can get even though it's expensive and wimpy.
 	having 100 15k rpm SAS disks as JBOD under SW raid would make
 	little sense, since the disks, expanders, backplanes and controllers
 	overwhelm the cost savings.

I think it boils down to your personal weighting of factors in TCO.

"classic" best practices, for instance, emphasizes device reliability
to maintain extreme uptime and minimize admin monkeywork.  that's fine,
but it's completely opposite to the less ideological, more market-reality
driven approach that recognizes disks cost $30/TB and dropping, and that 
with appropriate use of redundancy, mass-market hardware can still achive 
however many nines you set your heart on.

it is convenient that a 2u node supporting 6-12 disks can be done with 
the free/builtin controller with SW raid and delivers bandwidth that matches 
relevant network interfaces (10G, IB).  I like the fact that a single unit
like that has no "extra" firmware to maintain, or over-smart controllers
to go bonkers.  IPMI power control includes the disks.  SMART works directly.
and in a pinch, the content can be brought online via any old PC.

I've used MD since it was new in the kernel, and never had problems with it.

regards, mark hahn