[Lustre-discuss] New to lustre - Help setting up Lustre

[Jason Rappleye]

Hi,

On Dec 23, 2009, at 3:08 PM, Brian J. Murrell wrote:

>> Should it be avoided or there is not much of a performance hit if we
>> use LVM.
>
> I've never been convinced that there is a significant performance hit.
> I have personally been using LVM as my exclusive block device
> provisioning tool for a dozen years or more.  YMMV however.  If you  
> want
> to have scientific results before you start, simply benchmark a raw
> device and then create an LV and benchmark it.  Report here your
> differences just for interest's sake.

We just put one large (860TB), and are about to put two more 430TB  
filesystems into production using LVM. Backend disk for the OSTs is  
7200RPM 1TB SATA on DDN 9900 controllers, and for the MDT is 15000RPM  
400GB SAS on an SGI IS200 in a RAID 10. Performance *without*  
snapshots seemed to be fine. However, with snapshots in play on the  
OSTs, performance is about 1/10th of what we normally see, due to a  
mix of small (8KB-64KB) reads and writes when performing copy-on-write  
of blocks written when a snapshot is active. IIRC, the 8K I/Os were  
with the default snapshot chunk size, and the 64KB I/Os were with a  
512KB chunk size.

We decided to do snapshots for two reasons - live backups of the MDT,  
and to perform live fscks on the MDT and OSTs if we suspect data  
corruption. For those limited uses, we're willing to occasionally take  
the performance hit. I wouldn't do this to regularly back up the OSTs,  
though, which is not something we do with these large scratch  
filesystems.

YMMV. As Brian said, nothing beats doing the testing yourself to  
understand how your hardware behaves, for both benchmarks and, more  
importantly, the applications you run.

j

--
Jason Rappleye
System Administrator
NASA Advanced Supercomputing Division
NASA Ames Research Center
Moffett Field, CA 94035
jason.rappleye at nasa.gov