[Lustre-discuss] Performance Expectations of Lustre

[Nick Jennings]

Hello (and a special hello to all my ex-co-workers from the CFS days :)

  The company where I work now has grown fast in the past year and we 
suddenly find ourselves in need of a lot of storage. For 5 years the 
company ran on a 60gig server, last year we got a 1TB RAID that is now 
almost full. In 1-2 years we could easily be using 10-15TB of storage.

  Instead of just adding another 1TB server, I need to plan for a more 
scalable solution. Immediately Lustre came to mind, but I'm wondering 
about the performance. Basically our company does niche web-hosting for 
"Creative Professionals" so we need fast access to the data in order to 
have snappy web services for our clients. Typically these are smaller 
files (2MB pictures, 50MB videos, .swf files, etc.).

  Also I'm wondering about the best way set this up in terms of speed 
and ease of growth. I want the web-servers and the storage pool to be 
independent of each other. So I can add web-servers as the web traffic 
increases, and add more storage ass our storage needs grow. We have the 
option of an MD3000 or MD3000i for back-end storage.

  I was thinking initially we could start with 2 servers, both attached 
to the storage array. setup as OSS' and functioning as (load balanced) 
web-servers as well. In the future I could separate this out so that we 
have the web-servers on the "front line" mounting the data from the OSS' 
which will be on a private (gigE) network.

  Now, it's been years since I've played with Lustre, I'm sure some 
stuff will come back to me as I start using it again, other things I'll 
probably have to re-learn. I wanted to get some input from the Lustre 
community on whether or not this seems like a reasonable use for Lustre? 
Are there alternatives out there which might fit my needs more? 
(specifically speed and a shared storage pool). Also, what kind of 
performance can I expect, am I out of touch to expect something similar 
to a directly attached RAID array?

  I appreciate any and all feedback, suggestions, comments etc.

Thanks,
- Nick

--
Nick Jennings
Senior Programmer & Systems Administrator
Creative Motion Design
nick at creativemotiondesign.com