[Lustre-discuss] Performance Expectations of Lustre

[Daniel Leaberry]

Nick Jennings wrote:
> Hello (and a special hello to all my ex-co-workers from the CFS days :)
> 
>   The company where I work now has grown fast in the past year and we 
> suddenly find ourselves in need of a lot of storage. For 5 years the 
> company ran on a 60gig server, last year we got a 1TB RAID that is now 
> almost full. In 1-2 years we could easily be using 10-15TB of storage.
> 
>   Instead of just adding another 1TB server, I need to plan for a more 
> scalable solution. Immediately Lustre came to mind, but I'm wondering 
> about the performance. Basically our company does niche web-hosting for 
> "Creative Professionals" so we need fast access to the data in order to 
> have snappy web services for our clients. Typically these are smaller 
> files (2MB pictures, 50MB videos, .swf files, etc.).

<snip>

I'm going to send you down a different direction based on my experience.

We run a 90TB lustre array on datadirect storage and while it works well 
we picked a different design for our website storage. We did this 
because although lustre works well it didn't provide the robustness we 
needed with a website. This is no slight to the lustre team, just what I 
have observed over the last 2 years of lustre in production. 
Specifically failover takes time and locks the filesystem.

For our web storage we use mogilefs. We serve images (about 50 million 
and growing) and have 150TB of storage. It's never been a problem, it's 
written in perl and easy to follow the code, numerous other websites use 
it and it works. The only downside is mogilefs uses an api and there is 
no direct filesystem access. This is managable in a web infrastructure 
though. The benefits of lustre are speed and being able to take a 
pounding from clients. Neither is necessary in a web environment where 
if you're lucky you'll push 100 mbit/sec.

Again, I have large instances of both lustre and mogilefs. For a 4-5 
nines website with people pointing fingers at me if it breaks I would go 
with mogile. For a backend production system that needs to push 500+ 
MB/sec from 150 processing nodes, go with lustre.

Daniel