[Lustre-discuss] is lustre for us?

[Peter Grandi]

>> At my university we have a big storage problem. Currently we
>> have 25 Linux servers each equiped with 3 TB of data. Totaling
>> to 75TB of storage. We use each server as a NFS server and
>> have clients mount up the filesystem. They are all connected
>> by a GigE network.

>> The problem we are facing is performance and scalibility.
>> When a users access data, many times they access the same
>> server. This is becoming a huge performance problem. The
>> files sizes range from 8GB to 70GB.

The impression I get from the above is that it is not clear what
kind of performance and scalability problems you have. It is far
from clear what is the status of any of the following "details":

* Storage system structure and speed.
* Network performance tuning.
* How many users tend to access the same fileserver at once.
* Speed of the clients.
* Performance of the network switches/routers.
* ....

You have large scale storage issues, and perhaps the way they
are expressed above is less clear than it should be.

For example, in the best of cases you can get around 100MB/s
reads out of a single NFS server to a single NFS (or Lustre)
client, and 3GB will take 30s and 70GB over 10 minutes just to
transfer at this speed, irrespective of filesystem performance
and scalability. In a similar environment I have been pushing
10G switches/routers and 10G cards for servers and even for
clients (and too bad that 10GBASE-T products are still a dream).

>> Would Lustre solve our performance and management
>> problem?

Which management problem? Above you mention only performance.
You also don't mention any reliability/availalability goals,
or backup.

Also, the performance is not at all clear. Single client speed? 
Aggregate speed? Server load? ...

>> How would lustre scale?

To what? A petabyte? A thousand clients? A 10G network? ...

>> I have been reading the quickstart guide, is it easy to
>> implement?

Implementing a high performance, scalable storage pool for
3-70GB files starting at 75TB is not a trivial task and requires
a lot of clarity and experience irrespective of the network
filesystem used...

BTW, I have written some of my ineffable wisdom :-) on the
subject here:

  http://WWW.sabi.co.UK/blog/0804apr.html#080417b

However as AndreasD says, Lustre overall seems pretty good at
medium-large scientific data storage systems, but the vague
nature of your concerns makes a bit difficult to evaluate them.

Perhaps those that have only a vague idea of their goals should
read a few articles detailing users experiences with Lustre,
there are quite a few pointers on the Lustre Wiki, and in
various years of the HEPiX workshop.

> hello andreas, i have totally different enviroment than mag's
> we have many small file which less than 1Mb and i didn't
> (actually i can't) know exactly how many it is.  it seems more
> than 10 millions of files..  in our situation, is lustre
> suitable??

10^6 each less than 10^6 large. Looks like 1TB, for which an
external FW800 or eSATA drive with a small-file friendly
filesystem seems a good match. Perhaps JFS or ReiserFS; but
'fsck' times are going however to be potentially long (days to
weeks if the filesystems gets damaged). Lustre has the great
advantage that it can 'fsck' its OSTs in parallel, so perhaps
dividing that 1TB into 50-100G OSTs may be of benefit as each
filesystem would end up having a lot less files.

Note however that Lustre in effect is based on the idea of
clients mounting individual files (or file slices) instead of
filetrees (with the MDS in the role of the automount directory
server), so each file open involves at least two network round
trips, which means that latency on metadata-intensive loads will
be less awesome.

>From the very vague goals you give above it is not at all clear
to me whether any of this matters or not.

As to the several and often underrated aspects on which a file
system design can be evaluated, here is more of my incredible
insight :-):

  http://WWW.sabi.co.UK/blog/0804apr.html#080415