[Lustre-discuss] small cluster

[Andreas Dilger]

On Sep 05, 2008  07:59 -0600, Kevin Van Maren wrote:
> The biggest problem with using Lustre in this environment is that the 
> Lustre _servers_ (those providing the disk space) cannot _use_ the 
> filesystem (cannot have a client and a server on same machine).

Well, it depends on how much memory pressure there is.  It definitely
works with client-on-OST (I run my home system that way, and lots of
Lustre developers do functional tests in that config) but it has the
chance of deadlock if there are memory-hungry apps running on the
same OSS.

This could be avoided by forcing all writes to be to remote OSS nodes,
possibly at the cost of some performance, though network IO on a 1GigE
is faster than to a single disk if you have a good switch.

The other (more major, IMHO) issue is that if the client/OSS node
crashes, it takes its disk with it, and now other clients cannot access
the data there.  Similarly, without RAID of the disk, any disk loss
would mean loss of 1/36 of the filesystem.

This could be avoided by having RAID1 of the disks using DRBD to a
remote node that is configured as the OST failover.  There is a wiki
page on the lustre wiki about using DRBD.

Please keep the list informed (and wiki updated) if you decide to
use this configuration.

> Paolo Supino wrote:
> >   I'm new to the world of HPC and cluster :-( and I was given the task 
> > of setting up a new (small) HPC cluster based on 36 Sun X2200 M2 
> > systems. Each system as 1 136GB SAS HD. I've installed CentOS 5.2 as 
> > the base OS on all the systems and through partitinoing managed to 
> > have a unused partition size 65GB (and with 36 nodes I get +2TB of 
> > distributed storage). Tthe rest of the HD is as follows: 4xRAM = 64GB 
> > swap and ~10GB for the OS itself. I want ot put a Clustered FS on the 
> > unused partitions and from looking around seems that Lustre is king of 
> > the hill ...
> >   The question I have is: Will I gain anything from using Lustre in 
> > such a tight environemnt or am I going to degrade the cluster's 
> > performnce that it simply not worth the effort?

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.