[Lustre-discuss] 64 nodes cluster storage suggestions?

[Cristi Magherusan]

Hi,

I'd like to setup a Lustre filesystem on a 64-nodes cluster with Gbit
connection (maybe 2Gbit w/port aggregation), and I need some suggestions
on what architecture to choose.

These are our needs:
We need to have filesystem mounted on all the nodes, with good I/O
throughput, and we would prefer to have the nodes configuration as
uniform as possible so it can be administred centrally using Puppet.
The filesystem doesn't need be too large, at least 100GB would do but we
appreciate any extra space we get.

1TB would be more than enough, 

One of the nodes that need to access the storage will run virtualized in
a Xen VM, along with some other nodes that won't access the storage
cluster. The Xen hosts will be redundant, using DRBD and heartbeat or
pacemaker.
Each node has one 146G SAS drive, the Xen hosts have 6 of those in 
RAID1+0.

We would like our data to be redundant if possible, and to support
storage node reinstallations without losing data (RAID-like redundancy).

I don't know if Lustre is the boot tool for this job, but I have a few
scenarios in my mind, one of them not including Lustre.

The 64 nodes will be running CentOS and CPU-intensive applications.
There will be a moderate to low disk I/O on the shared directory and
moderate to low network I/O.


1) 6 dedicated Lustre nodes: 2 dedicated MDS and 4 dedicated OSS nodes,
mirrored in pairs using Linux HA.

2) a MDS node running as a Xen VM, and all the other 64 nodes behaving
both as OSS nodes and clients. Does Lustre provide RAID features yet so
we don't loose data in case one node crashes or gets reinstalled?

3) 6 nodes in a GFS2 cluster with data available using NFS, mirrored in
pairs using Linux HA.

What options do you recommend? Or if you have anything else in mind..

In case we choose the second option, Would a high CPU usage and moderate
disk and network I/O on these nodes affect Lustre I/O performance in a
measurable way so it would be worse than the others?

Also, do you know of any Yum repository that could be used to keep the
Lustre software up to date? I couldn't find any, and the manual download
and install procedure really sucks. Other Sun software does it, so why
not Lustre too?

Thanks in advance,
Cristi

-- 
Cristi Măgherușan, alumnus System/Network Engineer
Technical University of Cluj-Napoca, Romania
http://cc.utcluj.ro  +40264 401247

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100509/dae8d8eb/attachment.pgp>