[Lustre-discuss] Hardware Question

[Andreas Dilger]

On Oct 05, 2007  11:02 -0600, Daniel Leaberry wrote:
> Aaron Knister wrote:
> > I'm planning to hang 58 terabytes off of a PowerEdge 1950 with 4 CPUS  
> > and 8 gigs of memory. My MDS is a dual core Opteron with a 250gig  
> > raid1 metadata volume and 2GB of ram. Do you think this hardware  
> > configuration is sane?
>
> We have about 85TB of disk (in 24 luns) hanging off 4 PE2950's with 
> those same specs. They are set up in failover pairs (each handles 6 
> luns) but I can't run too long on a single machine before it starts 
> thrashing when it takes over the other nodes 6 luns.

If you have 12 OSTs on a single node, that means up to 12 * 400MB = 4.8GB
of RAM pinned just by the ext3 journal.  Either you need a lot more RAM
than this (8TB for example), or you need to shrink the journal size like
128MB (tune2fs to remove then re-add it).  Using 128MB should be fine
unless you have many hundreds of clients doing concurrent IO.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.