[Lustre-discuss] Limit of OSTs per OSS?

Jim Garlick garlick at llnl.gov
Wed Aug 19 10:11:31 PDT 2009


The journal pages are effectively pinned I think, so with 400MB journals
on each OST, count on losing 400MB of memory at least.

Here is some anecdotal data from our environment:

We seem to do OK with 4G nodes having 3 OST's each, including sometimes
running with 6 during failover.  On the other hand 2G nodes with 2 OST's
each would run out of memory when running with 4 during failover.
We had to reduce the journal size from 400MB to 200MB for failover to work
reliably on the 2G nodes.  In summary, with 400MB journals:
 - 0.66G per OST works
 - 0.50G per OST doesn't

That's with servers running Lustre 1.6.6, RHEL 5.3, x86_64 arch, and socklnd.

Jim


On Wed, Aug 19, 2009 at 02:40:35PM +0200, Arne Wiebalck wrote:
> 
> OK, I will keep that in mind.
> 
> Thanks,
>  Arne
> 
> Sébastien Buisson wrote:
> >Hi,
> >
> >Just a small feedback from our own experience.
> >I agree with Brian about the fact that there is no strong limit on the 
> >number of OSTs per OSS in the Lustre code. But one should really take 
> >into account the available memory on OSSes when defining the number of 
> >OSTs per OSS (and so the size of each OST). If you do not have 1GB or 
> >1.2 GB of memory per OST on your OSSes, you will run into serious 
> >trouble with "out of memory" messages.
> >
> >For instance, if you want 8 OSTs per OSS, your OSSes should have at 
> >least 10GB of RAM.
> >
> >Unfortunately we experienced those "out of memory" problems, so I advise 
> >you to read Lustre Operations Manual chapter 33.12 "OSS RAM Size for a 
> >Single OST".
> >
> >Cheers,
> >Sebastien.
> >
> >
> >Arne Wiebalck a écrit :
> >>Brian J. Murrell wrote:
> >>>On Wed, 2009-08-19 at 11:45 +0200, Arne Wiebalck wrote:
> >>>>is there a limit on the number of OSTs per OSS?
> >>>I have never run across one so there is likely no *practical* limit.  I
> >>>could very well be corrected on this though.
> >>>
> >>>Unless you are making lots and lots of small OSTs -- which is not
> >>>usually beneficial anyway -- typically, you will run into resource
> >>>limitations (memory, bus bandwidth, etc.) on an OSS before you hit a
> >>>limit on the number of OSTs.
> >>I just thought I remembered there was something like 8 OSTs per OSS,
> >>but apparently I was wrong.
> >>
> >>>Just how many OSTs are you considering, and how big will they be?
> >>My OSSs will have 10 OSTs with 1TB each.
> >>
> >>Thanks for the quick reply,
> >> Arne
> >>
> >>
> >>------------------------------------------------------------------------
> >>
> >>_______________________________________________
> >>Lustre-discuss mailing list
> >>Lustre-discuss at lists.lustre.org
> >>http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 



> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://*lists.lustre.org/mailman/listinfo/lustre-discuss




More information about the lustre-discuss mailing list