[Lustre-discuss] Out of Memory on MDS

[Roger Spellman]

Andreas,

Thanks.  I've opened a bug,
https://bugzilla.lustre.org/show_bug.cgi?id=19950

In that bug I show that the number of ldlm_locks exceeds the limit
reported by lctl get_param ldlm.namespaces.*.pool.limit.

Do you agree that if we write to lru_size on every client, then that
will set a limit on ldlm_locks on the servers?

Do you know of a way to limit ldiskfs_inode_cache?

Thanks.

-Roger

> -----Original Message-----
> From: Andreas.Dilger at sun.com [mailto:Andreas.Dilger at sun.com] On Behalf
Of
> Andreas Dilger
> Sent: Tuesday, June 23, 2009 6:07 PM
> To: Roger Spellman
> Cc: Cliff.White at sun.com; lustre-discuss at lists.lustre.org
> Subject: Re: [Lustre-discuss] Out of Memory on MDS
> 
> On Jun 23, 2009  16:50 -0400, Roger Spellman wrote:
> > The servers are 1.6.7.1.  The clients are a mix of 1.6.7.1 and
1.6.6.
> >
> > Is lru_size an upper limit on the number of entries?
> >
> > Also, lru_max_age does not seem to be working.  I set it to 10
seconds,
> > and it did not clean anything up.
> 
> This is worth filing a bug on, if it isn't working.  The lock LRU size
> should be limited by the size of the RAM.  The upper limit on the
number
> of locks being granted can be gotten via:
> 
> 	lctl get_param ldlm.namespaces.*.pool.limit
> 
> The default limit should be 50 locks per 1MB of RAM.  In your case,
> 4GB is 4096MB, so the LRU limit should be 50 * 4096 = 204800 locks.
> 
> > > Roger Spellman wrote:
> > > > I have an MDS that is crashing with out-of-memory.
> > > >
> > > > Prior to the crash, I started collecting /proc/slabinfo.  I see
that
> > > > ldlm_locks is up to 4,500,000, and each one is 512 bytes, for a
> > > > total of 2.2GB, which is more than half my RAM.
> > > >
> > > > Is there a way to limit this?
> > >
> > > You don't mention the version of Lustre - lru_size might have an
> > > impact, I am not certain. I believe it is the only lock tuneable
> > > of note. (and is auto-sized in recent Lustre)
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.