[Lustre-discuss] Lustre Client - Memory Issue

[Dmitry Zogin]

Hello Jagga,

I checked the data, and indeed this does not look like a lustre memory 
leak, rather than a slab fragmentation, which assumes there might be a 
kernel issue here. From the slabinfo (I only keep three first columns here):

name            <active_objs> <num_objs>
ll_obdo_cache          0 452282156    208

means that there are no active objects, but the memory pages are not 
released back from slab allocator to the free pool (the num value is 
huge). That looks like a slab fragmentation - you can get more 
description at
http://kerneltrap.org/Linux/Slab_Defragmentation

Checking your mails, I wonder if this only happens on clients which 
have  SLES11 installed? As the RAM size is around 192Gb, I assume they 
are NUMA systems?
If so, SLES11 has defrag_ratio tunables in /sys/kernel/slab/xxx
 From the source of get_any_partial()

#ifdef CONFIG_NUMA

        /*
         * The defrag ratio allows a configuration of the tradeoffs between
         * inter node defragmentation and node local allocations. A lower
         * defrag_ratio increases the tendency to do local allocations
         * instead of attempting to obtain partial slabs from other nodes.
         *
         * If the defrag_ratio is set to 0 then kmalloc() always
         * returns node local objects. If the ratio is higher then kmalloc()
         * may return off node objects because partial slabs are obtained
         * from other nodes and filled up.
         *
         * If /sys/kernel/slab/xx/defrag_ratio is set to 100 (which makes
         * defrag_ratio = 1000) then every (well almost) allocation will
         * first attempt to defrag slab caches on other nodes. This means
         * scanning over all nodes to look for partial slabs which may be
         * expensive if we do it every time we are trying to find a slab
         * with available objects.
         */

Could you please verify that your clients have defrag_ratio tunable and 
try to use various values?
It looks like the value of 100 should be the best, unless there is a 
bug, then may be even 0 gets the desired result?

Best regards,
Dmitry


Jagga Soorma wrote:
> Hi Johann,
>
> I am actually using 1.8.1 and not 1.8.2:
>
> # rpm -qa | grep -i lustre
> lustre-client-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default
> lustre-client-modules-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default
>
> My kernel version on the SLES 11 clients is:
> # uname -r
> 2.6.27.29-0.1-default
>
> My kernel version on the RHEL 5.3 mds/oss servers is:
> # uname -r
> 2.6.18-128.7.1.el5_lustre.1.8.1.1
>
> Please let me know if you need any further information.  I am still 
> trying to get the user to help me run his app so that I can run the 
> leak finder script to capture more information.
>
> Regards,
> -Simran
>
> On Tue, Apr 27, 2010 at 7:20 AM, Johann Lombardi <johann at sun.com 
> <mailto:johann at sun.com>> wrote:
>
>     Hi,
>
>     On Tue, Apr 20, 2010 at 09:08:25AM -0700, Jagga Soorma wrote:
>     > Thanks for your response.* I will try to run the leak-finder
>     script and
>     > hopefully it will point us in the right direction.* This only
>     seems to be
>     > happening on some of my clients:
>
>     Could you please tell us what kernel you use on the client side?
>
>     >    client104: ll_obdo_cache********* 0 433506280*** 208** 19***
>     1 : tunables*
>     >    120** 60*** 8 : slabdata***** 0 22816120***** 0
>     >    client116: ll_obdo_cache********* 0 457366746*** 208** 19***
>     1 : tunables*
>     >    120** 60*** 8 : slabdata***** 0 24071934***** 0
>     >    client113: ll_obdo_cache********* 0 456778867*** 208** 19***
>     1 : tunables*
>     >    120** 60*** 8 : slabdata***** 0 24040993***** 0
>     >    client106: ll_obdo_cache********* 0 456372267*** 208** 19***
>     1 : tunables*
>     >    120** 60*** 8 : slabdata***** 0 24019593***** 0
>     >    client115: ll_obdo_cache********* 0 449929310*** 208** 19***
>     1 : tunables*
>     >    120** 60*** 8 : slabdata***** 0 23680490***** 0
>     >    client101: ll_obdo_cache********* 0 454318101*** 208** 19***
>     1 : tunables*
>     >    120** 60*** 8 : slabdata***** 0 23911479***** 0
>     >    --
>     >
>     >    Hopefully this should help.* Not sure which application might
>     be causing
>     >    the leaks.* Currently R is the only app that users seem to be
>     using
>     >    heavily on these clients.* Will let you know what I find.
>
>     Tommi Tervo has filed a bugzilla ticket for this issue, see
>     https://bugzilla.lustre.org/show_bug.cgi?id=22701
>
>     Could you please add a comment to this ticket to describe the
>     behavior of the application "R" (fork many threads, write to
>     many files, use direct i/o, ...)?
>
>     Cheers,
>     Johann
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100519/2649f91a/attachment.htm>