[Lustre-discuss] Lustre Client - Memory Issue
Jagga Soorma
jagga13 at gmail.com
Thu Aug 26 17:42:55 PDT 2010
Hi Dmitry,
I am still running into this issue on some nodes:
client109: ll_obdo_cache 0 152914489 208 19 1 : tunables
120 60 8 : slabdata 0 8048131 0
client102: ll_obdo_cache 0 308526883 208 19 1 : tunables
120 60 8 : slabdata 0 16238257 0
How can I calculate how much memory this is holding on to. My system shows
a lot of memory that is being used up but none of the jobs are using that
much memory. Also, these clients are running a smp sles 11 kernel but I
can't find any /sys/kernel/slab directory.
Linux client102 2.6.27.29-0.1-default #1 SMP 2009-08-15 17:53:59 +0200
x86_64 x86_64 x86_64 GNU/Linux
What makes you say that this does not look like a lustre memory leak? I
thought all the ll_* objects in slabinfo are lustre related? To me it looks
like lustre is holding on to this memory but I don't know much about lustre
internals.
Also, memused on these systems are:
client102: 2353666940
client109: 2421645924
Any help would be greatly appreciated.
Thanks,
-J
On Wed, May 19, 2010 at 8:08 AM, Dmitry Zogin <dmitry.zoguine at oracle.com>wrote:
> Hello Jagga,
>
> I checked the data, and indeed this does not look like a lustre memory
> leak, rather than a slab fragmentation, which assumes there might be a
> kernel issue here. From the slabinfo (I only keep three first columns here):
>
>
> name <active_objs> <num_objs>
> ll_obdo_cache 0 452282156 208
>
> means that there are no active objects, but the memory pages are not
> released back from slab allocator to the free pool (the num value is huge).
> That looks like a slab fragmentation - you can get more description at
> http://kerneltrap.org/Linux/Slab_Defragmentation
>
> Checking your mails, I wonder if this only happens on clients which have
> SLES11 installed? As the RAM size is around 192Gb, I assume they are NUMA
> systems?
> If so, SLES11 has defrag_ratio tunables in /sys/kernel/slab/xxx
> From the source of get_any_partial()
>
> #ifdef CONFIG_NUMA
>
> /*
> * The defrag ratio allows a configuration of the tradeoffs between
> * inter node defragmentation and node local allocations. A lower
> * defrag_ratio increases the tendency to do local allocations
> * instead of attempting to obtain partial slabs from other nodes.
> *
> * If the defrag_ratio is set to 0 then kmalloc() always
> * returns node local objects. If the ratio is higher then
> kmalloc()
> * may return off node objects because partial slabs are obtained
> * from other nodes and filled up.
> *
> * If /sys/kernel/slab/xx/defrag_ratio is set to 100 (which makes
> * defrag_ratio = 1000) then every (well almost) allocation will
> * first attempt to defrag slab caches on other nodes. This means
> * scanning over all nodes to look for partial slabs which may be
> * expensive if we do it every time we are trying to find a slab
> * with available objects.
> */
>
> Could you please verify that your clients have defrag_ratio tunable and try
> to use various values?
> It looks like the value of 100 should be the best, unless there is a bug,
> then may be even 0 gets the desired result?
>
> Best regards,
> Dmitry
>
>
> Jagga Soorma wrote:
>
> Hi Johann,
>
> I am actually using 1.8.1 and not 1.8.2:
>
> # rpm -qa | grep -i lustre
> lustre-client-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default
> lustre-client-modules-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default
>
> My kernel version on the SLES 11 clients is:
> # uname -r
> 2.6.27.29-0.1-default
>
> My kernel version on the RHEL 5.3 mds/oss servers is:
> # uname -r
> 2.6.18-128.7.1.el5_lustre.1.8.1.1
>
> Please let me know if you need any further information. I am still trying
> to get the user to help me run his app so that I can run the leak finder
> script to capture more information.
>
> Regards,
> -Simran
>
> On Tue, Apr 27, 2010 at 7:20 AM, Johann Lombardi <johann at sun.com> wrote:
>
>> Hi,
>>
>> On Tue, Apr 20, 2010 at 09:08:25AM -0700, Jagga Soorma wrote:
>> > Thanks for your response.* I will try to run the leak-finder script
>> and
>> > hopefully it will point us in the right direction.* This only seems to
>> be
>> > happening on some of my clients:
>>
>> Could you please tell us what kernel you use on the client side?
>>
>> > client104: ll_obdo_cache********* 0 433506280*** 208** 19*** 1 :
>> tunables*
>> > 120** 60*** 8 : slabdata***** 0 22816120***** 0
>> > client116: ll_obdo_cache********* 0 457366746*** 208** 19*** 1 :
>> tunables*
>> > 120** 60*** 8 : slabdata***** 0 24071934***** 0
>> > client113: ll_obdo_cache********* 0 456778867*** 208** 19*** 1 :
>> tunables*
>> > 120** 60*** 8 : slabdata***** 0 24040993***** 0
>> > client106: ll_obdo_cache********* 0 456372267*** 208** 19*** 1 :
>> tunables*
>> > 120** 60*** 8 : slabdata***** 0 24019593***** 0
>> > client115: ll_obdo_cache********* 0 449929310*** 208** 19*** 1 :
>> tunables*
>> > 120** 60*** 8 : slabdata***** 0 23680490***** 0
>> > client101: ll_obdo_cache********* 0 454318101*** 208** 19*** 1 :
>> tunables*
>> > 120** 60*** 8 : slabdata***** 0 23911479***** 0
>> > --
>> >
>> > Hopefully this should help.* Not sure which application might be
>> causing
>> > the leaks.* Currently R is the only app that users seem to be using
>> > heavily on these clients.* Will let you know what I find.
>>
>> Tommi Tervo has filed a bugzilla ticket for this issue, see
>> https://bugzilla.lustre.org/show_bug.cgi?id=22701
>>
>> Could you please add a comment to this ticket to describe the
>> behavior of the application "R" (fork many threads, write to
>> many files, use direct i/o, ...)?
>>
>> Cheers,
>> Johann
>>
>
> ------------------------------
>
> _______________________________________________
> Lustre-discuss mailing listLustre-discuss at lists.lustre.orghttp://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100826/f8abaf3f/attachment.htm>
More information about the lustre-discuss
mailing list