[Lustre-discuss] Memory (?) problem with 1.8.1

[Andreas Dilger]

On 13-Oct-09, at 04:58, Brian J. Murrell wrote:
> On Mon, 2009-10-12 at 17:06 -0700, David Simas wrote:
>> During this operation, we notice the value of "buffers"
>> reported in '/proc/meminfo' on the OSSs involved increasing  
>> monotonically
>> until it apparently take up all the system's memory - 32 GB.
>
> This would likely be OSS read cache, if you have it enabled.  If you  
> do,
> you should disable it due to a potential corruption issue.  Details  
> were
> given previously on this list on how to do that.  Check the archives.
>
> But that's not directly related to what you are seeing.
>
> Having "buffers" consume all of available memory is SOP (standard
> Operating Procedure) for Linux.  The philosophy is that  
> "free" (unused)
> memory is wasted memory and as such, any memory not needed by
> applications or other kernel processing is used to buffer disk I/O.   
> The
> performance spiff of such is obvious I think.
>
>> Then 'kswapd'
>> starts consuming a large amount of CPU, the load increases (100+),  
>> and the
>> system, including Lustre, slows to crawl and becomes quite useless.
>
> This doesn't sound normal or good.

There is a recent bug for memory pressure on the OSS.  I believe it was
fixed for the next release.


>> If we
>> stop Lustre I/O at this point, 'kswapd' and the system load calm  
>> down, but
>> the "buffers" value does not decrease.
>
> Right.  The buffers will not get "emptied" until something else needs
> the memory.  Again, unused memory is wasted memory.
>
>> We have observed the monotonically increasing "buffers" condition
>> with non-Lustre I/O on systems running the Lustre 1.8.1 kernel
>> (2.6.18-128.1.14.el5_lustre.1.8.1),
>
> Indeed.  The using up of memory by the buffer cache is a standard  
> (i.e.
> non-Lustre-specific) feature, and you will find the same thing on
> non-Lustre kernels as well.
>
> b.
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss


Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.