[Lustre-discuss] Memory (?) problem with 1.8.1

[Brian J. Murrell]

On Mon, 2009-10-12 at 17:06 -0700, David Simas wrote:
> Hello,

Hi,

> During this operation, we notice the value of "buffers"
> reported in '/proc/meminfo' on the OSSs involved increasing monotonically
> until it apparently take up all the system's memory - 32 GB.

This would likely be OSS read cache, if you have it enabled.  If you do,
you should disable it due to a potential corruption issue.  Details were
given previously on this list on how to do that.  Check the archives.

But that's not directly related to what you are seeing.

Having "buffers" consume all of available memory is SOP (standard
Operating Procedure) for Linux.  The philosophy is that "free" (unused)
memory is wasted memory and as such, any memory not needed by
applications or other kernel processing is used to buffer disk I/O.  The
performance spiff of such is obvious I think.

> Then 'kswapd'
> starts consuming a large amount of CPU, the load increases (100+), and the
> system, including Lustre, slows to crawl and becomes quite useless.

This doesn't sound normal or good.

> If we
> stop Lustre I/O at this point, 'kswapd' and the system load calm down, but
> the "buffers" value does not decrease.

Right.  The buffers will not get "emptied" until something else needs
the memory.  Again, unused memory is wasted memory.

> We have observed the monotonically increasing "buffers" condition
> with non-Lustre I/O on systems running the Lustre 1.8.1 kernel
> (2.6.18-128.1.14.el5_lustre.1.8.1),

Indeed.  The using up of memory by the buffer cache is a standard (i.e.
non-Lustre-specific) feature, and you will find the same thing on
non-Lustre kernels as well.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20091013/bc1bf710/attachment.pgp>