[Lustre-discuss] MDS memory usage

[Bernd Schubert]

Hello Frederik,


On Wednesday, August 25, 2010, Frederik Ferner wrote:
> Hi Bernd,
> 
> thanks for your reply.
> 
> Bernd Schubert wrote:
> > On Tuesday, August 24, 2010, Frederik Ferner wrote:
> >> on our MDS we noticed that all memory seems to be used. (And it's not
> >> just normal buffers/cache as far as I can tell.)
> >> 
> >> When we put load on the machine, for example by starting rsync
> >> on a few clients, generating file lists to copy data from Lustre to
> >> local disks or just running a MDT backup locally using dd/gzip to copy a
> >> LVM snapshot to a remote server, kswapd starts using a lot of CPU
> >> time, sometimes up to 100% of one CPU core.
> >> 
> >> This is on a Lustre 1.6.7.2.ddn3.5 based file system with about 200TB,
> >> the MDT is 800GB with 200M inodes, ACLs enabled.
> > 
> > Did you recompile it, or did you use the binaries from my home page (or
> > those you got from CV)?
> 
> This is a recompiled Lustre version to include the patch from bug   	
> 22820.
> 
> > Possibly it is a LRU auto-resize problem, but which has been disabled in
> > DDN builds. As our 1.6 releases didn't include a patch for that, you
> > would need to specify the correct command options if you recompiled it.
> 
> I guess it's likely that I have not specified the correct option. So the
>   binaries on your home page are compiled with '--disable-lru-resize'?
> Any other options that you used?

I always enable the health-write, which will help pacemaker to detect IO 
errors (by monitoring /proc/fs/lustre/health_check)

 --enable-health-write 

> 
> > Another reason might be bug 22771, although that should only come up on
> > MDS with more memory you have.
> 
> I had a look at that bug and while we have a default stripe count of 1
> so the stripe count should fit into the inode. On the other hand we use
> ACLs in quite a few places, so it seems we might hit this bug if we
> increase the memory from the 16GB currently, correct?

Yeah and I think 16GB should be sufficient for the MDS. 



-- 
Bernd Schubert
DataDirect Networks