[lustre-discuss] Limiting Lustre memory use?

[bill broadley]

On a cluster I managed (without Lustre), we had many problems with users running 
nodes out of ram which often killed the node.  We added cgroup support to slurm 
and those problems disappeared.  Nearly 100% of the time get a cgroup OOM 
instead of a kernel OOM and the nodes would stay up and stable. This became 
doubly important when we started allowing jobs to share nodes and didn't want 
job A to be able to crash job B.

I've tried similar on a Lustre enabled cluster and it seems like the memory used 
by Lustre (which I believe is in the kernel and outside of the job's cgroup).  I 
think part of the problem is I believe Lustre caches metadata in the linux page 
cache, but not data.  I've tried reducing the ram available to slurm, but still 
getting kernel OOMs instead of cgroup OOMs.

Anyone have a suggestion for fixing this?  Is there any way to limit Lustre's 
memory use in the kernel?  Or force that caching into userspace and inside the 
cgroup?  Or possibly out of ram and onto a client local NVMe?