[lustre-discuss] Limiting Lustre memory use?
bill at broadley.org
Fri Feb 18 13:42:39 PST 2022
On a cluster I managed (without Lustre), we had many problems with users running
nodes out of ram which often killed the node. We added cgroup support to slurm
and those problems disappeared. Nearly 100% of the time get a cgroup OOM
instead of a kernel OOM and the nodes would stay up and stable. This became
doubly important when we started allowing jobs to share nodes and didn't want
job A to be able to crash job B.
I've tried similar on a Lustre enabled cluster and it seems like the memory used
by Lustre (which I believe is in the kernel and outside of the job's cgroup). I
think part of the problem is I believe Lustre caches metadata in the linux page
cache, but not data. I've tried reducing the ram available to slurm, but still
getting kernel OOMs instead of cgroup OOMs.
Anyone have a suggestion for fixing this? Is there any way to limit Lustre's
memory use in the kernel? Or force that caching into userspace and inside the
cgroup? Or possibly out of ram and onto a client local NVMe?
More information about the lustre-discuss