[Lustre-discuss] Swap over lustre

[Andreas Dilger]

On 2011-08-17, at 8:43 PM, John Hanks wrote:
> I've been trying to get swap on lustre to work with not much success
> using blockdev_attach and the resulting lloop0 device and using
> losetup and the resulting loop device. This thread
> (http://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg00856.html)
> claims that it works, but in all my attempts almost as soon as swap is
> used (testing with memhog), the host hangs. In some cases it hangs
> hard, but on occasion if I'm patient enough the OOM will eventually
> kill something and the node will become responsive again. If I
> carefully increase memory with each successive memhog run I can get
> some pages to swap, but any real pressure always results in a hang.
> I'm attempting this on Redhat EL 5.6 with lustre 1.8.4 patchless
> client over IB.

Using IB is important to try this out, since the Lustre RDMA will
use preallocate pages for the RPC, unlike TCP where there can be
be problems allocating the TCP receive buffers.

That said, the swap-on-Lustre code was never really finished.  If
you are interested to debug this and have some coding skills you
could probably get some help for debugging on the list.

You need to have a serial console attached to the client node, and
grab the stack traces from the client to see where it is stuck
allocating memory, and then remove/avoid/preallocate it.

Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.