[Lustre-discuss] Swap on Lustre (was: Client is not accesible when OSS/OST server is down)

[Robin Humble]

Hi Serge,

On Wed, Apr 30, 2008 at 12:17:03PM +0300, Serge Ryabchun wrote:
>2008/4/30, Robin Humble <rjh+lustre at cita.utoronto.ca>:
>>  we too are very interested in swap on Lustre using o2ib, so I'm happy
>>  to hear it'll be in 1.6.6.
>I used swap with loop on lustre over tcp and o2ib more then 2 years.

thanks for that.

I tried that with o2ib last week and as pretty much as soon as I use
any swap the node deadlocks hard in the low memory situation and has to
be reset.
possibly it's more to do with how loop.c uses aops and alloc's mem than
it is to do with Lustre, but either way, sadly, it's not a solution.

swap to iSER (backed by a file on Lustre on another server) has worked
and survived harsh testing for me in the past, as has swap over iSCSI
(TCP) when the clients have Peter Zijlstra's VM deadlock prevention
patches applied. presumably swap to SRP would work too. there are now
software targets available for all of these protocols now which is
nice.

swap directly or via loop to Lustre has never been successful for me.
possibly the loop approach might work if a variant of the Zijlstra
patches was applied to the loop code.
a working swap to native Lustre would be much better and faster though.

on a similar subject, loop on Lustre does work fine for making 'local'
filesystems on nodes, which can sustain local disk metadata rates
without any load on the Lustre MDS.
we use that here and it works well... +/- some VM problems in RedHat
kernels under low memory situations that forced us to move to 2.6.23
patchless Lustre clients.

cheers,
robin


>cat /etc/rc.d/rc.swap
>#!/bin/bash
>
>SWAP=/mnt/adm/swap/${HOSTNAME}.swap
>
>if [ ! -f $SWAP ]; then
>   dd if=/dev/zero of=$SWAP bs=1M count=4096
>else
>   dd if=$SWAP of=/dev/null count=1 bs=1M
>fi
>
>losetup /dev/loop0 $SWAP
>mkswap /dev/loop0 && swapon /dev/loop0
>
>-- 
>Serge Ryabchun <serge.ryabchun at gmail.com>