[Lustre-discuss] Swap on Lustre (was: Client is not accesible when OSS/OST server is down)

[Robin Humble]

On Tue, Apr 29, 2008 at 04:27:42PM -0400, Brian J. Murrell wrote:
>On Tue, 2008-04-29 at 09:05 -0700, Kilian CAVALOTTI wrote:
>> I was precisely wondering if it was possible to use a file residing on a 
>> Lustre filesystem as a swap file.
>There certainly has been some historic use and testing of swap on
>Lustre.  I'm not sure what it's current state is TBH.  The last bug in
>bugzilla I can find about swap and Lustre is bug 5794/5749/5188 filed
>over two years ago (however updated just recently) in reference to some
>of our regression testing we do with swap on Lustre.
>
>It seems the final outcome is that it has been flagged (just today in
>fact) as a 1.6.6 blocker.  The timing is absolutely uncanny!
>
>> # cd /scratch
>> /scratch # grep /scratch /proc/mounts
>> 10.10.50.2 at o2ib:/scratch /scratch lustre rw 0 0
>> /scratch # dd if=/dev/zero of=./swapfile  bs=1024 count=1024
>> 10240+0 records in
>> 10240+0 records out
>> /scratch # mkswap ./swapfile
>> Setting up swapspace version 1, size = 104853 kB
>> /scratch # swapon -a ./swapfile
>> swapon: ./swapfile: Invalid argument
>> 
>> Is that expected?
>
>I would think not, no.  There is some reference in bug 5188 that we only
>test it for "real/remote OSTs" and "non-TCP" interconnects.  I wonder if
>either of those match your situation.  You might file a bug about it,
>including the usual data -- syslog from the client you did the swapon on
>for example.  A (-1) debug log from the client right at the time of the
>swapon would probably be useful.

when doing the 'swapon' if you look in dmesg you should see:
  kernel: swapon: swapfile has holes
even though the file on Lustre isn't sparse.

I was looking into this last week, and I think this message is because
Lustre (at least up to 1.6.4.3) doesn't appear to implement bmap() ->
  lustre/llite/rw26.c:289:        .bmap           = NULL
and the swapfile code in RedHat kernels and 2.6.23 mm/swapfile.c
setup_swap_extents() expects bmap() to be present, and assumes the file
is sparse otherwise :-/

we too are very interested in swap on Lustre using o2ib, so I'm happy
to hear it'll be in 1.6.6.

cheers,
robin