[Lustre-discuss] Swap over lustre

[Joe Landman]

On 08/17/2011 11:42 PM, David Dillow wrote:
> On Wed, 2011-08-17 at 22:57 -0400, Joe Landman wrote:
>> The lustre client (and most NFS or even network block devices) all do
>> memory allocation of buffers ... which is anathema to migrating pages
>> out to disk.  You can easily wind up in a "death spiral" race condition
>> (and it sounds like you are there).  You might be able to do something
>> with iSCSI or SRP (though these also do block allocations and could
>> trigger death spirals).
>
> Your post is generally correct, but minor nit here: there is no memory
> allocation on the command path of the Linux SRP initiator, so the death
> spiral is not possible there. I suspect the iSCSI initiator takes

Thanks for clarifying that.  I know that during startup there is an 
allocation, but I wasn't sure after that.

> similar precautions -- or uses mempools -- to avoid this fate, but I'm
> not as familiar with that code.

I think they also try to pre-allocate as much as possible.

One issue we've seen in conjunction with these has been on some network 
drivers with skb allocations, in tight memory situations, it can cause 
some problems when there are very few free pages.  Usually we get a 
bunch of messages in the logs, but on some occasions, the network device 
shuts down (unable to allocate send/receive buffers).  Have seen this on 
igb, e1000e, and e1000 based networks.



>
> Cheers,
> Dave


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615