[Lustre-discuss] Multi-Role/Tasking MDS/OSS Hosts

[Robert Read]

Hi,

On Sep 17, 2010, at 12:48 , Bernd Schubert wrote:

> On Friday, September 17, 2010, Andreas Dilger wrote:
>> On 2010-09-17, at 12:42, Jonathan B. Horen wrote:
>>> We're trying to architect a Lustre setup for our group, and want to
>>> leverage our available resources. In doing so, we've come to consider
>>> multi-purposing several hosts, so that they'll function simultaneously
>>> as MDS & OSS.
>> 
>> You can't do this and expect recovery to work in a robust manner.  The
>> reason is that the MDS is a client of the OSS, and if they are both on the
>> same node that crashes, the OSS will wait for the MDS "client" to
>> reconnect and will time out recovery of the real clients.
> 
> Well, that is some kind of design problem. Even on separate nodes it can 
> easily happen, that both MDS and OSS fail, for example power outage of the 
> storage rack. In my experience situations like that happen frequently...
> 
> I think some kind a pre-connection would be required, where a client can tell 
> a server, that it was rebooted and that the server shall not to wait any 
> longer for it. Actually, shouldn't be that difficult, as already different 
> connection flags exist. So if the client contacts a server and ask for an 
> initial connection, the server could check for that NID and then immediately 
> abort recovery for that client.

This is an interesting idea, but NID is not ideal as this wouldn't be compatible
with multiple mounts on the same node.  Not very useful in production, perhaps,
but very useful for testing.

Another option would be to hash the mount point  pathname (and some other data, 
such as the NID) and use this as the client uuid.  Then the client uuid would be persistent 
across reboots and the server would rely on flags to detect if this was a reconnect or a
new connection after a reboot or remount.

robert