[Lustre-discuss] LNET Topologies

[Charles Taylor]

There are a number of LNET routing examples in the Lustre  
documentation but the actual LNET topology requirements don't seem to  
be spelled out.       In the simplest terms, what are the actual  
network topology requirements for the MDS, OSS's, and clients in  
order for lustre to work correctly?

MDS NIDS
-------------------------
10.13.24.40 at o2ib
10.13.16.40 at tcp (ip/ethernet)  ----------------------------------------
                                                                       |
OSS NIDS                                                              |
-----------------------                                               |
10.13.24.42 at o2ib                                                      |
10.13.31.229 at tcp  (ipoib subnet)  --------------------------- ipoib  
gateway
                                                                       |
IB Client NIDS                                                        |
-----------------------                                               |
10.13.25.150 at o2ib                                                     |
                                                                       |
Ethernet Client NIDS                                                  |
----------------------------                                          |
10.13.18.152 at tcp ip/ethernet -------------------------------------------


There is no problem with the o2ib side of things.    The IB network  
is 10.13.24.0/21 so the clients, mds, and oss's are all on the same  
"network".

The ethernet clients have no access to the 10.13.24.0 network.      
They communicate with the MDS directly on the 10.13.16.0/22  
network.   However, the OSSs for the ethernet side all sit on their  
own subnetted /30 network each of which is bridged by an IPoIB  
gateway between the OSSs and the ethernet clients.     So the MDS and  
the clients talk to the OSSs through the IPoIB gateway and the OSSs  
can talk to each other through the IPoIB Gateway (though on separate  
subnets - there is a good reason for this, trust me :) ).   So there  
is complete connectivity in the "IP" sense.

However, after mounting the lustre file system on the ethernet  
clients (which succeeds), the clients are always evicted immediately  
following the obd_timeout period with a message such as...

Lustre: ufhpc-MDT0000: haven't heard from client  
f6c6db8a-6fbc-6464-6261-af9ecfc0cb60 (at 10.13.18.152 at tcp) in 2462  
seconds. I think it's dead, and I am evicting it.

Of course, once that happens, the client can no longer write to the  
file system although it works fine for the "obd_timeout" period.      
We don't completely understand this because all the players can talk  
to each other via IP.   Why would the MDT not be "hearing" from the  
ethernet client?   They are on the same IP network.  It seems like  
the problem is in the lnet topology?    Do we really have to  
introduce an lnet router even though we have complete IP connectivity  
among the various components (mds, oss, clients)?

I'm hoping that someone more familiar with the LNET abstraction layer  
can help us understand what the problem is.

Thanks,

charlie taylor
uf hpc center


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20071127/0819cf21/attachment.htm>