[Lustre-discuss] LNET Topologies

Isaac Huang He.Huang at Sun.COM
Tue Nov 27 08:10:51 PST 2007


On Tue, Nov 27, 2007 at 07:37:55AM -0500, Charles Taylor wrote:
>    There are a number of LNET routing examples in the Lustre documentation
>    but the actual LNET topology requirements don't seem to be spelled out.
>          In the simplest terms, what are the actual network topology
>    requirements for the MDS, OSS's, and clients in order for lustre to
>    work correctly?

The simplest topology is one where all nodes reside in a same LNET
network - like in your IB network, they're all in @o2ib.

[......]
>    However, after mounting the lustre file system on the ethernet clients
>    (which succeeds), the clients are always evicted immediately following
>    the obd_timeout period with a message such as...
> 
>    Lustre: ufhpc-MDT0000: haven't heard from client
>    f6c6db8a-6fbc-6464-6261-af9ecfc0cb60 (at 10.13.18.152 at tcp) in 2462
>    seconds. I think it's dead, and I am evicting it.
> 
>    Of course, once that happens, the client can no longer write to the
>    file system although it works fine for the "obd_timeout" period.     We
>    don't completely understand this because all the players can talk to
>    each other via IP.   Why would the MDT not be "hearing" from the
>    ethernet client?   They are on the same IP network.  It seems like the

Can you please run 'lctl ping client_ip at tcp' on the MDT and 'lctl ping
mdt_ip at tcp' on the client? It'd be also helpful to enable low level
network error console logging on both nodes before running the commands:
echo +neterror > /proc/sys/lnet/printk

>    problem is in the lnet topology?    Do we really have to introduce an
>    lnet router even though we have complete IP connectivity among the
>    various components (mds, oss, clients)?

No, you don't have to. But for better performance you perhaps will
need an LNET router between TCP clients and the OSSes.

The traffic goes through different protocol stacks, in the case of
IPoIB gateway:
Client (LNET:SOCKLND:TCP:IP)  => IPoIB GW (IP:IPoIB:IB) => OSS (IB:IPoIB:IP:TCP:SOCKLND:LNET)

In the case of an LNET router:
Client (LNET:SOCKLND:TCP:IP)  => LNET RTR (IP:TCP:SOCKLND:LNET:O2IBLND:IB) => OSS (IB:O2IBLND:LNET)

The total number of protocol stacks involved in the two paths are the
same, but:
1. The extra stacks on an LNET router (SOCKLND:LNET:O2IBLND) are
rather light-weight.
2. Running o2iblnd over IB shall yield better performance than running
socklnd over IPoIB.

HTH,
Isaac




More information about the lustre-discuss mailing list