[Lustre-discuss] Multiple interfaces in LNet

[Isaac Huang]

On Mon, Oct 13, 2008 at 05:29:14PM +0200, Danny Sternkopf wrote:
> ......
> Interesting is how to use multiple interfaces on the same server in
> Lustre/LNet. My understanding is that TCP(ksocklnd) can manage multiple
> physical interfaces as one LNet interface with one unique NID. Is that
> still correct and recommended to use? Or is it better to setup Ethernet
> bonding(under Linux) and bind that bonding interface to LNet?

Yes, ksocklnd is capable of load-balancing over multiple physical
interfaces under one LNet interface, but the Linux Ethernet bonding is
the recommended way to go.

> Beside of TCP it is only possible to use multiple interfaces on the same
> node with o2ib, right? With ko2iblnd one can setup several Lustre
> networks for each IB interface. In fact you must setup several Lustre
> networks otherwise only the 1st IB interface is used, correct?

Yes, ksocklnd and ko2iblnd are the only LNDs that support multiple
instances, i.e. LNet interfaces. Unlike ksocklnd, each ko2iblnd
interface only sits on one IB interface, and multiple ko2iblnd
interfaces can sit on a same IB interface.

> It is not clear for me how MGS, MDS, OSS and Client choose a NID for
> communication. I mean I know that LNet choose the best one, but who
> provides the list with all available NIDs for a server? Or does it work
> somehow different?

To be precise, for outgoing messages, LNet only chooses a local NID to
use, which is determined by the remote NID specified by PTLRPC (and
ultimately by Lustre). Though it's possible to specify a local NID
(e.g. the 'self' parameter of LNetPut), it is in fact used only for
sanity checks.

Isaac