[Lustre-discuss] Adding IB to tcp only cluster

Hendelman, Rob Rob.Hendelman at magnetar.com
Thu Oct 30 07:07:09 PDT 2008

On Sun, Oct 12, 2008 at 10:15:01AM -0400, Brock Palen wrote:
>> ......
>> Currently we don't put any lustre modules in modprobe.conf,  lustre  
>> loads the correct modules when mounting the filesystem.  We do this  
>> to keep our loads simple as we have several.
>When nothing has been specified, LNet by default loads the ksocklnd,
>which in turn by default uses the 1st usable interface returned by
>> It would be nice if LNET picked IB without being told.  Similar to  
>> the way OpenMPI has network weights of which to try first.
>No such automatic mechanism exists. The LNet NIs can only be specified
>statically via module options ('networks' or 'ip2nets').
>As to choice of path for multi-homed LNet, the decision is solely
>based on the destination NID. If the NID belongs to a local network
> (e.g. at o2ib0 is on my local network @o2ib0 if I have a NI in
>@o2ib0 too), traffic would go through the local NI. If the NID is on a
>remote network (e.g. 3 at ptl0 if I don't have a NI in @ptl0), a router
>would be picked out among available routes based on load already
>queued on routers, and the local NI to that router would be used for
>outgoing traffic (e.g. the NI in @tcp0 would be used if
> at tcp0 is the router chosen).

So if I had an MGS/MDT with 1 nic with say 4 nids on different subnets
and specified tpc0 at subnet1, tcp1 at subnet2, etc in modprobe.conf, could I
then, on an OSS with 4 nics with 1 ip on each subnet per nic, format
each OST with parameters to force it to use a certain local nic to
connect to the mgs/mdt ?

>In other words, the LNet path from a multi-homed client to a
>multi-homed server is determined by the server NID. For example, if
>both the client and the server are on @tcp0 and @o2ib0, the client
>would choose IB network if the server NID is in @o2ib0, and TCP
>network otherwise. The server NID used by Lustre clients should
>somehow come from the MGS but I'm not sure about it. LNet has no
>knowledge about whether a peer is multi-homed, so it couldn't figure
>out that the IB network is a better path to reach a peer in @tcp.

I assume I could do the same with clients to reach the servers and force
the clients to use certain nics as well...correct?



The information contained in this message and its attachments 
is intended only for the private and confidential use of the 
intended recipient(s).  If you are not the intended recipient 
(or have received this e-mail in error) please notify the 
sender immediately and destroy this e-mail. Any unauthorized 
copying, disclosure or distribution of the material in this e-
mail is strictly prohibited.

More information about the lustre-discuss mailing list