[Lustre-discuss] Adding IB to tcp only cluster

[Brock Palen]

On Oct 12, 2008, at 1:36 AM, Andreas Dilger wrote:

> On Oct 10, 2008  15:09 -0400, Brian J. Murrell wrote:
>> Regardless, once LNET has it's list of interfaces and networks it  
>> routes
>> requests accordingly depending on their destination.  If a target
>> (either an MDS or OSS) only has a TCP path that path will be  
>> used.  If a
>> target has more than one path the "best" path will be chosen.  I  
>> tend to
>> think o2iblnd trumps socklnd.
>>
>> Maybe if there is an LNET engineer reading he can give you more  
>> details
>> on how this best path is chosen.
>
> I'm no LNET expert, but my understanding is:
>
> LNET will pick the network with the fewest LNET network hops (TCP  
> network
> hops are not a consideration), and if they are equal then it will  
> (AFAIK)
> pick the first network listed in the "networks" module option as  
> the "best"
> network and use that exclusively until it fails.

Currently we don't put any lustre modules in modprobe.conf,  lustre  
loads the correct modules when mounting the filesystem.  We do this  
to keep our loads simple as we have several.

It would be nice if LNET picked IB without being told.  Similar to  
the way OpenMPI has network weights of which to try first.

>
> That means "for clients with IB, put the o2iblnd interface first  
> and the
> TCP interface second" in the "lnet networks" module option.  I  
> think this
> is discussed in the Lustre manual already.
>
>>> If that is not possible now, I think some of the patches announced
>>> that are for 1.8 or 2.0 had the ability to select a OSS for only
>>> given users.  Am I correct?
>>
>> I think you are talking about OST pools.  I'm not sure which release
>> that is targeted for.
>
> While OST pools is going to be included into 1.8.0, it doesn't have
> the policy engine yet, so the "pick a pool for UID X" doesn't exist  
> yet.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>