[Lustre-discuss] LNET over multiple NICs

Christopher J. Walker C.J.Walker at qmul.ac.uk
Mon Jan 7 08:44:26 PST 2013


On 19/12/12 23:42, White, Cliff wrote:
> 
> 
> On 12/19/12 2:03 PM, "Alexander Oltu" <Alexander.Oltu at uni.no> wrote:
> 
>>
>>> I have no experience in doing multirail on ethernet, sorry. The
>>> principle is exactly the same as for Infiniband, but as Infiniband
>>> interfaces cannot be bonded (except for IPoIB which is not of
>>> interest when considering performance), I cannot tell.
>>
>> Looks like --network option is not helping. I have tried to unmount osts
>> and run  for half of OSTs:
>>
>> tunefs.lustre --network=tcp0
>>
>> and for another half:
>>
>> tunefs.lustre --network=tcp1
>>
>> Mounted OSTs back. The client still sends all traffic through tcp0. I
>> have suspicion that --network option is to separate from different
>> network types, like tcp, o2ib, etc.
>>
>> Probably I will go bonding way.
> 
> For Ethernet, bonding is the preferred method. Use the standard method, and
> then point Lustre at the 'bond' interface.

One point worth remembering with bonding is that the onboard ethernet
cards (at least on the machine we bought) have adjacent MAC addresses.
With mode4 (lacp) bonding, the default linux (layer2) hashing algorithm,
the traffic is likely to be unevenly distributed.  We ended up using
xmit_hash_policy=layer3+4 to distribute traffic more evenly between the
cards.

We didn't perform detailed tests on whether layer2+3 would have done,
and how much difference the switch (which I suspect uses layer2+3) made.
We subsequently upgraded to 10Gig - which has made this moot for a while.


> The modprobe option 'networks=' is used to bind hardware interfaces, for
> kernel bonding you would normally use: options lnet networks=tcp(bond0)
> 
> See 
> http://wiki.lustre.org/manual/LustreManual20_HTML/SettingUpBonding.html#504
> 38258_99571
> For more (perhaps a little old) info.
> cliffw
> 

Chris


Chris

>>
>> Thank you for your help,
>> Alex.
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss




More information about the lustre-discuss mailing list