[Lustre-discuss] controlling which eth interface lustre uses

Lundgren, Andrew Andrew.Lundgren at Level3.com
Thu Oct 21 08:07:38 PDT 2010


Just as a FYI, you can set most of the bonding options in the ifcfg-bond0 file.

IE:

BONDING_OPTS="arp_ip_target=10.248.58.254 arp_interval=500 mode=active-backup primary=eth0"

Then your modprobe.conf only needs:

alias bond0 bonding

-----Original Message-----
From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Bob Ball
Sent: Thursday, October 21, 2010 8:41 AM
To: lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] controlling which eth interface lustre uses

OK, quick startup on bonding, as we use it for our OSS here.

We have 2 NICs we bond (SL5.5, an RHEL variant), eth1 at 1Gb and eth2 at 
10Gb using Myricom hardware.  10.10.1.2 is the network gateway, a 
convenient arp target that should always be up.

[root at umdist04 network-scripts]# cat ifcfg-bond0
DEVICE=bond0
IPADDR=10.10.2.24
NETMASK=255.255.252.0
BOOTPROTO=static
ONBOOT=yes
VLAN=no
MTU=1500

[root at umdist04 network-scripts]# cat ifcfg-eth1
DEVICE=eth1
ONBOOT=no
BOOTPROTO=none
MTU=1500
MASTER=bond0
SLAVE=yes

[root at umdist04 network-scripts]# cat ifcfg-eth2
DEVICE=eth2
BOOTPROTO=none
ONBOOT=no
MTU=1500
MASTER=bond0
SLAVE=yes

[root at umdist04 etc]# cat modprobe.conf
...
alias eth1 bnx2
alias eth2 myri10ge
...
alias bond0 bonding
options bond0 mode=1 arp_interval=250 arp_ip_target=10.10.1.2 primary=eth2
options lnet networks=tcp0(bond0)
...

You can check /proc/net/bonding/bond0 afterwards for information.

bob


On 10/21/2010 9:59 AM, Bob Ball wrote:
> Why do you need both active?  If one is a backup to the other, then bond
> them as a primary/backup pair, meaning only one will be active at at a
> time, ie, your designated primary (unless it goes down).
>
> bob
>
> On 10/21/2010 9:51 AM, Brock Palen wrote:
>> On Oct 21, 2010, at 9:48 AM, Joe Landman wrote:
>>
>>> On 10/21/2010 09:37 AM, Brock Palen wrote:
>>>> We recently added a new oss, it has 1 1Gb interface and 1 10Gb
>>>> interface,
>>>>
>>>> The 10Gb interface is eth4 10.164.0.166 The 1Gb   interface is eth0
>>>> 10.164.0.10
>>> They look like they are on the same subnet if you are using /24 ...
>> You are correct
>>
>> Both interfaces are on the same subnet:
>>
>> [root at oss4-gb ~]# route
>> Kernel IP routing table
>> Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
>> 10.164.0.0      *               255.255.248.0   U     0      0        0 eth0
>> 10.164.0.0      *               255.255.248.0   U     0      0        0 eth4
>> 169.254.0.0     *               255.255.0.0     U     0      0        0 eth4
>> default         10.164.0.1      0.0.0.0         UG    0      0        0 eth0
>>
>> There is no way to mask the lustre service away from the 1Gb interface?
>>
>>>> In modprobe.conf I have:
>>>>
>>>> options lnet networks=tcp0(eth4)
>>>>
>>>> lctl list_nids 10.164.0.166 at tcp
>>>>
>>>>>    From a host I run:
>>>> lctl which_nid oss4 10.164.0.166 at tcp
>>>>
>>>> But yet I still see traffic over eth0 the 1Gb management network,
>>>> might higher than I would expect (upto 100MB/s) The management
>>>> interface is oss4-gb  So If I do from a client:
>>>>
>>>> lctl which_nid oss4-gb 10.164.0.10 at tcp
>>>>
>>>> Why If I have netwroks=tcp0(eth4)  and that list_nids showa only the
>>>> 10Gb interface, do I have so much traffic over the 1Gb interface?
>>>> There is some traffic on the 10Gb interface, but I would like to tell
>>>> lustre 'don't use the 1Gb interface'.
>>> If they are on the same subnet, its possible that the 1GbE sees the arp
>>> response first.  And then its pretty much guaranteed to have the traffic
>>> go out that port.
>>>
>>> If your subnets are different, this shouldn't be the issue.
>>>
>>>> Thanks!
>>>>
>>>> Brock Palen www.umich.edu/~brockp Center for Advanced Computing
>>>> brockp at umich.edu (734)936-1985
>>>>
>>>>
>>>>
>>>> _______________________________________________ Lustre-discuss
>>>> mailing list Lustre-discuss at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>> -- 
>>> Joseph Landman, Ph.D
>>> Founder and CEO
>>> Scalable Informatics Inc.
>>> email: landman at scalableinformatics.com
>>> web  : http://scalableinformatics.com
>>>          http://scalableinformatics.com/jackrabbit
>>> phone: +1 734 786 8423 x121
>>> fax  : +1 866 888 3112
>>> cell : +1 734 612 4615
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>
>>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss



More information about the lustre-discuss mailing list