[Lustre-discuss] controlling which eth interface lustre uses

Brock Palen brockp at umich.edu
Thu Oct 21 07:29:30 PDT 2010



> Why do you need both active?  If one is a backup to the other, then bond 
> them as a primary/backup pair, meaning only one will be active at at a 
> time, ie, your designated primary (unless it goes down).

We could do this, the 10Gb drivers have been such a pain for us we wanted to have a 'back door' management network to get to the box should we have issues with the 10Gb driver.

Oddly I ran:

ifconfig eth0 down 

and I could nolonger ping the box over the eth4 interface, I had to power cycle it form management.  Very odd.

> 
> bob
> 
> On 10/21/2010 9:51 AM, Brock Palen wrote:
>> On Oct 21, 2010, at 9:48 AM, Joe Landman wrote:
>> 
>>> On 10/21/2010 09:37 AM, Brock Palen wrote:
>>>> We recently added a new oss, it has 1 1Gb interface and 1 10Gb
>>>> interface,
>>>> 
>>>> The 10Gb interface is eth4 10.164.0.166 The 1Gb   interface is eth0
>>>> 10.164.0.10
>>> They look like they are on the same subnet if you are using /24 ...
>> You are correct
>> 
>> Both interfaces are on the same subnet:
>> 
>> [root at oss4-gb ~]# route
>> Kernel IP routing table
>> Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
>> 10.164.0.0      *               255.255.248.0   U     0      0        0 eth0
>> 10.164.0.0      *               255.255.248.0   U     0      0        0 eth4
>> 169.254.0.0     *               255.255.0.0     U     0      0        0 eth4
>> default         10.164.0.1      0.0.0.0         UG    0      0        0 eth0
>> 
>> There is no way to mask the lustre service away from the 1Gb interface?
>> 
>>>> In modprobe.conf I have:
>>>> 
>>>> options lnet networks=tcp0(eth4)
>>>> 
>>>> lctl list_nids 10.164.0.166 at tcp
>>>> 
>>>>> From a host I run:
>>>> lctl which_nid oss4 10.164.0.166 at tcp
>>>> 
>>>> But yet I still see traffic over eth0 the 1Gb management network,
>>>> might higher than I would expect (upto 100MB/s) The management
>>>> interface is oss4-gb  So If I do from a client:
>>>> 
>>>> lctl which_nid oss4-gb 10.164.0.10 at tcp
>>>> 
>>>> Why If I have netwroks=tcp0(eth4)  and that list_nids showa only the
>>>> 10Gb interface, do I have so much traffic over the 1Gb interface?
>>>> There is some traffic on the 10Gb interface, but I would like to tell
>>>> lustre 'don't use the 1Gb interface'.
>>> If they are on the same subnet, its possible that the 1GbE sees the arp
>>> response first.  And then its pretty much guaranteed to have the traffic
>>> go out that port.
>>> 
>>> If your subnets are different, this shouldn't be the issue.
>>> 
>>>> Thanks!
>>>> 
>>>> Brock Palen www.umich.edu/~brockp Center for Advanced Computing
>>>> brockp at umich.edu (734)936-1985
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________ Lustre-discuss
>>>> mailing list Lustre-discuss at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>> 
>>> -- 
>>> Joseph Landman, Ph.D
>>> Founder and CEO
>>> Scalable Informatics Inc.
>>> email: landman at scalableinformatics.com
>>> web  : http://scalableinformatics.com
>>>        http://scalableinformatics.com/jackrabbit
>>> phone: +1 734 786 8423 x121
>>> fax  : +1 866 888 3112
>>> cell : +1 734 612 4615
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>> 
>>> 
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>> 
>> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
> 




More information about the lustre-discuss mailing list