[Lustre-discuss] controlling which eth interface lustre uses

Brock Palen brockp at umich.edu
Thu Oct 21 06:51:06 PDT 2010


On Oct 21, 2010, at 9:48 AM, Joe Landman wrote:

> On 10/21/2010 09:37 AM, Brock Palen wrote:
>> We recently added a new oss, it has 1 1Gb interface and 1 10Gb
>> interface,
>> 
>> The 10Gb interface is eth4 10.164.0.166 The 1Gb   interface is eth0
>> 10.164.0.10
> 
> They look like they are on the same subnet if you are using /24 ...

You are correct 

Both interfaces are on the same subnet:

[root at oss4-gb ~]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.164.0.0      *               255.255.248.0   U     0      0        0 eth0
10.164.0.0      *               255.255.248.0   U     0      0        0 eth4
169.254.0.0     *               255.255.0.0     U     0      0        0 eth4
default         10.164.0.1      0.0.0.0         UG    0      0        0 eth0

There is no way to mask the lustre service away from the 1Gb interface?  

> 
>> 
>> In modprobe.conf I have:
>> 
>> options lnet networks=tcp0(eth4)
>> 
>> lctl list_nids 10.164.0.166 at tcp
>> 
>>> From a host I run:
>> 
>> lctl which_nid oss4 10.164.0.166 at tcp
>> 
>> But yet I still see traffic over eth0 the 1Gb management network,
>> might higher than I would expect (upto 100MB/s) The management
>> interface is oss4-gb  So If I do from a client:
>> 
>> lctl which_nid oss4-gb 10.164.0.10 at tcp
>> 
>> Why If I have netwroks=tcp0(eth4)  and that list_nids showa only the
>> 10Gb interface, do I have so much traffic over the 1Gb interface?
>> There is some traffic on the 10Gb interface, but I would like to tell
>> lustre 'don't use the 1Gb interface'.
> 
> If they are on the same subnet, its possible that the 1GbE sees the arp 
> response first.  And then its pretty much guaranteed to have the traffic 
> go out that port.
> 
> If your subnets are different, this shouldn't be the issue.
> 
>> 
>> Thanks!
>> 
>> Brock Palen www.umich.edu/~brockp Center for Advanced Computing
>> brockp at umich.edu (734)936-1985
>> 
>> 
>> 
>> _______________________________________________ Lustre-discuss
>> mailing list Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
> 
> -- 
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics Inc.
> email: landman at scalableinformatics.com
> web  : http://scalableinformatics.com
>        http://scalableinformatics.com/jackrabbit
> phone: +1 734 786 8423 x121
> fax  : +1 866 888 3112
> cell : +1 734 612 4615
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
> 




More information about the lustre-discuss mailing list