[Lustre-discuss] controlling which eth interface lustre uses
Brock Palen
brockp at umich.edu
Thu Oct 21 06:51:06 PDT 2010
On Oct 21, 2010, at 9:48 AM, Joe Landman wrote:
> On 10/21/2010 09:37 AM, Brock Palen wrote:
>> We recently added a new oss, it has 1 1Gb interface and 1 10Gb
>> interface,
>>
>> The 10Gb interface is eth4 10.164.0.166 The 1Gb interface is eth0
>> 10.164.0.10
>
> They look like they are on the same subnet if you are using /24 ...
You are correct
Both interfaces are on the same subnet:
[root at oss4-gb ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.164.0.0 * 255.255.248.0 U 0 0 0 eth0
10.164.0.0 * 255.255.248.0 U 0 0 0 eth4
169.254.0.0 * 255.255.0.0 U 0 0 0 eth4
default 10.164.0.1 0.0.0.0 UG 0 0 0 eth0
There is no way to mask the lustre service away from the 1Gb interface?
>
>>
>> In modprobe.conf I have:
>>
>> options lnet networks=tcp0(eth4)
>>
>> lctl list_nids 10.164.0.166 at tcp
>>
>>> From a host I run:
>>
>> lctl which_nid oss4 10.164.0.166 at tcp
>>
>> But yet I still see traffic over eth0 the 1Gb management network,
>> might higher than I would expect (upto 100MB/s) The management
>> interface is oss4-gb So If I do from a client:
>>
>> lctl which_nid oss4-gb 10.164.0.10 at tcp
>>
>> Why If I have netwroks=tcp0(eth4) and that list_nids showa only the
>> 10Gb interface, do I have so much traffic over the 1Gb interface?
>> There is some traffic on the 10Gb interface, but I would like to tell
>> lustre 'don't use the 1Gb interface'.
>
> If they are on the same subnet, its possible that the 1GbE sees the arp
> response first. And then its pretty much guaranteed to have the traffic
> go out that port.
>
> If your subnets are different, this shouldn't be the issue.
>
>>
>> Thanks!
>>
>> Brock Palen www.umich.edu/~brockp Center for Advanced Computing
>> brockp at umich.edu (734)936-1985
>>
>>
>>
>> _______________________________________________ Lustre-discuss
>> mailing list Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics Inc.
> email: landman at scalableinformatics.com
> web : http://scalableinformatics.com
> http://scalableinformatics.com/jackrabbit
> phone: +1 734 786 8423 x121
> fax : +1 866 888 3112
> cell : +1 734 612 4615
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
More information about the lustre-discuss
mailing list