[Lustre-discuss] controlling which eth interface lustre uses

Wojciech Turek wjt27 at cam.ac.uk
Thu Oct 21 07:34:52 PDT 2010


Maybe I am missing a point here but can you explain me why would you need to
have two NICs in one host on the same subnet?
If you need additional access route to your host why not to configure eth0
on different subnet?

On 21 October 2010 15:29, Brock Palen <brockp at umich.edu> wrote:

>
>
> > Why do you need both active?  If one is a backup to the other, then bond
> > them as a primary/backup pair, meaning only one will be active at at a
> > time, ie, your designated primary (unless it goes down).
>
> We could do this, the 10Gb drivers have been such a pain for us we wanted
> to have a 'back door' management network to get to the box should we have
> issues with the 10Gb driver.
>
> Oddly I ran:
>
> ifconfig eth0 down
>
> and I could nolonger ping the box over the eth4 interface, I had to power
> cycle it form management.  Very odd.
>
> >
> > bob
> >
> > On 10/21/2010 9:51 AM, Brock Palen wrote:
> >> On Oct 21, 2010, at 9:48 AM, Joe Landman wrote:
> >>
> >>> On 10/21/2010 09:37 AM, Brock Palen wrote:
> >>>> We recently added a new oss, it has 1 1Gb interface and 1 10Gb
> >>>> interface,
> >>>>
> >>>> The 10Gb interface is eth4 10.164.0.166 The 1Gb   interface is eth0
> >>>> 10.164.0.10
> >>> They look like they are on the same subnet if you are using /24 ...
> >> You are correct
> >>
> >> Both interfaces are on the same subnet:
> >>
> >> [root at oss4-gb ~]# route
> >> Kernel IP routing table
> >> Destination     Gateway         Genmask         Flags Metric Ref    Use
> Iface
> >> 10.164.0.0      *               255.255.248.0   U     0      0        0
> eth0
> >> 10.164.0.0      *               255.255.248.0   U     0      0        0
> eth4
> >> 169.254.0.0     *               255.255.0.0     U     0      0        0
> eth4
> >> default         10.164.0.1      0.0.0.0         UG    0      0        0
> eth0
> >>
> >> There is no way to mask the lustre service away from the 1Gb interface?
> >>
> >>>> In modprobe.conf I have:
> >>>>
> >>>> options lnet networks=tcp0(eth4)
> >>>>
> >>>> lctl list_nids 10.164.0.166 at tcp
> >>>>
> >>>>> From a host I run:
> >>>> lctl which_nid oss4 10.164.0.166 at tcp
> >>>>
> >>>> But yet I still see traffic over eth0 the 1Gb management network,
> >>>> might higher than I would expect (upto 100MB/s) The management
> >>>> interface is oss4-gb  So If I do from a client:
> >>>>
> >>>> lctl which_nid oss4-gb 10.164.0.10 at tcp
> >>>>
> >>>> Why If I have netwroks=tcp0(eth4)  and that list_nids showa only the
> >>>> 10Gb interface, do I have so much traffic over the 1Gb interface?
> >>>> There is some traffic on the 10Gb interface, but I would like to tell
> >>>> lustre 'don't use the 1Gb interface'.
> >>> If they are on the same subnet, its possible that the 1GbE sees the arp
> >>> response first.  And then its pretty much guaranteed to have the
> traffic
> >>> go out that port.
> >>>
> >>> If your subnets are different, this shouldn't be the issue.
> >>>
> >>>> Thanks!
> >>>>
> >>>> Brock Palen www.umich.edu/~brockp <http://www.umich.edu/%7Ebrockp>Center for Advanced Computing
> >>>> brockp at umich.edu (734)936-1985
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________ Lustre-discuss
> >>>> mailing list Lustre-discuss at lists.lustre.org
> >>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >>>
> >>> --
> >>> Joseph Landman, Ph.D
> >>> Founder and CEO
> >>> Scalable Informatics Inc.
> >>> email: landman at scalableinformatics.com
> >>> web  : http://scalableinformatics.com
> >>>        http://scalableinformatics.com/jackrabbit
> >>> phone: +1 734 786 8423 x121
> >>> fax  : +1 866 888 3112
> >>> cell : +1 734 612 4615
> >>> _______________________________________________
> >>> Lustre-discuss mailing list
> >>> Lustre-discuss at lists.lustre.org
> >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >>>
> >>>
> >> _______________________________________________
> >> Lustre-discuss mailing list
> >> Lustre-discuss at lists.lustre.org
> >> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >>
> >>
> > _______________________________________________
> > Lustre-discuss mailing list
> > Lustre-discuss at lists.lustre.org
> > http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >
> >
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20101021/c50099fd/attachment.htm>


More information about the lustre-discuss mailing list