<div dir="ltr"><div><div>I am not able to picture your network architecture.<br></div><div>Would it possible for you to post a network diagram?<br></div><div><br></div>LACP is generally used between switches so that there is no looping issues when you have two switches cascaded using multiple cables. In such a case, only 1 cable is active and the second one takes over only when the first CABLE FAILS and not the switch it self.<br>
<br></div><div>Hope you haven't connected *just these 2 switches* using LACP. If so, you will have to change it as it won't give you desired effect. LACP is used for UPLINK cables and not for peer-to-peer connectivity.<br>
</div><div><br></div><div>With just 2 Switches, the right thing to do is 'stacking'. <br></div><div>And one shouldn't use LACP on a Server-Switch connectivity. <br>Normal non-LACP bonding is the correct way to do it.<br>
<br></div><div>Will be to tell you more if you could post a diagram. Even a rough one would do.<br><br></div><div><br><br>Indivar Nair<br></div><div><br><br></div><div><br><br><br></div></div><div class="gmail_extra"><br>
<br><div class="gmail_quote">On Thu, Jun 27, 2013 at 12:28 PM, Alfonso Pardo <span dir="ltr"><<a href="mailto:alfonso.pardo@ciemat.es" target="_blank">alfonso.pardo@ciemat.es</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div style="font-size:12pt;font-family:'Calibri'">
<div>I would have high-availability if I have a bonding mode 0,1 or 2. But
I have LACP bonding attached to the same switch, if I loose a switch, my OSS
will be down.</div>
<div style="font-style:normal;font-size:small;display:inline;text-decoration:none;font-family:'Calibri';font-weight:normal">
<div style="FONT:10pt tahoma">
<div> </div>
<div style="BACKGROUND:#f5f5f5">
<div><b>From:</b> <a title="indivar.nair@techterra.in" href="mailto:indivar.nair@techterra.in" target="_blank">Indivar Nair</a> </div>
<div><b>Sent:</b> Thursday, June 27, 2013 8:30 AM</div>
<div><b>To:</b> <a title="alfonso.pardo@ciemat.es" href="mailto:alfonso.pardo@ciemat.es" target="_blank">Alfonso Pardo</a> </div>
<div><b>Cc:</b> <a title="shuey@purdue.edu" href="mailto:shuey@purdue.edu" target="_blank">Michael
Shuey</a> ; <a title="lustre-discuss@lists.lustre.org" href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss</a> </div><div><div class="h5">
<div><b>Subject:</b> Re: [Lustre-discuss] Lustre over two TCP
interfaces</div></div></div></div></div>
<div> </div></div><div><div class="h5">
<div style="font-style:normal;font-size:small;display:inline;text-decoration:none;font-family:'Calibri';font-weight:normal">
<div dir="ltr">
<div>
<div>
<div>Then follow the instructions in my earlier mail.<br></div>
<div>No need to have bond0 and bond1.<br></div>You will achieve
high-availability even with one bonded
interface.<br><br></div>Cheers,<br><br><br></div>Indivar Nair<br></div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On Thu, Jun 27, 2013 at 11:55 AM, Alfonso Pardo <span dir="ltr"><<a href="mailto:alfonso.pardo@ciemat.es" target="_blank">alfonso.pardo@ciemat.es</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">
<div dir="ltr">
<div dir="ltr">
<div style="FONT-SIZE:12pt;FONT-FAMILY:'Calibri'">
<div>Yes I have two swtiches, one to the bond0 interface and other switch to
the second bond1 interface.</div>
<div> </div>
<div> </div>
<div style="FONT-SIZE:small;FONT-FAMILY:'Calibri';FONT-WEIGHT:normal;FONT-STYLE:normal;TEXT-DECORATION:none;DISPLAY:inline">
<div style="FONT:10pt tahoma">
<div> </div>
<div style="BACKGROUND:#f5f5f5">
<div><b>From:</b> <a title="indivar.nair@techterra.in" href="mailto:indivar.nair@techterra.in" target="_blank">Indivar Nair</a> </div>
<div><b>Sent:</b> Wednesday, June 26, 2013 8:05 PM</div>
<div><b>To:</b> <a title="alfonso.pardo@ciemat.es" href="mailto:alfonso.pardo@ciemat.es" target="_blank">Alfonso Pardo</a> </div>
<div><b>Cc:</b> <a title="shuey@purdue.edu" href="mailto:shuey@purdue.edu" target="_blank">Michael Shuey</a> ; <a title="WC-Discuss.Migration@intel.com" href="mailto:WC-Discuss.Migration@intel.com" target="_blank">WC-Discuss</a> ; <a title="lustre-discuss@lists.lustre.org" href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss</a>
</div>
<div>
<div>
<div><b>Subject:</b> Re: [Lustre-discuss] Lustre over two TCP
interfaces</div></div></div></div></div>
<div> </div></div>
<div>
<div>
<div style="FONT-SIZE:small;FONT-FAMILY:'Calibri';FONT-WEIGHT:normal;FONT-STYLE:normal;TEXT-DECORATION:none;DISPLAY:inline">
<div dir="ltr">
<div>
<div>
<div>Hi Alfonso,<br><br></div>I guess, you have two switches, with 2
interfaces (bond0) connected to one switch and the other 2 interfaces (bond1)
to the second switch.<br><br>---<br><br></div>What you need to do is merge the
switches using a 'stacking' cable (if the switches are stackable) and create a
single trunk using 2 ports from each switch.<br>Then create a single bond on
the Linux side using all the 4 Interfaces (and have just 1 IP).<br><br></div>
<div>Use bonding mode balance-rr or 0 without LACP to get load balancing
across all the 4 NICs.<br></div>
<div> </div>If the switches aren't stackable and a single trunk cannot be
created on the switch side, then use bonding mode balance-alb or 6 on the
Linux side.<br>
<div><br>No changes need to be done to the cabling in either
case.<br><br>---<br><br></div>
<div>This way you get Load Balancing and H/A across NICs.<br></div>
<div><br><br><br>Indivar Nair<br><br><br></div></div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On Wed, Jun 26, 2013 at 5:47 PM, Michael Shuey <span dir="ltr"><<a href="mailto:shuey@purdue.edu" target="_blank">shuey@purdue.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">
<div>That will probably be slow - the machine you use to proxy the IPVS
address would be a bottleneck. Out of curiosity, what problem are you
trying to solve here? Do you anticipate whole-subnet outages to be an
issue (and if so, why)?</div>
<div> </div>
<div>--<br>Mike Shuey</div>
<div>
<div><br><br>
<div class="gmail_quote">On Wed, Jun 26, 2013 at 4:53 AM, Alfonso Pardo <span dir="ltr"><<a href="mailto:alfonso.pardo@ciemat.es" target="_blank">alfonso.pardo@ciemat.es</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">oooh!<br><br><br>Thanks
for you reply! May be another way is a floating IP between two interfaces
with IPVS (corosync).<br><br>-----Mensaje original----- From: Brian
O'Connor<br>Sent: Wednesday, June 26, 2013 10:15 AM<br>To: Alfonso
Pardo<br>Cc: 'Michael Shuey' ; 'WC-Discuss' ; <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.<u></u>org</a><br>Subject: Re:
[Lustre-discuss] Lustre over two TCP interfaces
<div>
<div><br><br><br><br><br>On 06/26/2013 04:16 PM, Alfonso Pardo wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">But....
if I configure the OST assigning to the first interface of the<br>OSS
(bond0) and as failover OSS the second inteface of the OSS. If
the<br>bond0 network down, the client will try to connect to the
failover, that<br>is the second interface of the OSS.<br>is it
possible?<br></blockquote><br><br>I stand to be corrected, but no, I don't
think so. As I understand it<br>the failover code looks for a different
server instance, rather than a<br>different nid.<br><br>See<br><br><a href="http://lists.opensfs.org/pipermail/lustre-devel-opensfs.org/2012-August/000028.html" target="_blank">http://lists.opensfs.org/<u></u>pipermail/lustre-devel-<u></u>opensfs.org/2012-August/<u></u>000028.html</a><br>
<br><br>
<blockquote class="gmail_quote" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid">*From:*
Brian O'Connor <mailto:<a href="mailto:briano@sgi.com" target="_blank">briano@sgi.com</a>><br>*Sent:* Wednesday, June 26, 2013
1:09 AM<br>*To:* 'Alfonso Pardo' <mailto:<a href="mailto:alfonso.pardo@ciemat.es" target="_blank">alfonso.pardo@ciemat.<u></u>es</a>> ; 'Michael
Shuey'<br><mailto:<a href="mailto:shuey@purdue.edu" target="_blank">shuey@purdue.edu</a>><br>*Cc:* 'WC-Discuss'
<mailto:<a href="mailto:WC-Discuss.Migration@intel.com" target="_blank">WC-Discuss.Migration@<u></u>intel.com</a>>
;<br>mailto:<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.<u></u>lustre.org</a><br>*Subject:*
RE: [Lustre-discuss] Lustre over two TCP interfaces<br>Unless something
has changed in the new versions of lustre, I don't<br>think lustre can
do failover between nids on the same machine.<br><br>It can choose the
available nid at mount time, but if an active nid goes<br>away after you
are mounted then the client chooses the failover nid, and<br>this must
be on a different server.<br><br>Check the archives for more discussion
in this topic :)<br><br><br><br>-----Original Message-----<br>*From:
*Alfonso Pardo [<a href="mailto:alfonso.pardo@ciemat.es" target="_blank">alfonso.pardo@ciemat.es</a><br><mailto:<a href="mailto:alfonso.pardo@ciemat.es" target="_blank">alfonso.pardo@ciemat.<u></u>es</a>>]<br>
*Sent:
*Tuesday, June 25, 2013 07:23 AM Central Standard Time<br>*To: *Michael
Shuey<br>*Cc: *WC-Discuss; <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.<u></u>org</a><br>*Subject:
*Re: [Lustre-discuss] Lustre over two TCP interfaces<br><br>thank
Michael,<br>This is my second step, I will change the lnet with “options
lnet<br>networks=tcp0(bond0,bond1)” because my machines has 4 nics. I
have a<br>bond0 and bond1 with LACP. I need to comunicate the clients
with two<br>network for HA network.<br>If the bond0 network is down, the
clients can reach the OSS by the<br>second network bond1.<br>If I change
the modprobe with “options
lnet<br>networks=tcp0(bond0),tcp1(<u></u>bond1)”, how the clients mount
the filesystem<br>to reach the OSS by two network?<br>*From:* Michael
Shuey <mailto:<a href="mailto:shuey@purdue.edu" target="_blank">shuey@purdue.edu</a>><br>*Sent:* Tuesday, June 25, 2013
2:14 PM<br>*To:* Alfonso Pardo <mailto:<a href="mailto:alfonso.pardo@ciemat.es" target="_blank">alfonso.pardo@ciemat.<u></u>es</a>><br>*Cc:* <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.<u></u>org</a><br>
<mailto:<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.<u></u>lustre.org</a>> ;
WC-Discuss<br><mailto:<a href="mailto:WC-Discuss.Migration@intel.com" target="_blank">WC-Discuss.Migration@<u></u>intel.com</a>><br>*Subject:*
Re: [Lustre-discuss] Lustre over two TCP interfaces<br>Different
interfaces need to be declared with different LNET networks
-<br>something like "networks=tcp0(eth0),tcp1(<u></u>eth1)". Of
course, that<br>assumes your clients are configured to use a mix of tcp0
and tcp1 for<br>connections (with each client only using one of the
two). This is<br>really only useful in corner cases, when you're
doing something strange;<br>if eth0 and eth1 are in the same subnet (as
in your example), this is<br>almost certainly not productive.<br>A
better bet might be to use a single LNET, and bond the two
interfaces<br>together - either as an active/passive pair, or
active/active (e.g.,<br>LACP). Then you'd declare
networks=tcp0(bond0), give the bond a single<br>IP address, and client
traffic would be split across the two members in<br>the bond more like
you probably expect (given the limits of the bond<br>protocol you're
using).<br>--<br>Mike Shuey<br><br><br>On Tue, Jun 25, 2013 at 8:06 AM,
Alfonso Pardo <<a href="mailto:alfonso.pardo@ciemat.es" target="_blank">alfonso.pardo@ciemat.es</a><br><mailto:<a href="mailto:alfonso.pardo@ciemat.es" target="_blank">alfonso.pardo@ciemat.<u></u>es</a>>>
wrote:<br><br> hello friends,<br> I
need to comunicate my OSS by two ethernet TCP interfaces: eth0
and<br> eth1.<br> I have configured
this feature in my modprobe.d with:<br> “options lnet
networks=tcp0(eth0,eth1)”<br> And I can see two
interfaces with:<br> lctl --net tcp
interface_list<br> <a href="http://sa-d4-01.ceta-ciemat.es" target="_blank">sa-d4-01.ceta-ciemat.es</a> <<a href="http://sa-d4-01.ceta-ciemat.es" target="_blank">http://sa-d4-01.ceta-ciemat.<u></u>es</a>>:<br>
(<a href="http://192.168.11.15/255.255.255.0" target="_blank">192.168.11.15/255.255.255.0</a> <<a href="http://192.168.11.15/255.255.255.0" target="_blank">http://192.168.11.15/255.255.<u></u>255.0</a>>)<br>
npeer 0 nroute 2<br> <a href="http://sa-d4-01.ceta-ciemat.es" target="_blank">sa-d4-01.ceta-ciemat.es</a> <<a href="http://sa-d4-01.ceta-ciemat.es" target="_blank">http://sa-d4-01.ceta-ciemat.<u></u>es</a>>:<br>
(<a href="http://192.168.11.35/255.255.255.0" target="_blank">192.168.11.35/255.255.255.0</a> <<a href="http://192.168.11.35/255.255.255.0" target="_blank">http://192.168.11.35/255.255.<u></u>255.0</a>>)<br>
npeer 0 nroute 0<br> But, the clients only can
communicate with the first interface:<br> lctl ping
192.168.11.15<br> 12345-0@lo<br>
12345-192.168.11.15@tcp<br> lctl ping
192.168.11.35<br> failed to ping 192.168.11.35@tcp:
Input/output error<br> Any suggestions how to “enable”
the second interface?<br> thank in
advance<br><br> /Alfonso Pardo
Diaz/<br> /*System Administrator /
Researcher*/<br> /c/ Sola nº 1; 10200 TRUJILLO,
SPAIN/<br> /Tel: <a href="tel:%2B34%20927%2065%2093%2017" value="+34927659317" target="_blank">+34 927 65 93 17</a>
<tel:%2B34%20927%2065%2093%<u></u>2017> Fax: +34
927<br> 32 32 37/<br><br>
CETA-Ciemat logo <<a href="http://www.ceta-ciemat.es/" target="_blank">http://www.ceta-ciemat.es/</a>><br><br>
---------------------------- Confidencialidad: Este mensaje y
sus<br> ficheros adjuntos se dirige exclusivamente a
su destinatario y puede<br> contener información
privilegiada o confidencial. Si no es vd. el<br>
destinatario indicado, queda notificado de que la
utilización,<br> divulgación y/o copia sin
autorización está prohibida en virtud de<br> la
legislación vigente. Si ha recibido este mensaje por error,
le<br> rogamos que nos lo comunique inmediatamente
respondiendo al mensaje<br> y proceda a su
destrucción. Disclaimer: This message and its<br>
attached files is intended exclusively for its recipients and
may<br> contain confidential information. If you
received this e-mail in<br> error you are hereby
notified that any dissemination, copy or<br>
disclosure of this communication is strictly prohibited and may
be<br> unlawful. In this case, please notify us by a
reply and delete this<br> email and its contents
immediately. ----------------------------<br><br>
______________________________<u></u>_________________<br>
Lustre-discuss mailing list<br> <a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.<u></u>org</a> <mailto:<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.<u></u>lustre.org</a>><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/<u></u>mailman/listinfo/lustre-<u></u>discuss</a><br><br></blockquote><br><br>--
<br>Brian
O'Connor<br>------------------------------<u></u>------------------------------<u></u>-<br>SGI
Consulting<br>Email: <a href="mailto:briano@sgi.com" target="_blank">briano@sgi.com</a>, Mobile <a href="tel:%2B61%20417%20746%20452" value="+61417746452" target="_blank">+61
417 746 452</a><br>Phone: <a href="tel:%2B61%203%209963%201900" value="+61399631900" target="_blank">+61 3 9963 1900</a>, Fax: <a href="tel:%2B61%203%209963%201902" value="+61399631902" target="_blank">+61
3 9963 1902</a><br>691 Burke Road, Camberwell, Victoria,
3124<br>AUSTRALIA<br><a href="http://www.sgi.com/support/services" target="_blank">http://www.sgi.com/support/<u></u>services</a><br>------------------------------<u></u>------------------------------<u></u>-
<br></div></div></blockquote></div>
<div> </div></div></div><br>_______________________________________________<br>Lustre-discuss
mailing list<br><a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br></blockquote></div>
<div> </div></div></div></div></div></div></div></div></blockquote></div>
<div> </div></div></div></div></div></div></div></div>
</blockquote></div><br></div>