[Lustre-discuss] Lustre over two TCP interfaces

Alfonso Pardo alfonso.pardo at ciemat.es
Wed Jun 26 23:23:57 PDT 2013

The virtual inferface will be the OSS. A virtual IP will start in the first interface, and if the network attached to the first interface is failed, the virtual IP is move to the second interface.

I need have two networks, because I need to guarantee to my clients a network in HA and minimized the lustre downs for networks maintenance.

From: Michael Shuey 
Sent: Wednesday, June 26, 2013 2:17 PM
To: Alfonso Pardo 
Cc: Brian O'Connor ; WC-Discuss ; lustre-discuss at lists.lustre.org 
Subject: Re: [Lustre-discuss] Lustre over two TCP interfaces

That will probably be slow - the machine you use to proxy the IPVS address would be a bottleneck.  Out of curiosity, what problem are you trying to solve here?  Do you anticipate whole-subnet outages to be an issue (and if so, why)?

Mike Shuey

On Wed, Jun 26, 2013 at 4:53 AM, Alfonso Pardo <alfonso.pardo at ciemat.es> wrote:


  Thanks for you reply! May be another way is a floating IP between two interfaces with IPVS (corosync).

  -----Mensaje original----- From: Brian O'Connor
  Sent: Wednesday, June 26, 2013 10:15 AM
  To: Alfonso Pardo
  Cc: 'Michael Shuey' ; 'WC-Discuss' ; lustre-discuss at lists.lustre.org
  Subject: Re: [Lustre-discuss] Lustre over two TCP interfaces 

  On 06/26/2013 04:16 PM, Alfonso Pardo wrote:

    But.... if I configure the OST assigning to the first interface of the
    OSS (bond0) and as failover OSS the second inteface of the OSS. If the
    bond0 network down, the client will try to connect to the failover, that
    is the second interface of the OSS.
    is it possible?

  I stand to be corrected, but no, I don't think so. As I understand it
  the failover code looks for a different server instance, rather than a
  different nid.



    *From:* Brian O'Connor <mailto:briano at sgi.com>
    *Sent:* Wednesday, June 26, 2013 1:09 AM
    *To:* 'Alfonso Pardo' <mailto:alfonso.pardo at ciemat.es> ; 'Michael Shuey'
    <mailto:shuey at purdue.edu>
    *Cc:* 'WC-Discuss' <mailto:WC-Discuss.Migration at intel.com> ;
    mailto:lustre-discuss at lists.lustre.org
    *Subject:* RE: [Lustre-discuss] Lustre over two TCP interfaces
    Unless something has changed in the new versions of lustre, I don't
    think lustre can do failover between nids on the same machine.

    It can choose the available nid at mount time, but if an active nid goes
    away after you are mounted then the client chooses the failover nid, and
    this must be on a different server.

    Check the archives for more discussion in this topic :)

    -----Original Message-----
    *From: *Alfonso Pardo [alfonso.pardo at ciemat.es
    <mailto:alfonso.pardo at ciemat.es>]
    *Sent: *Tuesday, June 25, 2013 07:23 AM Central Standard Time
    *To: *Michael Shuey
    *Cc: *WC-Discuss; lustre-discuss at lists.lustre.org
    *Subject: *Re: [Lustre-discuss] Lustre over two TCP interfaces

    thank Michael,
    This is my second step, I will change the lnet with “options lnet
    networks=tcp0(bond0,bond1)” because my machines has 4 nics. I have a
    bond0 and bond1 with LACP. I need to comunicate the clients with two
    network for HA network.
    If the bond0 network is down, the clients can reach the OSS by the
    second network bond1.
    If I change the modprobe with “options lnet
    networks=tcp0(bond0),tcp1(bond1)”, how the clients mount the filesystem
    to reach the OSS by two network?
    *From:* Michael Shuey <mailto:shuey at purdue.edu>
    *Sent:* Tuesday, June 25, 2013 2:14 PM
    *To:* Alfonso Pardo <mailto:alfonso.pardo at ciemat.es>
    *Cc:* lustre-discuss at lists.lustre.org
    <mailto:lustre-discuss at lists.lustre.org> ; WC-Discuss
    <mailto:WC-Discuss.Migration at intel.com>
    *Subject:* Re: [Lustre-discuss] Lustre over two TCP interfaces
    Different interfaces need to be declared with different LNET networks -
    something like "networks=tcp0(eth0),tcp1(eth1)".  Of course, that
    assumes your clients are configured to use a mix of tcp0 and tcp1 for
    connections (with each client only using one of the two).  This is
    really only useful in corner cases, when you're doing something strange;
    if eth0 and eth1 are in the same subnet (as in your example), this is
    almost certainly not productive.
    A better bet might be to use a single LNET, and bond the two interfaces
    together - either as an active/passive pair, or active/active (e.g.,
    LACP).  Then you'd declare networks=tcp0(bond0), give the bond a single
    IP address, and client traffic would be split across the two members in
    the bond more like you probably expect (given the limits of the bond
    protocol you're using).
    Mike Shuey

    On Tue, Jun 25, 2013 at 8:06 AM, Alfonso Pardo <alfonso.pardo at ciemat.es
    <mailto:alfonso.pardo at ciemat.es>> wrote:

        hello friends,
        I need to comunicate my OSS by two ethernet TCP interfaces: eth0 and
        I have configured this feature in my modprobe.d with:
        “options lnet networks=tcp0(eth0,eth1)”
        And I can see two interfaces with:
        lctl --net tcp interface_list
        sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.es>:
        ( <>)
        npeer 0 nroute 2
        sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.es>:
        ( <>)
        npeer 0 nroute 0
        But, the clients only can communicate with the first interface:
        lctl ping
        12345-0 at lo
        12345- at tcp
        lctl ping
        failed to ping at tcp: Input/output error
        Any suggestions how to “enable” the second interface?
        thank in advance

        /Alfonso Pardo Diaz/
        /*System Administrator / Researcher*/
        /c/ Sola nº 1; 10200 TRUJILLO, SPAIN/
        /Tel: +34 927 65 93 17 <tel:%2B34%20927%2065%2093%2017> Fax: +34 927
        32 32 37/

        CETA-Ciemat logo <http://www.ceta-ciemat.es/>

        ---------------------------- Confidencialidad: Este mensaje y sus
        ficheros adjuntos se dirige exclusivamente a su destinatario y puede
        contener información privilegiada o confidencial. Si no es vd. el
        destinatario indicado, queda notificado de que la utilización,
        divulgación y/o copia sin autorización está prohibida en virtud de
        la legislación vigente. Si ha recibido este mensaje por error, le
        rogamos que nos lo comunique inmediatamente respondiendo al mensaje
        y proceda a su destrucción. Disclaimer: This message and its
        attached files is intended exclusively for its recipients and may
        contain confidential information. If you received this e-mail in
        error you are hereby notified that any dissemination, copy or
        disclosure of this communication is strictly prohibited and may be
        unlawful. In this case, please notify us by a reply and delete this
        email and its contents immediately. ----------------------------

        Lustre-discuss mailing list
        Lustre-discuss at lists.lustre.org <mailto:Lustre-discuss at lists.lustre.org>

  Brian O'Connor
  SGI Consulting
  Email: briano at sgi.com, Mobile +61 417 746 452
  Phone: +61 3 9963 1900, Fax:  +61 3 9963 1902
  691 Burke Road, Camberwell, Victoria, 3124

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130627/8d995d7e/attachment.htm>

More information about the lustre-discuss mailing list