[Lustre-discuss] Problems with lustre router setup IB <-> TCP

[Thomas Roth]

Hi Markus,

I have a working test setup of this, seems it just depends on having the
correct modprobe-confs for the involved machines.

On the MDS/OSS we have /etc/modprobe.d/lustre.conf with
> options lnet networks=o2ib0(ib0) routes="tcp1 192.168.0.[3-4]@o2ib0"
dead_router_check_interval=300
where 192.168.0.[3-4] are the (IBo)IPs of the two LNET-routers we installed.

On the LNET-router box, we have
> options lnet forwarding="enabled"
> options lnet accept=all
> options lnet networks="o2ib0(ib0),tcp1(eth0)" accept_timeout=15

And on the clients in the ethernet part, we have
> options lnet networks=tcp1(eth0) routes="o2ib 10.1.1.[15,18]@tcp1"
dead_router_check_interval=300
where 10.1.1.[15,18] are the IPs of the LNET routers on the ethernet side.

Cheers,
Thomas

On 12/19/2011 02:46 PM, Markus Kirsch wrote:
> Hello,
> 
>  
> 
> I have a big problem finding an example how to setup a lustre-router.
> 
> Maybe someone can direct me to the right direction.
> 
>  
> 
> Inside the server room we have a lustre storage system with 1 combined
> MDS/OSS node and 3 OSS nodes.
> 
> Around 100 compute nodes connect to this lustre storage over IB. So
> right now all lnet communication is done
> 
> over IB. We have to connect now a small number of workstations outside
> the server room to the lustre storage system.
> 
> I would prefer to do this with a lustre router, instead of enabling
> communication for the whole system over IB and TCP.
> 
>  
> 
>  
> ##################################################
> #                server room                     #  
> # +---------+                                    #
> # | MDS/OSS |                                    #
> # +---------+                                    #
> # |   OSS   |\                                   #                  
> +-------------+
> # +---------+ \                                  #                   |
> workstation |
> # |   OSS   |  \                                 #                 
> /+-------------+
> # +---------+   \+-----------+      +--------+   #   +------------+/
> # |   OSS   |    | IB-switch |--IB--| router |--TCP--| TCP-switch |
> # +---------+   /+-----------+      +--------+   #   +------------+\
> #              /                                 #                 
> \+-------------+
> # +---------+ /                                  #                   |
> workstation |
> # | clients |/                                   #                  
> +-------------+
> # +---------+                                    #
> #                                                #
> ##################################################
>  
> Markus Kirsch
> Consulting
>  
> ict ag - Aachen
> information · communication · technology
> a Fujitsu Company
> Kackertstraße 20
> 52072 Aachen
> www.ictag.net
> www.hpcline.de
> www.ictag-shop.de
>  
> Tel: +49 241 88949 156
> Fax: +49 241 88949 110
> Mobil: +49 177 88949 22
> Mail: mak at ictag.net
>  
> 
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

-- 
--------------------------------------------------------------------
Thomas Roth
Department: Informationstechnologie
Location: SB3 1.262
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986

GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1
64291 Darmstadt
www.gsi.de

Gesellschaft mit beschränkter Haftung
Sitz der Gesellschaft: Darmstadt
Handelsregister: Amtsgericht Darmstadt, HRB 1528

Geschäftsführung: Professor Dr. Dr. h.c. Horst Stöcker,
Dr. Hartmut Eickhoff

Vorsitzende des Aufsichtsrates: Dr. Beatrix Vierkorn-Rudolph
Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt