[Lustre-discuss] Dual NICs issue -- How to enforce Lustre to use the second NIC

[Isaac Huang]

On Wed, Nov 11, 2009 at 04:07:39PM -0600, Daneil Goodman wrote:
>    Hello list,
>    By searching the archive, I found a similar message dated back in
>    January 2008 -- How do you make an MGS/OSS listen on 2 NICs? Looks like
>    there is no final solution and I am facing the similar situation and
>    need your help.
>    I am running centos 5 on both server (MGS, MDS and OSS are on same
>    node) and clients: 2.6.18-128.1.6.el5_lustre.1.8.0.1smp. To simplify
>    the issue, suppose the network is consist of one lustre server node and
>    two lustre client nodes. The server node has two NICs: eth0(100Mb) and
>    eth1(1Gb), each client node only has one NIC:eth0. The network layout
>    is as below.
>    Server node eth0: 72.203.10.1 (Public network)    <==> Switch1 <==>
>    Public node eth0:  72.203.10.2 (Public network)
>    Server node eth1: 192.168.10.1 (Internal network) <==> Switch2 <==>
>    Private node eth0: 192.168.10.2 (Internal network)
>    Both SELinux and Fireware are turned off. Public node does not know
>    Private node, but Private node do knows Public node.
>    The modprobe.conf likes the following:
>    On server: options lnet networks="tcp0(eth0),tcp1(eth1)"
>    On clients: options lnet networks=tcp  <--- since there is only one

I think you'd need to make clients in the 72.203.10.* network use tcp0
and clients in the 192.168.10.* tcp1. To create a uniform module
option that works across the whole cluster, 'ip2nets' is your friend:

options lnet 'ip2nets="tcp0(eth0) 72.203.10.*; tcp1(eth1) 192.168.10.[1-10]; tcp1(eth0) 192.168.10.[100-200]"

(assuming that servers are 192.168.10.[1-10] and clients are
192.168.10.[100-200].)

Isaac