Hello list,<br><br>By searching the archive, I found a similar message dated back in January 2008 -- How do you make an MGS/OSS listen on 2 NICs? Looks like there is no final solution and I am facing the similar situation and need your help.<br>
<br>I am running centos 5 on both server (MGS, MDS and OSS are on same node) and clients: 2.6.18-128.1.6.el5_lustre.1.8.0.1smp. To simplify the issue, suppose the network is consist of one lustre server node and two lustre client nodes. The server node has two NICs: eth0(100Mb) and eth1(1Gb), each client node only has one NIC:eth0. The network layout is as below.<br>
<br>Server node eth0: 72.203.10.1 (Public network) <==> Switch1 <==> Public node eth0: 72.203.10.2 (Public network)<br>Server node eth1: 192.168.10.1 (Internal network) <==> Switch2 <==> Private node eth0: 192.168.10.2 (Internal network)<br>
<br>Both SELinux and Fireware are turned off. Public node does not know Private node, but Private node do knows Public node.<br><br>The modprobe.conf likes the following:<br><br>On server: options lnet networks="tcp0(eth0),tcp1(eth1)"<br>
On clients: options lnet networks=tcp <--- since there is only one NIC, I did not specify it as tcp(eth0).<br><br>The procedure to build MGS, MDT and OST on server node as bellow:<br><br>[root@server ~]# mkfs.lustre --mgs /dev/sdb1<br>
[root@server ~]# mkdir /mgs<br>[root@server ~]# mount -t lustre /dev/sdb1 /mgs<br>[root@server ~]# mkfs.lustre --mdt --fsname=data --mgsnode="72.203.10.1@tcp0,192.168.10.1@tcp1" /dev/sdb2<br>[root@server ~]# mkdir /data<br>
[root@server ~]# mount -t lustre /dev/sdb2 /data<br>[root@server ~]# mkfs.lustre --ost --fsname=data --mgsnode="72.203.10.1@tcp0,192.168.10.1@tcp1" /dev/sdc1<br>[root@server ~]# mkdir /mnt/data<br>[root@server ~]# mount -t lustre /dev/sdc1 /mnt/data<br>
<br>On server node:<br>[root@server ~]# lctl list_nids <br>72.203.10.1@tcp<br>192.168.10.1@tcp1<br><br>[root@server ~]# lctl ping 72.203.10.1@tcp<br>12345-0@lo<br>12345-72.203.10.1@tcp<br>12345-192.168.10.1@tcp1<br><br>[root@server ~]# lctl ping 192.168.10.1@tcp1<br>
12345-0@lo<br>
12345-72.203.10.1@tcp<br>
12345-192.168.10.1@tcp1<br><br>Therefore, every thing on server node looks good, and I can mount /data on public node:<br>[root@public ~]# mount -t lustre 72.203.10.1@tcp:/data /data<br><br>Also I can mount /data on private node using server's eth0 (public NIC)<br>
[root@private ~]# mount -t lustre 72.203.10.1@tcp:/data /data<br><br>But I can not mount /data on private node using server's eth1 (private NIC). <br>[root@private ~]# mount -t lustre 192.168.10.1@tcp1:/data /data<br>
mount.lustre: /data inaccessible: No such file or directory<br><br>On private node:<br>[root@private ~]# lctl which_nid 192.168.10.1@tcp1<br>No reachable NID<br><br>[root@private ~]# lctl which_nid 192.168.10.1@tcp<br>192.168.10.1@tcp<br>
<br>[root@private ~]# lctl which_nid 72.203.10.1@tcp<br>
72.203.10.1@tcp<br><br>[root@private ~]# lctl ping 72.203.10.1@tcp<br>12345-0@lo<br>12345-72.203.10.1@tcp<br>12345-192.168.10.1@tcp1<br><br>We see the NID of 192.168.10.1@tcp from the private node, but can not ping it:<br>
[root@private ~]# lctl ping 192.168.10.1@tcp1<br>failed to ping 192.168.10.1@tcp1: Input/output error<br><br>[root@private ~]# lctl ping 192.168.10.1@tcp<br>
failed to ping 192.168.10.1@tcp: Input/output error<br><br>The error message in /var/log/messages are<br>LustreError: 27435:0:(lib-move.c:1265:lnet_send()) No route to 12345-192.168.10.1@tcp1<br>LustreError: 27435:0:(lib-move.c:2450:LNetGet()) error sending GET to 12345-192.168.10.1@tcp1: -113<br>
<br>But I can ping 192.168.10.1 using regular ping.<br><br>According to the response of Marc:<br>"As far as I know, LNET will use the shortest path on the network, so if you have two equivalent tcp networks, tcp0 and tcp1, LNET will just use the first one. If it fails, it should use the second one. If both NICs are in the same tcp network, LNET should use both.....". <br>
<br>Therefore, I can understand why I can mount /data on private node using server's eth0. But looks like LNET is not smart enough to pick up the "closer" NIC to mount the /data. <br><br>My questions are:<br>
1. Is something wrong of the eth1configuration?<br>2. How to enforce LNET to use the second NIC(eth1) to mount /data on private node? I prefer to use
eth1 not eth0 because eth1 connects to switch2 directly and fast (eth1-1Gb) than
eth0 (100Mb).<br><br>Thanks for reading the detailed procedure and appreciate for your help in advance,<br><br>Goodman<br><br><br>