On Wed, Nov 11, 2009 at 9:20 PM, Isaac Huang <span dir="ltr"><<a href="mailto:He.Huang@sun.com" target="_blank">He.Huang@sun.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>On Wed, Nov 11, 2009 at 04:07:39PM -0600, Daneil Goodman wrote:<br>
> Hello list,<br>
> By searching the archive, I found a similar message dated back in<br>
> January 2008 -- How do you make an MGS/OSS listen on 2 NICs? Looks like<br>
> there is no final solution and I am facing the similar situation and<br>
> need your help.<br>
> I am running centos 5 on both server (MGS, MDS and OSS are on same<br>
> node) and clients: 2.6.18-128.1.6.el5_lustre.1.8.0.1smp. To simplify<br>
> the issue, suppose the network is consist of one lustre server node and<br>
> two lustre client nodes. The server node has two NICs: eth0(100Mb) and<br>
> eth1(1Gb), each client node only has one NIC:eth0. The network layout<br>
> is as below.<br>
> Server node eth0: 72.203.10.1 (Public network) <==> Switch1 <==><br>
> Public node eth0: 72.203.10.2 (Public network)<br>
> Server node eth1: 192.168.10.1 (Internal network) <==> Switch2 <==><br>
> Private node eth0: 192.168.10.2 (Internal network)<br>
> Both SELinux and Fireware are turned off. Public node does not know<br>
> Private node, but Private node do knows Public node.<br>
> The modprobe.conf likes the following:<br>
> On server: options lnet networks="tcp0(eth0),tcp1(eth1)"<br>
> On clients: options lnet networks=tcp <--- since there is only one<br>
<br>
</div>I think you'd need to make clients in the 72.203.10.* network use tcp0<br>
and clients in the 192.168.10.* tcp1. To create a uniform module<br>
</blockquote><div><br>It did the trick! Thanks!<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">option that works across the whole cluster, 'ip2nets' is your friend:<br>
<br>
options lnet 'ip2nets="tcp0(eth0) 72.203.10.*; tcp1(eth1) 192.168.10.[1-10]; tcp1(eth0) 192.168.10.[100-200]"<br>
<br>
(assuming that servers are 192.168.10.[1-10] and clients are<br>
192.168.10.[100-200].)<br>
<font color="#888888"><br></font></blockquote><div><br>There are three small issues about ip2nets:<br><br>1. Looks like LNET does not like the single quotes 'ip2nets="tcp0(eth0) 72.203.10.*; tcp1(eth1) 192.168.10.*"'. It says<br>
<br>lnet: Unknown parameter `'ip2nets'<br><br>After removed single quotes, I can load lnet module.<br><br>2. According to my observation, on public network, to mount /data using above ip2nets option is slower than networks option.<br>
<br>3. On private network node, I cannot start LNET using ip2nets option<br>[root@private ~]# lsmod |grep lnet<br>lnet 273084 1 ksocklnd<br>libcfs 136180 2 ksocklnd,lnet<br>[root@private ~]# lctl network configure<br>
LNET configure error 100: Network is down<br><br>/var/log/messages shows:<br>LustreError: 31927:0:(socklnd.c:2545:ksocknal_startup()) Interface eth1 is down<br>LustreError: 105-4: Error -100 starting up LNI tcp<br><br>But if you use networks option (options lnet networks=tcp1), it works well. Do you think what is the problem?<br>
<br>Thanks,<br>Goodman<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><font color="#888888">
Isaacc<br>
</font></blockquote></div><br>