OK here's where I am now.<div><br></div><div>The public client can ping the routers public address but not the private address.</div><div><br></div><div><div>[root@routed-client lnet]$ cat /etc/modprobe.conf</div><div>
alias eth0 e1000</div><div>alias eth1 e1000</div><div>alias scsi_hostadapter megaraid_mbox</div><div>alias scsi_hostadapter1 ata_piix</div><div># eth0 is part of tcp1 (NYU-NET)</div><div># In order to get to tcp (Cluster private), use the network on</div>
<div># 128.122.X.Y at tcp1</div><div>options lnet accept=all</div><div>options lnet networks=tcp1(eth0) routes="tcp 128.122.X.Y@tcp1"</div><div><br></div><div>[root@routed-client lnet]$ lctl network up</div><div>
LNET configured</div><div><br></div><div>[root@routed-client lnet]$ cat /proc/sys/lnet/routes</div><div>Routing disabled</div><div>net hops state router</div><div>tcp 1 up 128.122.109.28@tcp1</div><div>
<br></div><div>[root@routed-client lnet]$ cat /proc/sys/lnet/routers</div><div>ref rtr_ref alive_cnt state last_ping router</div><div>3 1 0 up 0 128.122.109.28@tcp1</div><div><br></div>
<div>[root@routed-client lnet]$ lctl ping 128.122.109.28@tcp1</div><div>12345-0@lo</div><div>12345-10.1.255.247@tcp</div><div>12345-128.122.109.28@tcp1</div><div><br></div><div>[root@routed-client lnet]$ lctl ping 10.1.255.252@tcp</div>
<div>failed to ping 10.1.255.252@tcp: Input/output error</div><div><br></div>I can see traffic between the routed-client and the router as well as between the router and the MGS/MDS (10.1.255.252@tcp)</div><div><br></div>
<div>The mgs has the following config.</div><div><br></div><div><div>[root@mgs-0-0 lnet]# cat /etc/modprobe.conf</div><div>alias scsi_hostadapter mptbase</div><div>alias scsi_hostadapter1 mptsas</div><div>alias scsi_hostadapter2 usb-storage</div>
<div>alias eth0 e1000</div><div>alias eth1 e1000</div><div>alias eth2 e1000</div><div>alias eth3 e1000</div><div>options lnet forwarding="enabled"</div><div>options lnet accept=all</div><div>options lnet networks=tcp(eth0) routes="tcp1 10.1.255.247@tcp"</div>
<div><br></div><div>[root@mgs-0-0 lnet]# lctl network up</div><div>LNET configured</div><div><br></div><div>But it doesn't see any routes or routers.</div><div><div><br></div><div>[root@mgs-0-0 lnet]# cat /proc/sys/lnet/routes</div>
<div>Routing disabled</div><div>net hops state router</div><div><br></div><div>[root@mgs-0-0 lnet]# cat /proc/sys/lnet/routers</div><div>ref rtr_ref alive_cnt state last_ping router</div><div><br></div><div>And this is what /var/log/messages and dmesg contain with or without enabling neterror logging</div>
</div><div><div>Jun 11 22:41:07 mgs-0-0 kernel: LustreError: 10869:0:(lib-move.c:1250:lnet_send()) No route to 12345-128.122.X.Y@tcp1</div><div>Jun 11 22:41:07 mgs-0-0 kernel: LustreError: 10869:0:(lib-move.c:1723:lnet_parse_get()) 10.1.255.252@tcp: Unable to send REPLY for GET from 12345-128.122.X.Y@tcp1: -113</div>
<div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_quote">On Fri, Jun 5, 2009 at 12:48 PM, Isaac Huang <span dir="ltr"><<a href="mailto:He.Huang@sun.com">He.Huang@sun.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">On Thu, Jun 04, 2009 at 01:59:48PM -0400, Erik Froese wrote:<br>
> Thanks Andreas and Natalie,<br>
><br>
> I've made the changes you suggested (setting tcp1 as the external<br>
> network) and I'm able to lctl ping the 128.122.x.y address but I still<br>
> cannot ping the private address for the MDS.<br>
<br>
</div>Please show us the commands you've run and their outputs, together<br>
with error messages in dmesg. It'd help to "echo +neterror ><br>
/proc/sys/lnet/printk" before running the commands.<br>
<div class="im"><br>
> Could the problem be that the lustre fs on the private network is<br>
> actually called tcp and not tcp0? Are those synonymous?<br>
<br>
</div>No, 'tcp' is just a shorthand for 'tcp0' - they are 100% equivalent<br>
to each other.<br>
<font color="#888888"><br>
Isaac<br>
</font></blockquote></div><br></div>