<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.6000.16587" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><SPAN class=541061919-07032008><FONT face=Arial
color=#0000ff size=2>Taking note from another thread, I ran some ping tests
between my machines in rack b to their peers in both rack b and rack
a.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=541061919-07032008><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=541061919-07032008><FONT face=Arial
color=#0000ff size=2>The machines in rack b can ping those in a, but not those
in b. I get the following:</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=541061919-07032008></SPAN><SPAN
class=541061919-07032008><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=541061919-07032008><FONT face=Arial
color=#0000ff size=2>[root@dintnyc1304 ~]# lctl ping <A
href="mailto:4.23.36.37@tcp0">4.23.36.37@tcp0</A><BR>failed to ping <A
href="mailto:4.23.36.37@tcp">4.23.36.37@tcp</A>: Input/output
error<BR>[root@dintnyc1304 ~]# lctl ping <A
href="mailto:4.23.36.38@tcp0">4.23.36.38@tcp0</A><BR>failed to ping <A
href="mailto:4.23.36.38@tcp">4.23.36.38@tcp</A>: Input/output
error<BR>[root@dintnyc1304 ~]# lctl ping <A
href="mailto:4.23.36.39@tcp0">4.23.36.39@tcp0</A><BR>failed to ping <A
href="mailto:4.23.36.39@tcp">4.23.36.39@tcp</A>: Input/output
error<BR>[root@dintnyc1304 ~]# lctl ping <A
href="mailto:4.23.36.40@tcp0">4.23.36.40@tcp0</A><BR><A
href="mailto:12345-0@lo">12345-0@lo</A><BR><A
href="mailto:12345-4.23.36.40@tcp">12345-4.23.36.40@tcp</A><BR>[root@dintnyc1304
~]# lctl ping <A
href="mailto:4.23.36.41@tcp0">4.23.36.41@tcp0</A><BR>failed to ping <A
href="mailto:4.23.36.41@tcp">4.23.36.41@tcp</A>: Input/output
error<BR></FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=541061919-07032008><FONT face=Arial
color=#0000ff size=2>The one that works above, is the the machine that I ran the
ping from (It can ping itself).</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=541061919-07032008><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=541061919-07032008><FONT face=Arial
color=#0000ff size=2>--</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=541061919-07032008><FONT face=Arial
color=#0000ff size=2>Andrew</DIV></FONT></SPAN><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> lustre-discuss-bounces@lists.lustre.org
[mailto:lustre-discuss-bounces@lists.lustre.org] <B>On Behalf Of </B>Lundgren,
Andrew<BR><B>Sent:</B> Friday, March 07, 2008 12:03 PM<BR><B>To:</B>
'Lustre-discuss@clusterfs.com'<BR><B>Subject:</B> [Lustre-discuss] Odd
connectivity problem...<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>I am
trying to bring up our first "real" lustre cluster and running into an
unexpected problem.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Let me
dump out some background on the configuration first:</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>I am
running CentOS 5.0 x86_64bit with Lustre 1.6.4.2.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>I have 2
racks of machines located next to each other. Each rack has its own
router. </FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Each
router has its own subnet. </FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Each
machine has 2 NICs one that is connected into the local rack, and one
connected into adjacent rack.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Each
machine has local disks in it that serve as the OST
storage.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Each
machine has a lustre client running on it with visibility into the lustre
cluster.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008></SPAN><SPAN
class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>We are not
using bonding, so each machine has visibility into each subnet via the cross
connecting. </FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>We have
this options lnet networks=tcp0(eth1,eth0) in the modprobe.conf
file.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>We are not
yet using a routing protocol yet. The default route out for each machine
it's local router connected to eth0.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>ASCII
Pictures:</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Machines
in Rack A are all wired like this:</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>RACK
A
RACK B </FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2>--------------------
----------------------</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>|Router
(4.23.37.1)| <SPAN
class=161100918-07032008><FONT face="Courier New" size=2>|Router A
(4.23.36.1)|</FONT></SPAN></FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2>-------------------- ---------------------</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2> |
eth0 (4.23.37.10) |</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2><SPAN
class=161100918-07032008><FONT face="Courier New"
size=2>----------</FONT></SPAN>
|</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008></SPAN><SPAN
class=161100918-07032008><FONT face="Courier New" size=2>| Machine
|--eth1--(4.23.36.42)-|</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>----------
def gw (4.23.37.1)</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>
</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Machines
in Rack B are all wired like
this:</FONT></SPAN></DIV></FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>RACK
A
RACK B </FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2>--------------------
----------------------</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>|Router
(4.23.37.1)| <SPAN
class=161100918-07032008><FONT face="Courier New" size=2>|Router A
(4.23.36.1)|</FONT></SPAN></FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2>-------------------- ---------------------</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2> |
| eth0 (4.23.36.10)</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2><SPAN
class=161100918-07032008><FONT face="Courier New"
size=2> |
----------</FONT></SPAN> </FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2> --eth1--(4.23.37.42)--|
Machine |</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008></SPAN><SPAN
class=161100918-07032008><FONT face="Courier New"
size=2> def gw
(4.23.36.1) ----------</FONT></SPAN></DIV></FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face=Arial size=2><FONT
face="Courier New"></FONT></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face=Arial size=2><FONT
face="Courier New">The primary MGS is at the top of one of the
racks (RACK A for this email.)</FONT></FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>The
machines are all up and running with lustre on them, they can ping/ssh between
each other.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>No for the
problem, All of the clients in rack A, the rack with the MGS can see all
of the OSTs being served from all of the OSSs in both
racks.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>The
clients in rack B can only see the MGS and the OSSes in rack A, none in their
own rack, rack B. They all show timeouts like this:</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT size=2><FONT
face="Courier New">Mar 7 18:43:31 dintnyc1303 kernel: LustreError:
2914:0:(events.c:55:request_out_callback()) Skipped 275 previous similar
messages<BR>Mar 7 18:45:41 dintnyc1303 kernel: LustreError:
2914:0:(client.c:975:ptlrpc_expire_one_request()) @@@ timeout (sent at
1204915536, 5s ago) </FONT><A href="mailto:req@ffff81020327f800"><FONT
face="Courier New">req@ffff81020327f800</FONT></A><FONT face="Courier New">
x123942/t0 o8->content-OST001c_UUID@4.23.36.41@tcp:6 lens 240/272 ref 1 fl
Rpc:/0/0 rc 0/-22<BR>Mar 7 18:45:41 dintnyc1303 kernel: LustreError:
2914:0:(client.c:975:ptlrpc_expire_one_request()) Skipped 403 previous similar
messages<BR>Mar 7 18:53:31 dintnyc1303 kernel: LustreError:
2914:0:(events.c:55:request_out_callback()) @@@ type 4, status -5
</FONT><A href="mailto:req@ffff810203276000"><FONT
face="Courier New">req@ffff810203276000</FONT></A><FONT face="Courier New">
x124871/t0 o8->content-OST002d_UUID@4.23.36.46@tcp:6 lens 240/272 ref 2 fl
Rpc:/0/0 rc 0/-22<BR>Mar 7 18:53:31 dintnyc1303 kernel: LustreError:
2914:0:(events.c:55:request_out_callback()) Skipped 278 previous similar
messages<BR>Mar 7 18:56:01 dintnyc1303 kernel: LustreError:
2914:0:(client.c:975:ptlrpc_expire_one_request()) @@@ network error (sent at
1204916161, 0s ago) </FONT><A href="mailto:req@ffff8101f84dae00"><FONT
face="Courier New">req@ffff8101f84dae00</FONT></A><FONT face="Courier New">
x125140/t0 o8->content-OST001a_UUID@4.23.36.40@tcp:6 lens 240/272 ref 1 fl
Rpc:/0/0 rc 0/-22<BR>Mar 7 18:56:01 dintnyc1303 kernel: LustreError:
2914:0:(client.c:975:ptlrpc_expire_one_request()) Skipped 445 previous similar
messages</FONT></FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>I am a bit
stuck.</FONT></SPAN></DIV></BLOCKQUOTE></BODY></HTML>