<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.6000.16587" name=GENERATOR></HEAD>
<BODY>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>I am trying
to bring up our first "real" lustre cluster and running into an unexpected
problem.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Let me dump
out some background on the configuration first:</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>I am running
CentOS 5.0 x86_64bit with Lustre 1.6.4.2.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>I have 2
racks of machines located next to each other. Each rack has its own
router. </FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Each router
has its own subnet. </FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Each machine
has 2 NICs one that is connected into the local rack, and one connected into
adjacent rack.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Each machine
has local disks in it that serve as the OST storage.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Each machine
has a lustre client running on it with visibility into the lustre
cluster.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008></SPAN><SPAN class=161100918-07032008><FONT
face="Courier New" size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>We are not
using bonding, so each machine has visibility into each subnet via the cross
connecting. </FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>We have this
options lnet networks=tcp0(eth1,eth0) in the modprobe.conf
file.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>We are not
yet using a routing protocol yet. The default route out for each machine
it's local router connected to eth0.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>ASCII
Pictures:</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Machines in
Rack A are all wired like this:</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>RACK
A
RACK B </FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2>--------------------
----------------------</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>|Router
(4.23.37.1)| <SPAN
class=161100918-07032008><FONT face="Courier New" size=2>|Router A
(4.23.36.1)|</FONT></SPAN></FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2>-------------------- ---------------------</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2> |
eth0 (4.23.37.10) |</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2><SPAN
class=161100918-07032008><FONT face="Courier New"
size=2>----------</FONT></SPAN>
|</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008></SPAN><SPAN class=161100918-07032008><FONT
face="Courier New" size=2>| Machine |--eth1--(4.23.36.42)-|</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>----------
def gw (4.23.37.1)</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>
</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>Machines in
Rack B are all wired like this:</FONT></SPAN></DIV></FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>RACK
A
RACK B </FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2>--------------------
----------------------</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>|Router
(4.23.37.1)| <SPAN
class=161100918-07032008><FONT face="Courier New" size=2>|Router A
(4.23.36.1)|</FONT></SPAN></FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2>-------------------- ---------------------</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2> |
| eth0 (4.23.36.10)</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2><SPAN
class=161100918-07032008><FONT face="Courier New"
size=2> |
----------</FONT></SPAN> </FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2> --eth1--(4.23.37.42)--|
Machine |</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008></SPAN><SPAN class=161100918-07032008><FONT
face="Courier New"
size=2> def gw
(4.23.36.1) ----------</FONT></SPAN></DIV></FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face=Arial size=2><FONT
face="Courier New"></FONT></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face=Arial size=2><FONT
face="Courier New">The primary MGS is at the top of one of the racks
(RACK A for this email.)</FONT></FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>The machines
are all up and running with lustre on them, they can ping/ssh between each
other.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>No for the
problem, All of the clients in rack A, the rack with the MGS can see all
of the OSTs being served from all of the OSSs in both racks.</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>The clients
in rack B can only see the MGS and the OSSes in rack A, none in their own rack,
rack B. They all show timeouts like this:</FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT size=2><FONT
face="Courier New">Mar 7 18:43:31 dintnyc1303 kernel: LustreError:
2914:0:(events.c:55:request_out_callback()) Skipped 275 previous similar
messages<BR>Mar 7 18:45:41 dintnyc1303 kernel: LustreError:
2914:0:(client.c:975:ptlrpc_expire_one_request()) @@@ timeout (sent at
1204915536, 5s ago) </FONT><A href="mailto:req@ffff81020327f800"><FONT
face="Courier New">req@ffff81020327f800</FONT></A><FONT face="Courier New">
x123942/t0 o8->content-OST001c_UUID@4.23.36.41@tcp:6 lens 240/272 ref 1 fl
Rpc:/0/0 rc 0/-22<BR>Mar 7 18:45:41 dintnyc1303 kernel: LustreError:
2914:0:(client.c:975:ptlrpc_expire_one_request()) Skipped 403 previous similar
messages<BR>Mar 7 18:53:31 dintnyc1303 kernel: LustreError:
2914:0:(events.c:55:request_out_callback()) @@@ type 4, status -5
</FONT><A href="mailto:req@ffff810203276000"><FONT
face="Courier New">req@ffff810203276000</FONT></A><FONT face="Courier New">
x124871/t0 o8->content-OST002d_UUID@4.23.36.46@tcp:6 lens 240/272 ref 2 fl
Rpc:/0/0 rc 0/-22<BR>Mar 7 18:53:31 dintnyc1303 kernel: LustreError:
2914:0:(events.c:55:request_out_callback()) Skipped 278 previous similar
messages<BR>Mar 7 18:56:01 dintnyc1303 kernel: LustreError:
2914:0:(client.c:975:ptlrpc_expire_one_request()) @@@ network error (sent at
1204916161, 0s ago) </FONT><A href="mailto:req@ffff8101f84dae00"><FONT
face="Courier New">req@ffff8101f84dae00</FONT></A><FONT face="Courier New">
x125140/t0 o8->content-OST001a_UUID@4.23.36.40@tcp:6 lens 240/272 ref 1 fl
Rpc:/0/0 rc 0/-22<BR>Mar 7 18:56:01 dintnyc1303 kernel: LustreError:
2914:0:(client.c:975:ptlrpc_expire_one_request()) Skipped 445 previous similar
messages</FONT></FONT></SPAN></DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New"
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=161100918-07032008><FONT face="Courier New" size=2>I am a bit
stuck.</FONT></SPAN></DIV></BODY></HTML>