<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:87510341;
mso-list-template-ids:1027003422;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">John,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Sounds like a complex network, thus simplifying the problem might help. One way to simplify would be to setup the client LNet config exactly as you think it
should be, then try to “lctl ping” the MGS on each file system from *<b>that</b>* client. If each works, you’re close – if not, sniff the network to see if the client pings make it to the MGS’s, and if they do, then check the route(s) back as well.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Occasionally I’ve found that the source knows how to route to the destination, but the destination has no route *<b>back</b>*. Hence, lctl ping should allow
you to test this out – at least for the MGS’s. After that come the other servers…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<table class="MsoTableGrid" border="0" cellspacing="0" cellpadding="0" align="left" width="361" style="width:271.05pt;border-collapse:collapse;border:none">
<tbody>
<tr style="height:49.15pt">
<td width="78" valign="top" style="width:58.75pt;padding:0in 5.4pt 0in 5.4pt;height:49.15pt">
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:page;mso-element-left:95.45pt;mso-element-top:1.0pt;mso-height-rule:exactly">
<span style="font-size:9.0pt;font-family:"Tahoma","sans-serif";color:#1F497D"><img width="61" height="41" id="Picture_x0020_1" src="cid:image001.png@01CF484A.FD24F070" alt="Description: cid:image001.gif@01C9FE3D.1D8A68C0"></span><span style="font-family:"Tahoma","sans-serif";color:#1F497D"><o:p></o:p></span></p>
</td>
<td width="283" valign="top" style="width:212.3pt;padding:0in 5.4pt 0in 5.4pt;height:49.15pt">
<p class="MsoNormal" style="page-break-before:always;page-break-after:avoid;mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:page;mso-element-left:95.45pt;mso-element-top:1.0pt;mso-height-rule:exactly">
<span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#3C69B0">Dr. Brett Lee, Solutions Architect</span><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal" style="page-break-before:always;page-break-after:avoid;mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:page;mso-element-left:95.45pt;mso-element-top:1.0pt;mso-height-rule:exactly">
<span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#3C69B0">High Performance Data Division, Intel </span><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal" style="page-break-before:always;page-break-after:avoid;mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:page;mso-element-left:95.45pt;mso-element-top:1.0pt;mso-height-rule:exactly">
<span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:#3C69B0">+1.303.625.3595</span><span style="font-size:10.5pt;font-family:"Tahoma","sans-serif";color:#3C69B0"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> lustre-discuss-bounces@lists.lustre.org [mailto:lustre-discuss-bounces@lists.lustre.org]
<b>On Behalf Of </b>John Lalande<br>
<b>Sent:</b> Tuesday, March 25, 2014 2:17 PM<br>
<b>To:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [Lustre-discuss] multi-homed lustre with both IB and TCP<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi, Ron-<br>
<br>
Thanks for sharing your config with me. I tried tweaking ours, and it's still a no go. I think the main difference here is that it's our client (not the servers) that is multi-homed.<br>
<br>
The client needs to access:<o:p></o:p></p>
<ol start="1" type="1">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
one (eventually more) Lustre filesystem(s) via direct attached InfiniBand.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
one Lustre file system via TCP (no TCP->IB routing)<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
several Lustre file systems via routed InfiniBand (TCP->IB)<o:p></o:p></li></ol>
<p class="MsoNormal">I can't get #1 and #3 working together ... can get one or the other working depending on how I've configured the lnet networks in modprobe.d/lustre.conf, but not both. (#2 works either way)<br>
<br>
Does anyone else have ideas on this?<br>
<br>
Thanks!<br>
<br>
John<br>
<br>
<br>
On 3/24/14, 4:13 PM, Jerome, Ron wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>Hi John,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Don't know if you got this working, but I can tell you that I have more or less the same setup working. Basically I have a client on a public TCP network connecting to an LNET router (via TCP) which then forwards via IB to the Luster cluster. (all the lustre servers are multi-homed and have a tcp0 network internally, thus the "tcp1" for the external TCP network)<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>The external client config is... (where 132.246.x.x is the TCP address of the router)<o:p></o:p></pre>
<pre>---------------------------------<o:p></o:p></pre>
<pre>options lnet networks=tcp1(eth0) routes="o2ib0 <a href="mailto:132.246.x.x@tcp1">132.246.x.x@tcp1</a>"<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>...<o:p></o:p></pre>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre><o:p> </o:p></pre>
<pre>Ron. <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>-----Original Message-----<o:p></o:p></pre>
<pre>From: <a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a> [<a href="mailto:lustre-discuss-bounces@lists.lustre.org">mailto:lustre-discuss-bounces@lists.lustre.org</a>] On Behalf Of John Lalande<o:p></o:p></pre>
<pre>Sent: March 21, 2014 3:56 PM<o:p></o:p></pre>
<pre>To: <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><o:p></o:p></pre>
<pre>Subject: [Lustre-discuss] multi-homed lustre with both IB and TCP<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Hi-<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>I am trying to set up a robinhood policy engine server that will watch <o:p></o:p></pre>
<pre>several different Lustre file systems -- one of which will have a direct <o:p></o:p></pre>
<pre>Infiniband connection, one via TCP without an intermediate Lustre router <o:p></o:p></pre>
<pre>and several other Lustre file systems via TCP through Lustre routers.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>I can mount filesystems via IB and direct TCP, but not the routed ones. <o:p></o:p></pre>
<pre>(I am able to mount the routed ones if I take out the config for o2ib0@ib0).<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>My modprobe.conf looks like this:<o:p></o:p></pre>
<pre>options lnet networks="o2ib0(ib0),tcp0(em1.497)" routes="o2ib1 <o:p></o:p></pre>
<pre>ROUTER1_IP@tcp0; o2ib1 ROUTER2_IP@tcp0; o2ib1 ROUTER3_IP@tcp0"<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>where router1_IP, router2_IP, etc. are actual IP addresses on our <o:p></o:p></pre>
<pre>University's subnet that I don't want to publish here.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>/etc/fstab looks like this:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><a href="mailto:172.17.1.5@o2ib0:/ib_filesystem">172.17.1.5@o2ib0:/ib_filesystem</a> /ib_filesystem lustre <o:p></o:p></pre>
<pre>defaults,_netdev,user_xattr 0 0<o:p></o:p></pre>
<pre><a href="mailto:172.16.24.5@o2ib1:/routedfs1">172.16.24.5@o2ib1:/routedfs1</a> /fs1 lustre <o:p></o:p></pre>
<pre>defaults,_netdev,user_xattr 0 0<o:p></o:p></pre>
<pre><a href="mailto:172.16.23.14@o2ib1:/routedfs2">172.16.23.14@o2ib1:/routedfs2</a> /fs2 lustre <o:p></o:p></pre>
<pre>defaults,_netdev,user_xattr 0 0<o:p></o:p></pre>
<pre><a href="mailto:172.16.25.189@o2ib1:/routedfs3">172.16.25.189@o2ib1:/routedfs3</a> /fs3 lustre <o:p></o:p></pre>
<pre>defaults,_netdev,user_xattr 0 0<o:p></o:p></pre>
<pre><a href="mailto:172.16.25.241@o2ib1:/routedfs4">172.16.25.241@o2ib1:/routedfs4</a> /fs4 lustre <o:p></o:p></pre>
<pre>defaults,_netdev,user_xattr 0 0<o:p></o:p></pre>
<pre><a href="mailto:128.104.X.X@tcp:/tcpfs1">128.104.X.X@tcp:/tcpfs1</a> /tcpfs1 lustre <o:p></o:p></pre>
<pre>defaults,_netdev 0 0<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>In dmesg, I see:<o:p></o:p></pre>
<pre>Lustre: 6923:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request <o:p></o:p></pre>
<pre>sent has timed out for slow reply: [sent 1395431267/real 1395431267] <o:p></o:p></pre>
<pre>req@ffff880c2aa04800 x1463215106031860/t0(0) <o:p></o:p></pre>
<pre>o250-><a href="mailto:MGC172.16.24.5@o2ib1@172.16.24.5@o2ib1:26/25">MGC172.16.24.5@o2ib1@172.16.24.5@o2ib1:26/25</a> lens 400/544 e 0 to 1 <o:p></o:p></pre>
<pre>dl 1395431272 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<o:p></o:p></pre>
<pre>LustreError: 7239:0:(client.c:1052:ptlrpc_import_delay_req()) @@@ send <o:p></o:p></pre>
<pre>limit expired req@ffff880c2aa04000 x1463215106031864/t0(0) <o:p></o:p></pre>
<pre>o101-><a href="mailto:MGC172.16.24.5@o2ib1@172.16.24.5@o2ib1:26/25">MGC172.16.24.5@o2ib1@172.16.24.5@o2ib1:26/25</a> lens 328/344 e 0 to 0 <o:p></o:p></pre>
<pre>dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<o:p></o:p></pre>
<pre>LustreError: 7230:0:(client.c:1052:ptlrpc_import_delay_req()) @@@ send <o:p></o:p></pre>
<pre>limit expired req@ffff88182b1fac00 x1463215106031872/t0(0) <o:p></o:p></pre>
<pre>o101-><a href="mailto:MGC172.16.24.5@o2ib1@172.16.24.5@o2ib1:26/25">MGC172.16.24.5@o2ib1@172.16.24.5@o2ib1:26/25</a> lens 328/344 e 0 to 0 <o:p></o:p></pre>
<pre>dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<o:p></o:p></pre>
<pre>LustreError: 7230:0:(client.c:1052:ptlrpc_import_delay_req()) @@@ send <o:p></o:p></pre>
<pre>limit expired req@ffff88182a1ab000 x1463215106031876/t0(0) <o:p></o:p></pre>
<pre>o101-><a href="mailto:MGC172.16.24.5@o2ib1@172.16.24.5@o2ib1:26/25">MGC172.16.24.5@o2ib1@172.16.24.5@o2ib1:26/25</a> lens 328/344 e 0 to 0 <o:p></o:p></pre>
<pre>dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<o:p></o:p></pre>
<pre>Lustre: 6923:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request <o:p></o:p></pre>
<pre>sent has timed out for slow reply: [sent 1395431292/real 1395431292] <o:p></o:p></pre>
<pre>req@ffff88182a2a3400 x1463215106031976/t0(0) <o:p></o:p></pre>
<pre>o250-><a href="mailto:MGC172.16.24.5@o2ib1@172.16.24.5@o2ib1:26/25">MGC172.16.24.5@o2ib1@172.16.24.5@o2ib1:26/25</a> lens 400/544 e 0 to 1 <o:p></o:p></pre>
<pre>dl 1395431302 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<o:p></o:p></pre>
<pre>LustreError: 7239:0:(client.c:1052:ptlrpc_import_delay_req()) @@@ send <o:p></o:p></pre>
<pre>limit expired req@ffff880c2aa04000 x1463215106031868/t0(0) <o:p></o:p></pre>
<pre>o101-><a href="mailto:MGC172.16.24.5@o2ib1@172.16.24.5@o2ib1:26/25">MGC172.16.24.5@o2ib1@172.16.24.5@o2ib1:26/25</a> lens 328/344 e 0 to 0 <o:p></o:p></pre>
<pre>dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>So ... is what we're trying to do here possible, and I'm just mangling <o:p></o:p></pre>
<pre>the config, or is Lustre over IB + Lustre via IB router not possible?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Thanks for any help!<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>John<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
</blockquote>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>John Lalande<o:p></o:p></pre>
<pre>Space Science & Engineering Center<o:p></o:p></pre>
<pre>University of Wisconsin - Madison<o:p></o:p></pre>
<pre><a href="mailto:john.lalande@ssec.wisc.edu">john.lalande@ssec.wisc.edu</a> / 608-263-2268<o:p></o:p></pre>
</div>
</div>
</body>
</html>