<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Kevin,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Just wild-guessing here. Have you tried playing with the live_router_check_interval, dead_router_check_interval and router_ping_timeout LNet parameters?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">HTH,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Alejandro<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> lustre-discuss [mailto:lustre-discuss-bounces@lists.lustre.org]
<b>On Behalf Of </b>Kevin M. Hildebrand<br>
<b>Sent:</b> Monday, October 30, 2017 1:47 PM<br>
<b>To:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> [lustre-discuss] Lustre routing help needed<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hello, I'm trying to set up some new Lustre routers between a set of Infiniband connected Lustre servers and a few hosts connected to an external 100G Ethernet network. The problem I'm having is that the routers work just fine for a minute
or two, and then shortly thereafter they're marked as 'down' and all traffic stops. If I unload/reload the lustre modules on the router, it'll work again for a short time and then stop again. The router shows errors like:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">[236528.801275] LNetError: 54389:0:(lib-move.c:2120:lnet_parse_get())
<a href="mailto:10.10.104.2@tcp2">10.10.104.2@tcp2</a>: Unable to send REPLY for GET from
<a href="mailto:12345-10.10.104.201@tcp2">12345-10.10.104.201@tcp2</a>: -113<o:p></o:p></p>
</div>
<p class="MsoNormal">My Lustre router has a Mellanox ConnectX-3 interface connecting to the Lustre servers, and a Mellanox ConnectX-5
<o:p></o:p></p>
<div>
<p class="MsoNormal">100G <o:p></o:p></p>
</div>
<p class="MsoNormal">interface connecting to a 100G switch to which my test client is connected.<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal">On the Infiniband side, I've got <o:p></o:p></p>
<div>
<p class="MsoNormal">lnet<o:p></o:p></p>
</div>
<p class="MsoNormal"> configured as o2ib1<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p></o:p></p>
</div>
<p class="MsoNormal">, and on the Ethernet side, as tcp2.<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Clients and servers are all running Lustre 2.8. The Lustre router at the moment is running Lustre 2.10.1, because of software dependencies to support the 100G card.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I've verified that I have stable network connectivity on both the IB and Ethernet sides.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">At the moment, I have very simple lnet configurations, using the built in defaults. lnet.conf on the server:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">options lnet ip2nets="o2ib1(ib0) 192.168.[64-95].*; tcp1 10.103.[128-159].*" routes="tcp0 192.168.64.[78-79]@o2ib1; tcp2 192.168.64.[78-79]@o2ib1"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On the lustre router:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">options lnet networks="o2ib1(ib0),tcp2(p1p1.104)" "forwarding=enabled"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">And on the client:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">options lnet networks="tcp2(p4p1.104)" routes="o2ib1 10.10.104.[2-3]@tcp2"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">All of the hosts (client, server, router) have the following in ko2iblnd.conf:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
alias ko2iblnd-opa ko2iblnd<br>
options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4<br>
<br>
install ko2iblnd /usr/sbin/ko2iblnd-probe<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Does anyone see anything I've missed, or have any thoughts on where I should look next?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Kevin<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Kevin Hildebrand<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">University of Maryland, College Park<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>