<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">You need LNet routes configured on all nodes. It should look something like this:<br>
<br>
# pdsh -w n0[0-3] 'lctl list_nids; lctl show_route' | dshbak -c<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">----------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">server<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">----------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="mailto:172.18.2.5@o2ib">172.18.2.5@o2ib</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net o2ib2 hops 2 gw
<a href="mailto:172.18.2.6@o2ib">172.18.2.6@o2ib</a> up pri 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">----------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">router1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">----------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="mailto:172.18.2.6@o2ib">172.18.2.6@o2ib</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="mailto:172.18.2.2@tcp">172.18.2.2@tcp</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net o2ib2 hops 1 gw
<a href="mailto:172.18.2.3@tcp">172.18.2.3@tcp</a> up pri 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">----------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">router2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">----------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="mailto:172.18.2.7@o2ib2">172.18.2.7@o2ib2</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="mailto:172.18.2.3@tcp">172.18.2.3@tcp</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net o2ib hops 1 gw
<a href="mailto:172.18.2.2@tcp">172.18.2.2@tcp</a> up pri 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">----------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">client<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">----------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="mailto:172.18.2.8@o2ib2">172.18.2.8@o2ib2</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">net o2ib hops 2 gw
<a href="mailto:172.18.2.7@o2ib2">172.18.2.7@o2ib2</a> up pri 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Chris Horn<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of John White via lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Date: </b>Wednesday, March 5, 2025 at 1:17</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">PM<br>
<b>To: </b>lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[lustre-discuss] multi-hop routing<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hello folks. I have a rare situation that I’m told some centers are successfully pulling off and am looking for guidance - multi-hop lnet routing.<br>
In short, I have 2 distinct o2ib fabrics at disparate geo sites joined by a routed ethernet fabric. I’m looking to use a 2-lnet-router chain to plumb the two o2ib fabrics together.<br>
<br>
servers on the left, clients on the right<br>
o2ib0(10.5.0.0/16) <-> router(o2ib0,tcp0) <-> routed eth (10.37.0.0/16, 10.38.0.0/16) <-> router(tcp0,o2ib2) <-> o2ib2(10.6.0.0/16)<br>
<br>
I have both sets of routers up but traffic absolutely fails the 2nd hop in either direction (I can `lctl ping` tcp0 from o2ib2 and o2ib0 but no further).<br>
<br>
I’ve tried adding a route ON the routers, that didn’t help. <br>
<br>
I’ve tried defining the 2nd hop on the client:<br>
options lnet routes="tcp0 10.6.0.[250-251]@o2ib2;\<br>
o2ib0 10.37.250.[162-163]@tcp0”<br>
<br>
but that failed with the following kern message on lnet load:<br>
74067:0:(router.c:644:lnet_add_route()) Cannot add route with gateway 10.37.250.162@tcp. There is no local interface configured on LNet tcp<br>
<br>
Does anyone have any hints here? It feels like I’m a syntax change or a routing hint away from getting this working.<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
<a href="https://urldefense.com/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!NpxR!keuGPb7MHd7CQc6Zi_uwIvFahK68FJfbq9MNIXgHpd0W8bi5vOYFHf-IixYY5DiOnJKx0z9-Ht8VqH1ew82XWtaTRaoq$">https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!NpxR!keuGPb7MHd7CQc6Zi_uwIvFahK68FJfbq9MNIXgHpd0W8bi5vOYFHf-IixYY5DiOnJKx0z9-Ht8VqH1ew82XWtaTRaoq$</a>
<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>