<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Apple Color Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:black">FYI, multi-rail in 2.12 will round robin traffic between both @tcp and @o2ib networks (assuming peers are reachable on both). If @o2ib flakes out then traffic should shift entirely to @tcp, but there isn’t a way
to specify that traffic go to @tcp only when there’s a problem with @o2ib. You need the user defined selection policy feature for that, and that feature is not slated to arrive until after 2.14 (afaik).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Chris Horn<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of "Spitz, Cory James" <cory.spitz@hpe.com><br>
<b>Date: </b>Thursday, February 11, 2021 at 3:17 PM<br>
<b>To: </b>"nathan.crawford@uci.edu" <nathan.crawford@uci.edu>, Lustre User Discussion Mailing List <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] LNET IB intermittent connection<br>
<b>Resent-From: </b><hornc@cray.com><br>
<b>Resent-Date: </b>Thursday, February 11, 2021 at 3:17 PM<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">Hi, Nate.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">You asked, “can LNET be easily configured to go over the @tcp connection when the @o2ib flakes out?”<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Yes, you can use LNet Multi-Rail for it and that _<i>is</i>_ covered in the “fine manual”, chapter 16
<span style="font-family:"Apple Color Emoji"">☺</span><o:p></o:p></p>
<p class="MsoNormal"><a href="https://doc.lustre.org/lustre_manual.xhtml#lnetmr">https://doc.lustre.org/lustre_manual.xhtml#lnetmr</a><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">-Cory<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 2/10/21, 4:54 PM, "lustre-discuss" <lustre-discuss-bounces@lists.lustre.org> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi All,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> I've recently been having a bunch of LNET over Infiniband connection-lost/-restored errors and am trying to find the cause and/or tune the system to better cope. There is a lot of stuff on the wiki (
<a href="https://wiki.lustre.org/Lustre_Resiliency:_Understanding_Lustre_Message_Loss_and_Tuning_for_Resiliency">https://wiki.lustre.org/Lustre_Resiliency:_Understanding_Lustre_Message_Loss_and_Tuning_for_Resiliency</a>), but that's from 2016, and I don't know what parts are superseded. I'm currently running Lustre 2.12.5 on CentOS 7.8, with a mix of Q-Logic/Intel QDR
and Mellanox EDR HCAs and switches (using CentOS in-box RDMA/opensm).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> Is there a better place to look (e.g. the fine manual, section X) for guidance? I've done a few searches on the Jira, but the most similar errors should have already been fixed in earlier releases.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> Assuming that there is actually some impending hardware issue, can LNET be easily configured to go over the @tcp connection when the @o2ib flakes out?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Nate<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<pre style="margin-left:.5in">Dr. Nathan Crawford <a href="mailto:nathan.crawford@uci.edu" target="_blank">nathan.crawford@uci.edu</a><o:p></o:p></pre>
<pre style="margin-left:.5in">Director of Scientific Computing<o:p></o:p></pre>
<pre style="margin-left:.5in">School of Physical Sciences<o:p></o:p></pre>
<pre style="margin-left:.5in">164 Rowland Hall Office: 2101 Natural Sciences II<o:p></o:p></pre>
<pre style="margin-left:.5in">University of California, Irvine Phone: 949-824-4508<o:p></o:p></pre>
<pre style="margin-left:.5in">Irvine, CA 92697-2025, USA<o:p></o:p></pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>