<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Apple Color Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas",serif;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Right, the round-robin-only approach may be a deal breaker. You might be able to come up with a “poor-man’s” solution, though I don’t think there is an obvious path. You might be able to force the health detection as you suggest or manually
bring up the 2<sup>nd</sup> interface when you know the first has failed. Chris or others might have a more specific approach or advice. While 2.14.0. is coming out “soon” (2.14.0-RC2 is tagged), I don’t think that UDSP made it. <a href="https://jira.whamcloud.com/browse/LU-9121">https://jira.whamcloud.com/browse/LU-9121</a>
isn’t listed in the <a href="https://wiki.lustre.org/Release_2.14.0">https://wiki.lustre.org/Release_2.14.0</a> project page.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Cory<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 2/11/21, 5:56 PM, "Nathan Crawford" <nrcrawfo@uci.edu> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi Chris and Cory,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> I remember looking at configuring multi-rail when 2.12 came out for this very reason, but stopped when it looked like round-robin only. Is there a way to trick the LNet Health system into seeing one interface
as "sick but not dead"?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> Also, when is 2.14 coming out :)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> For what it's worth, the client errors I'm trying to diagnose (only one client has them) are similar to:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">[Thu Feb 11 15:51:24 2021] LustreError: 11-0: DFS-L-OST0003-osc-ffff9cd07c339000: operation ost_set_info to node 10.201.32.48@o2ib1 failed: rc = -107<br>
[Thu Feb 11 15:51:24 2021] Lustre: DFS-L-OST0003-osc-ffff9cd07c339000: Connection to DFS-L-OST0003 (at 10.201.32.48@o2ib1) was lost; in progress operations using this service will wait for recovery to complete<br>
[Thu Feb 11 15:51:24 2021] LustreError: 167-0: DFS-L-OST0003-osc-ffff9cd07c339000: This client was evicted by DFS-L-OST0003; in progress operations using this service will fail.<br>
[Thu Feb 11 15:51:24 2021] Lustre: DFS-L-OST0003-osc-ffff9cd07c339000: Connection restored to 10.201.32.48@o2ib1 (at 10.201.32.48@o2ib1)</span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Nate<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">On Thu, Feb 11, 2021 at 1:25 PM Horn, Chris <<a href="mailto:chris.horn@hpe.com">chris.horn@hpe.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
FYI, multi-rail in 2.12 will round robin traffic between both @tcp and @o2ib networks. If @o2ib flakes out then traffic should shift entirely to @tcp, but there isn’t a way to specify that traffic go to @tcp only when there’s a problem with @o2ib. You need
the user defined selection policy feature for that, and that feature is not slated to arrive until after 2.14 (afaik).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
Chris Horn<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of "Spitz,
Cory James" <<a href="mailto:cory.spitz@hpe.com" target="_blank">cory.spitz@hpe.com</a>><br>
<b>Date: </b>Thursday, February 11, 2021 at 3:17 PM<br>
<b>To: </b>"<a href="mailto:nathan.crawford@uci.edu" target="_blank">nathan.crawford@uci.edu</a>" <<a href="mailto:nathan.crawford@uci.edu" target="_blank">nathan.crawford@uci.edu</a>>, Lustre User Discussion Mailing List <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
<b>Subject: </b>Re: [lustre-discuss] LNET IB intermittent connection<br>
<b>Resent-From: </b><<a href="mailto:hornc@cray.com" target="_blank">hornc@cray.com</a>><br>
<b>Resent-Date: </b>Thursday, February 11, 2021 at 3:17 PM</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
Hi, Nate.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
You asked, “can LNET be easily configured to go over the @tcp connection when the @o2ib flakes out?”<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
Yes, you can use LNet Multi-Rail for it and that _<i>is</i>_ covered in the “fine manual”, chapter 16
<span style="font-family:"Apple Color Emoji"">☺</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<a href="https://doc.lustre.org/lustre_manual.xhtml#lnetmr" target="_blank">https://doc.lustre.org/lustre_manual.xhtml#lnetmr</a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
-Cory<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
On 2/10/21, 4:54 PM, "lustre-discuss" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
Hi All,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
I've recently been having a bunch of LNET over Infiniband connection-lost/-restored errors and am trying to find the cause and/or tune the system to better cope. There is a lot of stuff on the wiki (
<a href="https://wiki.lustre.org/Lustre_Resiliency:_Understanding_Lustre_Message_Loss_and_Tuning_for_Resiliency" target="_blank">https://wiki.lustre.org/Lustre_Resiliency:_Understanding_Lustre_Message_Loss_and_Tuning_for_Resiliency</a>), but that's from 2016, and I don't know what parts are superseded. I'm currently running Lustre 2.12.5 on CentOS 7.8, with a mix of Q-Logic/Intel QDR
and Mellanox EDR HCAs and switches (using CentOS in-box RDMA/opensm).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
Is there a better place to look (e.g. the fine manual, section X) for guidance? I've done a few searches on the Jira, but the most similar errors should have already been fixed in earlier releases.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
Assuming that there is actually some impending hardware issue, can LNET be easily configured to go over the @tcp connection when the @o2ib flakes out?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
Nate<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:1.0in">
-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<pre style="margin-left:1.0in">Dr. Nathan Crawford <a href="mailto:nathan.crawford@uci.edu" target="_blank">nathan.crawford@uci.edu</a><o:p></o:p></pre>
<pre style="margin-left:1.0in">Director of Scientific Computing<o:p></o:p></pre>
<pre style="margin-left:1.0in">School of Physical Sciences<o:p></o:p></pre>
<pre style="margin-left:1.0in">164 Rowland Hall Office: 2101 Natural Sciences II<o:p></o:p></pre>
<pre style="margin-left:1.0in">University of California, Irvine Phone: 949-824-4508<o:p></o:p></pre>
<pre style="margin-left:1.0in">Irvine, CA 92697-2025, USA<o:p></o:p></pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:.5in"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<pre style="margin-left:.5in">Dr. Nathan Crawford <a href="mailto:nathan.crawford@uci.edu" target="_blank">nathan.crawford@uci.edu</a><o:p></o:p></pre>
<pre style="margin-left:.5in">Director of Scientific Computing<o:p></o:p></pre>
<pre style="margin-left:.5in">School of Physical Sciences<o:p></o:p></pre>
<pre style="margin-left:.5in">164 Rowland Hall Office: 2101 Natural Sciences II<o:p></o:p></pre>
<pre style="margin-left:.5in">University of California, Irvine Phone: 949-824-4508<o:p></o:p></pre>
<pre style="margin-left:.5in">Irvine, CA 92697-2025, USA<o:p></o:p></pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>