<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">(Re-sending my response to the list)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:black">Yes, I believe that there are cases when problems on a remote node can be interpreted as local failures.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">"nathan.dauchy@noaa.gov" <nathan.dauchy@noaa.gov><br>
<b>Date: </b>Sunday, March 8, 2020 at 3:56 AM<br>
<b>To: </b>Chris Horn <hornc@cray.com>, "lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Cc: </b>"nathan.dauchy@noaa.gov" <nathan.dauchy@noaa.gov><br>
<b>Subject: </b>Re: [lustre-discuss] lnet_peer_ni_add_to_recoveryq<br>
<b>Resent-From: </b><hornc@cray.com><br>
<b>Resent-Date: </b>Sunday, March 8, 2020 at 4:56 AM<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">Chris, all,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We are also seeing similar messages primarily on our servers, but from lnet_handle_local_failure() instead. I don't find any issues with the local o2ib interfere, yet, but there _may_ be a correlation with a client hang. Could this also
be caused on a server by remote network problems or a client dropping out, in spite of the "local" name?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Nathan<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Mar 6, 2020 1:10 PM, Chris Horn <hornc@cray.com> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p>> lneterror: 10164:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) <br>
> lpni <address> added to recovery queue. Health = 900 <br>
<br>
The message means that the health value of a remote peer interface has been decremented, and as a result, the interface has been put into recovery mode. This mechanism is part of the LNet health feature.
<br>
<br>
Health values are decremented when a PUT or GET fails. Usually there are other messages in the log that can tell you more about the specific failure. Depending on your network type you should probably see messages from socklnd or o2iblnd. Network congestion
could certainly lead to message timeouts, which would in turn result in interfaces being placed into recovery mode.
<br>
<br>
Chris Horn <br>
<br>
On 3/6/20, 8:59 AM, "lustre-discuss on behalf of Michael Di Domenico" <lustre-discuss-bounces@lists.lustre.org on behalf of mdidomenico4@gmail.com> wrote:
<br>
<br>
along the aforementioned error i also see these at the same time <br>
<br>
lustreerror: 9675:0:(obd_config.c:1428:class_modify_config()) <br>
<...>-clilov-<...>; failed to send uevent qos_threshold_rr=100 <br>
<br>
On Fri, Mar 6, 2020 at 9:39 AM Michael Di Domenico <br>
<mdidomenico4@gmail.com> wrote: <br>
> <br>
> On Fri, Mar 6, 2020 at 9:36 AM Degremont, Aurelien <degremoa@amazon.com> wrote:
<br>
> > <br>
> > Did you see any actual error on your system? <br>
> > <br>
> > Because there is a patch that is just decreasing the verbosity level of such messages, which looks like could be ignored.
<br>
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.whamcloud.com_browse_LU-2D13071&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc&s=jp8DpDcylEQYlbd9-s3efysfDy2KdLvBrptsplqR1ks&e=
<br>
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__review.whamcloud.com_-23_c_37718_&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc&s=8EUQ5wHRCuFFbd4PKxQCnTB_L9IgffvkzFw4_v6MEHg&e=
<br>
> <br>
> thanks. it's not entirely clear just yet. i'm trying to track down a <br>
> "slow jobs" issue. i see these messages everywhere, so it might be a <br>
> non issue or a sign of something more pressing. <br>
_______________________________________________ <br>
lustre-discuss mailing list <br>
lustre-discuss@lists.lustre.org <br>
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=ByOR33WN61jv0rEVZTtNhUgN313iSqbgrdfakY-TAjc&s=d36yZXUxMDJOjluQt2LUPivEkfLhScuCLIQT6Fl-Qhs&e=
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________ <br>
lustre-discuss mailing list <br>
lustre-discuss@lists.lustre.org <br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwQGaQ&c=C5b8zRQO1miGmBeVZ2LFWg&r=hIaFpo9yRyCwkkAs6y0c7W-QqT7uZMMSOkAIByhcA-I&m=xjhlFKAxRoTIY1jLm_ZOO79SIHjnFFvd-sHl1eMEQQM&s=Wvg4NbAeA1O-DrqWqy5rrQ4OrwfrO7V220OCeVGeWdg&e=">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
<o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>