<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Hi Sid,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">if you are using a CentOS 7.9 kernel newer than 3.10.0-1160.6.1.el7.x86_64 then check out
<b>LU-14341</b> as these kernel versions cause a timer related regression:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><a href="https://jira.whamcloud.com/browse/LU-14341">https://jira.whamcloud.com/browse/LU-14341</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">We learnt this the hard way during the last couple of days and downgraded to kernel-3.10.0-1160.2.1.el7.x86_64 (which is the officially supported kernel version of lustre 2.12.6). We
use ZFS. YMMV.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">-- <br>
</span><b><span lang="EN-US" style="font-size:9.0pt;font-family:"Verdana",sans-serif;color:#1F497D">Karsten Weiss<br>
<br>
</span></b><span lang="EN-US" style="font-size:8.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> lustre-discuss <lustre-discuss-bounces@lists.lustre.org>
<b>On Behalf Of </b>Sid Young via lustre-discuss<br>
<b>Sent:</b> Tuesday, March 2, 2021 02:37<br>
<b>To:</b> lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> [lustre-discuss] OSS node crash/high CPU latency when deleting 100's of emty test files<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p style="text-align:justify;background:#7F0000"><strong><span lang="EN-US" style="font-family:"Calibri",sans-serif;color:white">Caution!</span></strong><span lang="EN-US" style="color:white"> External email. Do not open attachments or click links, unless this
email comes from a known sender and you know the content is safe.<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">G'Day all,<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I've been doing some file create/delete testing on our new Lustre storage which results in the OSS nodes crashing and rebooting due to high latency issues.<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I can reproduce it by running "dd" commands on the /lustre file system in a for loop and then do a rm -f testfile-*.text at the end.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">This results in console errors on our DL385 OSS nodes (running Centos 7.9) which basically show a stack of:<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> mlx5_core and bnxt_en error messages.... mlx5 being the Mellanox Driver for the 100G ConnectX5 cards followed by a stack of: <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">"NMI watchdog: BUG: soft lockup - CPU#"N stuck for XXs " <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">where the CPU number is around 4 different ones and XX is typical 20-24seconds...then the boxes reboot!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Before I log a support ticket to HPe, I'm going to try and disable the 100G cards and see if its repeatable via the 10G interfaces on the motherboards, but before I do that, does anyone use the mellanox ConnectX5 cards
on their Lustre Storage nodes and ethernet only and if so, which driver are you using and on which OS...<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal">Thanks in advance!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Sid Young<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>