<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">John<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">That is not a community version of Lustre – that is a DDN version. You should pursue this via support channels<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Peter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Calibri",sans-serif;color:black">From:
</span></b><span style="font-family:"Calibri",sans-serif;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of John Bauer via lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Reply-To: </b>John Bauer <bauerj@iodoctors.com><br>
<b>Date: </b>Wednesday, April 9, 2025 at 4:45 PM<br>
<b>To: </b>Andreas Dilger <adilger@ddn.com><br>
<b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] unkillable process using llapi_layout_file_open<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p>Andreas,<o:p></o:p></p>
<p>Thanks for the quick reply. The client version is 2.14.0_ddn173. The server version is also target_version: 2.14.0.173. This originally started as the result of user input error that requested an OST that does not exist. For my simple test case I request
an OST that does not exist, and probably never will exist. This issue is on plieades at NAS/NASA which doesn't change very much. I doubt that this related to an OST or MDT that may have been recently added.
<o:p></o:p></p>
<p>The admins are checking on LU-17334.<o:p></o:p></p>
<p>The admins also noticed thousands of error messages<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[root@r593i4n16 ~]# dmesg -T |grep LustreError</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:22 2025] LustreError: 11-0: nbp17-MDT0000-mdc-ffff963283f77000: operation ldlm_enqueue to node
<a href="mailto:10.151.27.142@o2ib">10.151.27.142@o2ib</a> failed: rc = -19</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:23 2025] LustreError: 11-0: nbp17-MDT0000-mdc-ffff963283f77000: operation ldlm_enqueue to node
<a href="mailto:10.151.27.142@o2ib">10.151.27.142@o2ib</a> failed: rc = -19</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:23 2025] LustreError: Skipped 1709 previous similar messages</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:24 2025] LustreError: 11-0: nbp17-MDT0000-mdc-ffff963283f77000: operation ldlm_enqueue to node
<a href="mailto:10.151.27.142@o2ib">10.151.27.142@o2ib</a> failed: rc = -19</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:24 2025] LustreError: Skipped 3491 previous similar messages</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:26 2025] LustreError: 11-0: nbp17-MDT0000-mdc-ffff963283f77000: operation ldlm_enqueue to node
<a href="mailto:10.151.27.142@o2ib">10.151.27.142@o2ib</a> failed: rc = -19</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:26 2025] LustreError: Skipped 7803 previous similar messages</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:30 2025] LustreError: 11-0: nbp17-MDT0000-mdc-ffff963283f77000: operation ldlm_enqueue to node
<a href="mailto:10.151.27.142@o2ib">10.151.27.142@o2ib</a> failed: rc = -19</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:30 2025] LustreError: Skipped 14891 previous similar messages</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:38 2025] LustreError: 11-0: nbp17-MDT0000-mdc-ffff963283f77000: operation ldlm_enqueue to node
<a href="mailto:10.151.27.142@o2ib">10.151.27.142@o2ib</a> failed: rc = -19</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:38 2025] LustreError: Skipped 29887 previous similar messages</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:54 2025] LustreError: 11-0: nbp17-MDT0000-mdc-ffff963283f77000: operation ldlm_enqueue to node
<a href="mailto:10.151.27.142@o2ib">10.151.27.142@o2ib</a> failed: rc = -19</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:36:54 2025] LustreError: Skipped 63032 previous similar messages</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:37:26 2025] LustreError: 11-0: nbp17-MDT0000-mdc-ffff963283f77000: operation ldlm_enqueue to node
<a href="mailto:10.151.27.142@o2ib">10.151.27.142@o2ib</a> failed: rc = -19</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:37:26 2025] LustreError: Skipped 120772 previous similar messages</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:38:30 2025] LustreError: 11-0: nbp17-MDT0000-mdc-ffff963283f77000: operation ldlm_enqueue to node
<a href="mailto:10.151.27.142@o2ib">10.151.27.142@o2ib</a> failed: rc = -19</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:38:30 2025] LustreError: Skipped 238498 previous similar messages</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:40:38 2025] LustreError: 11-0: nbp17-MDT0000-mdc-ffff963283f77000: operation ldlm_enqueue to node
<a href="mailto:10.151.27.142@o2ib">10.151.27.142@o2ib</a> failed: rc = -19</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:40:38 2025] LustreError: Skipped 515538 previous similar messages</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:44:54 2025] LustreError: 11-0: nbp17-MDT0000-mdc-ffff963283f77000: operation ldlm_enqueue to node
<a href="mailto:10.151.27.142@o2ib">10.151.27.142@o2ib</a> failed: rc = -19</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-autospace:none">
<span style="font-size:9.0pt;font-family:"Lucida Console",serif">[Wed Apr 9 15:44:54 2025] LustreError: Skipped 1040417 previous similar messages</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Lucida Console",serif">[root@r593i4n16 ~]#</span>
<o:p></o:p></p>
<p>John<o:p></o:p></p>
<div>
<p class="MsoNormal">On 4/9/2025 4:58 PM, Andreas Dilger wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">On Apr 9, 2025, at 14:28, John Bauer via lustre-discuss <a href="mailto:lustre-discuss@lists.lustre.org">
<lustre-discuss@lists.lustre.org></a> wrote: <o:p></o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p>I have created a small reproducer program (81 lines of code) that results in a process that appears to hang in the kernel, accumulating cpu time. The process is unresponsive to kill commands. From gdb backtrace, it appears the call is stuck somewhere in
fsetxattr() which is called by llapi_layout_file_open(). The problem happens only when a non-existent ost is added to the layout with a call to llapi_layout_ost_index_set(). The call to llapi_layout_sanity(), just before calling llapi_layout_file_open(),
returns 0. Is this a known issue? <o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal">Hard to say for sure.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I suspect this is related to LU-17334, which relates to newly-added MDTs and OSTs in the filesystem. There were a few patches which recently landed in 2.16.0 (and backported) that will sleep and retry for a short time to handle the case
where a client accesses a file or directory layout that references an OST or MDT that it doesn't know about. The assumption is that the OST/MDT is newly added and the configuration update hasn't quite made it to the client yet. The client should retry to
contact the new server for some time before giving up and returning an error (in case the layout is actually bad).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Whether this is fixed in your version depends on what the version is (not mentioned in your email). It may also be important what the server version is, which can be seen from "lctl get_param mdc.*.import | grep target_version", if you
can access this parameter. if your client & server versions have the LU-17734 fixes, then this would be unexpected, and if older versions then I'd say it is something I'd rather not revisit until the known fixes are in place.<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Cheers, Andreas<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">—<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Andreas Dilger<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Lustre Principal Architect<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Whamcloud/DDN<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
</div>
</body>
</html>