<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Nov 22, 2022, at 13:57, John Bauer <<a href="mailto:bauerj@iodoctors.com" class="">bauerj@iodoctors.com</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="">
<p class="">Hi all,</p>
<p class="">I am making a call to <font face="Courier New, Courier,
monospace" class="">
<b class="">llapi_layout_get_by_fd()</b></font> from each rank of a 16 rank MPI job. One rank per node.
<br class="">
</p>
<p class="">About 75% of the time, one of the ranks, typically rank 0, takes a very long time to complete this call. I have placed fprintf() calls with wall clock timers around the call. If it does take a long time it is generally about 260 seconds. Otherwise
it takes only micro-seconds.</p>
<p class="">How I access llapi_layout_get_by_fd() :<br class="">
</p>
<p class=""><font face="Courier New, Courier, monospace" class="">liblustreapi = dlopen("liblustreapi.so", RTLD_LAZY ) ;<br class="">
LLAPI.layout_get_by_fd = dlsym( liblustreapi, "llapi_layout_get_by_fd" ) ;<br class="">
</font></p>
<p class=""><font face="Courier New, Courier, monospace" class="">How I call llapi_layout_get_by_fd() :<br class="">
if(dbg)fprintf(stderr,"%s %12.8f %s() before LLAPI.layout_get_by_fd()\n",host,rtc(),__func__);<br class="">
struct llapi_layout *layout = (*LLAPI.layout_get_by_fd)( fd, 0);<br class="">
if(dbg)fprintf(stderr,"%s %12.8f %s() after LLAPI.layout_get_by_fd()\n",host,rtc(),__func__);</font><br class="">
</p>
<p class="">The resulting prints from rank 0 :<br class="">
</p>
<p class=""><font face="Courier New, Courier, monospace" class="">r401i2n10 7.22477698 LustreLayout_get_by_fd() before LLAPI.layout_get_by_fd()<br class="">
r401i2n10 269.52539992 LustreLayout_get_by_fd() after LLAPI.layout_get_by_fd()</font></p>
<p class="">Any ideas on what might be triggering this. The layout returned seems to be correct every time, whether it takes a long time or not. The layout returned has the correct striping information, but the component has no OSTs as the component has yet
to be instantiated for the new file.</p>
</div>
</div>
</blockquote>
<div class="">
<div class="">
<div class=""><br class="">
</div>
</div>
</div>
Running under strace/ltrace would show where the slowdown is, and Lustre kernel debug logs would be needed to isolate this to a specific piece of code. Given the length of time it is likely that an RPC is timing out (presumably nothing is printed on the console
logs), but you'd need to look at exactly what is happening.</div>
<div><br class="">
</div>
<div>It's a *bit* strange, because this call is essentially equivalent to "getxattr", but over the years a bunch of cruft has been added and it is probably doing a lot more than it should... You could potentially use (approximately):</div>
<div><br class="">
</div>
<div> fgetxattr(fd, "lustre.lov", buf, buflen);</div>
<div> llapi_layout_get_by_xattr(buf, buflen, 0);</div>
<div><br class="">
</div>
<div>but then we wouldn't know what is making this slow and you couldn't submit a patch to fix it...</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>