<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Andreas,</p>
<p>Thanks for the reply. After days of debugging, thinking it was
my instrumentation code at the root of the problem, this morning I
discovered the jobs will sometimes have the 260 second pause even
when not using my instrumentation package. The intermittent
nature sent me on that wild goose chase. What's odd is now the
260 second delay shows up during the MPI_File_write_at() phase of
the job. I no longer invoke my package's option to set and get
the striping info for the file, so I am no longer calling
llapi_layout_get_by_fd(). This is an h5perf run:</p>
<p><font face="Courier New, Courier, monospace">mpirun -n 16 h5perf
-A mpiio -B 2K -e 256M -i 1 -p 16 -P 16 -w -x 128M -X 128M -I -o
~/h5perf/${output}.log</font></p>
<p>Here's a few of my instrumentation plots of the pwrite64() calls
for all the ranks, with file Position on the Y axis and time on
the X. Note that each rank does 131072 pwrite64() calls of 2k
bytes, strided by 32k. Lots of potential for lock contention.
There is always one rank that gets off to a good start. I suppose
because once it is in the lead it does not have to deal with lock
contention. What's strange about this run is that it is the rank
that is in the lead that hits the "road block" first and has the
260s delay before resuming its writes. Some of the trailing ranks
blow through the "road block" and continue writing. The other
ranks also block on the same area of the file and pause for the
260 seconds. Again, all ranks are on different nodes(hosts). Why
would some ranks pause and some not? Why would the lead rank even
pause at all? Today I will try to associate which OSTs/OSSs are
behind the road block. Is there something in /proc/fs/lustre/osc/
that sheds light on RTC timeouts?</p>
<p>John<br>
</p>
<p><img moz-do-not-send="false"
src="cid:part1.n7VvreNr.vNWEuv9E@iodoctors.com"
alt="https://www.dropbox.com/s/6uac96xhr79pj73/h5perf_1.png?dl=0"
width="787" height="519"></p>
<p><img moz-do-not-send="false"
src="cid:part2.QZ0jIWNi.8JCVper0@iodoctors.com"
alt="https://www.dropbox.com/s/tebm1iy0jipnqgx/h5perf_2.png?dl=0"
width="777" height="515"></p>
<img moz-do-not-send="false"
src="cid:part3.5JAOrzP3.zkViRapV@iodoctors.com"
alt="https://www.dropbox.com/s/ydgzstkg9qrk6z4/h5perf_3.png?dl=0"
width="784" height="521"><br>
<div class="moz-cite-prefix">On 11/24/22 20:47, Andreas Dilger
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:EE8D1104-8596-494D-8C93-7B50DB783C6C@ddn.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
On Nov 22, 2022, at 13:57, John Bauer <<a
href="mailto:bauerj@iodoctors.com" class="moz-txt-link-freetext"
moz-do-not-send="true">bauerj@iodoctors.com</a>> wrote:<br
class="">
<div>
<blockquote type="cite" class=""><br
class="Apple-interchange-newline">
<div class="">
<div class="">
<p class="">Hi all,</p>
<p class="">I am making a call to <font class=""
face="Courier New, Courier, monospace">
<b class="">llapi_layout_get_by_fd()</b></font> from
each rank of a 16 rank MPI job. One rank per node.
<br class="">
</p>
<p class="">About 75% of the time, one of the ranks,
typically rank 0, takes a very long time to complete
this call. I have placed fprintf() calls with wall
clock timers around the call. If it does take a long
time it is generally about 260 seconds. Otherwise it
takes only micro-seconds.</p>
<p class="">How I access llapi_layout_get_by_fd() :<br
class="">
</p>
<p class=""><font class="" face="Courier New, Courier,
monospace">liblustreapi = dlopen("liblustreapi.so",
RTLD_LAZY ) ;<br class="">
LLAPI.layout_get_by_fd = dlsym( liblustreapi,
"llapi_layout_get_by_fd" ) ;<br class="">
</font></p>
<p class=""><font class="" face="Courier New, Courier,
monospace">How I call llapi_layout_get_by_fd() :<br
class="">
if(dbg)fprintf(stderr,"%s %12.8f %s() before
LLAPI.layout_get_by_fd()\n",host,rtc(),__func__);<br
class="">
struct llapi_layout *layout =
(*LLAPI.layout_get_by_fd)( fd, 0);<br class="">
if(dbg)fprintf(stderr,"%s %12.8f %s() after
LLAPI.layout_get_by_fd()\n",host,rtc(),__func__);</font><br
class="">
</p>
<p class="">The resulting prints from rank 0 :<br class="">
</p>
<p class=""><font class="" face="Courier New, Courier,
monospace">r401i2n10 7.22477698
LustreLayout_get_by_fd() before
LLAPI.layout_get_by_fd()<br class="">
r401i2n10 269.52539992 LustreLayout_get_by_fd() after
LLAPI.layout_get_by_fd()</font></p>
<p class="">Any ideas on what might be triggering this.
The layout returned seems to be correct every time,
whether it takes a long time or not. The layout
returned has the correct striping information, but the
component has no OSTs as the component has yet to be
instantiated for the new file.</p>
</div>
</div>
</blockquote>
<div class="">
<div class="">
<div class=""><br class="">
</div>
</div>
</div>
Running under strace/ltrace would show where the slowdown is,
and Lustre kernel debug logs would be needed to isolate this to
a specific piece of code. Given the length of time it is likely
that an RPC is timing out (presumably nothing is printed on the
console logs), but you'd need to look at exactly what is
happening.</div>
<div><br class="">
</div>
<div>It's a *bit* strange, because this call is essentially
equivalent to "getxattr", but over the years a bunch of cruft
has been added and it is probably doing a lot more than it
should... You could potentially use (approximately):</div>
<div><br class="">
</div>
<div> fgetxattr(fd, "lustre.lov", buf, buflen);</div>
<div> llapi_layout_get_by_xattr(buf, buflen, 0);</div>
<div><br class="">
</div>
<div>but then we wouldn't know what is making this slow and you
couldn't submit a patch to fix it...</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0,
0, 0); letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; word-spacing:
0px; -webkit-text-stroke-width: 0px; text-decoration: none;
word-wrap: break-word; -webkit-nbsp-mode: space; line-break:
after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color:
rgb(0, 0, 0); letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; word-wrap: break-word;
-webkit-nbsp-mode: space; line-break: after-white-space;"
class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color:
rgb(0, 0, 0); letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space:
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; word-wrap: break-word;
-webkit-nbsp-mode: space; line-break: after-white-space;"
class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color:
rgb(0, 0, 0); letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space:
normal; word-spacing: 0px; -webkit-text-stroke-width:
0px; text-decoration: none; word-wrap: break-word;
-webkit-nbsp-mode: space; line-break:
after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color:
rgb(0, 0, 0); letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;
word-wrap: break-word; -webkit-nbsp-mode: space;
line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0);
color: rgb(0, 0, 0); letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration:
none; word-wrap: break-word; -webkit-nbsp-mode:
space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</blockquote>
</body>
</html>