<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Andreas,</p>
<p>Thanks for the reply.</p>
<p>Client version is 2.14.0_ddn98. Here is a plot of the <b>write_RPCs_in_flight</b>
plot. Snapshot every 50ms. The max for any of the samples for
any of the OSCs was 1. No RPCs in flight while the OSCs were
dumping memory. The number following the OSC name in the legends
is the sum of the <b>write_RPCs_in flight</b> for all the
intervals. To be honest, I have never really looked at the RPCs
in flight numbers. I'm running as a lowly user, so I don't have
access to any of the server data, so I have nothing on
osd-ldiskfs.*.brw_stats.</p>
<p> I should also point out that the backing storage on the servers
is SSD, so I would think the commiting to storage on the server
side should be pretty quick.<br>
</p>
<p>I'm trying to get a handle on how Linux buffer cache works.
Everything I find on the web is pretty old. Here's one from 2012.
<a class="moz-txt-link-freetext" href="https://lwn.net/Articles/495543/">https://lwn.net/Articles/495543/</a></p>
<p>Can someone point me to something more current, and perhaps
Lustre related?<br>
</p>
<p>As for images, I think the list server strips the images. In
previous postings, when I would include images , what I got back
when the list server broadcast it out had the iamges stripped.
I'll include the images and also a link to the image on DropBox.</p>
<p>Thanks again,</p>
<p>John<br>
</p>
<p><a class="moz-txt-link-freetext" href="https://www.dropbox.com/scl/fi/fgmz4wazr6it9q2aeo0mb/write_RPCs_in_flight.png?rlkey=d3ri2w2n7isggvn05se4j3a6b&dl=0">https://www.dropbox.com/scl/fi/fgmz4wazr6it9q2aeo0mb/write_RPCs_in_flight.png?rlkey=d3ri2w2n7isggvn05se4j3a6b&dl=0</a><br>
</p>
<p><img src="cid:part1.PiUqdd4c.fwv8080p@iodoctors.com" alt=""></p>
<p><br>
</p>
<p><img src="cid:part2.mvIni2mu.swT8uSSY@iodoctors.com" alt=""></p>
<div class="moz-cite-prefix">On 12/5/23 22:33, Andreas Dilger wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1A1FEABB-771A-4236-95E2-79433D09E5C7@ddn.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<br class="">
<div>On Dec 4, 2023, at 15:06, John Bauer <<a
href="mailto:bauerj@iodoctors.com"
class="moz-txt-link-freetext" moz-do-not-send="true">bauerj@iodoctors.com</a>>
wrote:<br class="">
<blockquote type="cite" class=""><br
class="Apple-interchange-newline">
<div class="">
<div class="">I have a an OSC caching question. I am
running a dd process which writes an 8GB file. The file
is on lustre, striped 8x1M. This is run on a system that
has 2 NUMA nodes (cpu sockets). All the data is apparently
stored on one NUMA node (node1 in the plot below) until
node1 runs out of free memory. Then it appears that dd
comes to a stop (no more writes complete) until lustre
dumps the data from the node1. Then dd continues writing,
but now the data is stored on the second NUMA node,
node0. Why does lustre go to the trouble of dumping node1
and then not use node1's memory, when there was always
plenty of free memory on node0?<br class="">
<br class="">
I'll forego the explanation of the plot. Hopefully it is
clear enough. If someone has questions about what the
plot is depicting, please ask.<br class="">
<br class="">
<a
href="https://www.dropbox.com/scl/fi/pijgnnlb8iilkptbeekaz/dd.png?rlkey=3abonv5tx8w5w5m08bn24qb7x&dl=0"
class="" moz-do-not-send="true">https://www.dropbox.com/scl/fi/pijgnnlb8iilkptbeekaz/dd.png?rlkey=3abonv5tx8w5w5m08bn24qb7x&dl=0</a><br
class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
Hi John,</div>
<div>thanks for your detailed analysis. It would be good to
include the client kernel and Lustre version in this case, as
the page cache behaviour can vary dramatically between different
versions.</div>
<div><br class="">
</div>
<div>The allocation of the page cache pages may actually be out of
the control of Lustre, since they are typically being allocated
by the kernel VM affine to the core where the process that is
doing the IO is running. It may be that the "dd" is rescheduled
to run on node0 during the IO, since the ptlrpcd threads will be
busy processing all of the RPCs during this time, and then dd
will start allocating pages from node0.</div>
<div><br class="">
</div>
<div>That said, it isn't clear why the client doesn't start
flushing the dirty data from cache earlier? Is it actually
sending the data to the OSTs, but then waiting for the OSTs to
reply that the data has been committed to the storage before
dropping the cache?</div>
<div><br class="">
</div>
<div>It would be interesting to plot the
osc.*.rpc_stats::write_rpcs_in_flight and ::pending_write_pages
to see if the data is already in flight. The
osd-ldiskfs.*.brw_stats on the server would also useful to graph
over the same period, if possible.</div>
<div><br class="">
</div>
<div>It *does* look like the "node1 dirty" is kept at a low value
for the entire run, so it at least appears that RPCs are being
sent, but there is no page reclaim triggered until memory is
getting low. Doing page reclaim is really the kernel's job, but
it seems possible that the Lustre client may not be suitably
notifying the kernel about the dirty pages and kicking it in the
butt earlier to clean up the pages.</div>
<div><br class="">
</div>
PS: my preference would be to just attach the image to the email
instead of hosting it externally, since it is only 55 KB. Is this
blocked by the list server?
<div class=""><br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color:
rgb(0, 0, 0); letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; word-wrap: break-word;
-webkit-nbsp-mode: space; line-break: after-white-space;"
class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color:
rgb(0, 0, 0); letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space:
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; word-wrap: break-word;
-webkit-nbsp-mode: space; line-break: after-white-space;"
class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color:
rgb(0, 0, 0); letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space:
normal; word-spacing: 0px; -webkit-text-stroke-width:
0px; text-decoration: none; word-wrap: break-word;
-webkit-nbsp-mode: space; line-break:
after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color:
rgb(0, 0, 0); letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;
word-wrap: break-word; -webkit-nbsp-mode: space;
line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0);
color: rgb(0, 0, 0); letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration:
none; word-wrap: break-word; -webkit-nbsp-mode:
space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0);
color: rgb(0, 0, 0); letter-spacing: normal;
text-align: start; text-indent: 0px;
text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; word-wrap: break-word;
-webkit-nbsp-mode: space; line-break:
after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</blockquote>
</body>
</html>