<div dir="auto">Are you flushing cache between test runs?</div><br><div class="gmail_quote"><div dir="ltr">On Mon, Apr 2, 2018, 6:06 PM John Bauer <<a href="mailto:bauerj@iodoctors.com">bauerj@iodoctors.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
I am running dd 10 times consecutively to read a 64GB file (
stripeCount=4 stripeSize=4M ) on a Lustre client(version 2.10.3)
that has 64GB of memory.<br>
The client node was dedicated.<br>
<br>
<b><font face="Courier New, Courier, monospace">for pass in 1 2 3 4
5 6 7 8 9 10<br>
do<br>
of=/dev/null if=${file} count=128000 bs=512K<br>
done<br>
</font></b><br>
Instrumentation of the I/O from dd reveals varying performance. In
the plot below, the bottom frame has wall time<br>
on the X axis, and file position of the dd reads on the Y axis, with
a dot plotted at the wall time and starting file position of every
read. <br>
The slopes of the lines indicate the data transfer rate, which vary
from 475MB/s to 1.5GB/s. The last 2 passes have sharp breaks<br>
in the performance, one with increasing performance, and one with
decreasing performance.<br>
<br>
The top frame indicates the amount of memory used by each of the
file's 4 OSCs over the course of the 10 dd runs. Nothing terribly
odd here except that<br>
one of the OSC's eventually has its entire stripe ( 16GB ) cached
and then never gives any up.<br>
<br>
I should mention that the file system has 320 OSTs. I found LU-6370
which eventually started discussing LRU management issues on systems
with high<br>
numbers of OST's leading to reduced RPC sizes.<br>
<br>
Any explanations for the varying performance?<br>
Thanks, <br>
John<br>
<br>
<img src="cid:part1.FE7755F6.C36ADB75@iodoctors.com" alt="">
<pre class="m_179390239102776222moz-signature" cols="72">--
I/O Doctors, LLC
507-766-0378
<a class="m_179390239102776222moz-txt-link-abbreviated" href="mailto:bauerj@iodoctors.com" target="_blank" rel="noreferrer">bauerj@iodoctors.com</a></pre>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div>