<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
One last email on this. Creating 4 files to read, each using one of
the OSC that the stripeCount=4 file is using, and reading in a round<br>
robin fashion, is what I was working on when I got distracted by the
numctl notion. I just finished up the 4 file read round robin
read. Nothing terribly surprising<br>
except that reading the same amount of data (16GiB per OSC) from the
4 OSCs is about 10% faster ( max of 2.5GiB/s ) than<br>
reading from a 64GiB 4 way striped file. In the table farther
below, the amount of blocked time for the individual files does
point to 2 OSC's being responsible<br>
for the lion's share of blocked I/O time. ( I named the files with
the OSC that they resides on ).<br>
<br>
Again, thanks for your input.<br>
John<br>
<br>
<br>
<img src="cid:part1.2513864E.6FF31712@iodoctors.com" alt=""><br>
<br>
<br>
Sorted by Blocked.<br>
<img src="cid:part2.C3A942B4.04D2FEEC@iodoctors.com" alt=""><br>
<br>
Code for reading in round robin fashion<br>
<b><font face="Courier New, Courier, monospace">#include
<fcntl.h><br>
#include <stdio.h><br>
#include <unistd.h><br>
#include <stdlib.h><br>
<br>
#define BSIZE (512*1024)<br>
#define NREC 128000<br>
<br>
int main( int argc, char **argv ) {<br>
int fd[16] ;<br>
int i ;<br>
int nfile = argc -1 ;<br>
<br>
for(i=0;i<nfile;i++){<br>
fd[i] = open64( argv[i+1] , O_RDONLY|O_LARGEFILE ) ;<br>
fprintf(stderr,"%s() fs[%d]=%d\n",__func__,i,fd[i]);<br>
if( fd[i] < 0 ) exit(-1) ;<br>
}<br>
<br>
char *buff = malloc(BSIZE) ;<br>
<br>
for(i=0;i<NREC;i++){<br>
int f = fd[i%nfile] ;<br>
off64_t off = (off64_t)(i/nfile) * BSIZE ;<br>
ssize_t ret = pread64(f,buff,BSIZE,off) ;<br>
if( ret < 0 ) break ;<br>
}<br>
<br>
for(i=0;i<nfile;i++){<br>
close( fd[i] ) ;<br>
}<br>
}</font></b><br>
<br>
<br>
<div class="moz-cite-prefix">On 4/5/2018 12:33 PM, John Bent wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGfFL-xj1=ZZPNa8jTz7uswyDkA5qi5yRVcX-5oCkU4jRpggvA@mail.gmail.com">
<div dir="ltr">"I suspect that this OSC is using an OSS that is
under heavier load."
<div><br>
</div>
<div>If you want to confirm this, it seems like you could create
files with striping parameters such that you have a single
file on each OSS. Well, I know you can make stripe=1 so it's
only on one OSS but can you control/query on *which* OSS is
the stripe? Assuming you can, then you just benchmark
performance for each file (i.e. OSS) and you can discover more
explicitly whether you have a slow OSS. </div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Apr 5, 2018 at 9:31 AM, John
Bauer <span dir="ltr"><<a
href="mailto:bauerj@iodoctors.com" target="_blank"
moz-do-not-send="true">bauerj@iodoctors.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Rick,<br>
<br>
Thanks for reply. Also thanks to Patrick Farrell for
making me rethink this.<br>
<br>
I am coming to believe that it is an OSS issue. Every
time I run this job, the first pass of dd is slow, which I
now attribute to all the OSSs<br>
needing to initially read the data in from disk to OSS
cache. If the subsequent passes of dd get back soon
enough I then observe good performance.<br>
If not, performance goes back to initial rates. <br>
<br>
Inspecting the individual wait times for each of the dd
reads for one of the poor performing dd passes, <br>
and correlating them to the OSC that fulfills each
individual dd read, I see that 75% of the wait time is
from the a single OSC. I suspect that this OSC is<br>
using an OSS that is under heavier load.<br>
<br>
I don't have access to the OSS so I cant report on the
Lustre settings. I think the client side max cached is
50% of memory.<br>
<br>
After speaking with Doug Petesch of Cray, I though I
would look into numa effects on this job. I now also
monitor the contents of<br>
<b><font face="Courier New, Courier, monospace">/sys/devices/system/node/node?<wbr>/meminfo
</font></b><br>
and ran the job with <font face="Courier New, Courier,
monospace"><b>numactl --cpunodebind=0</b></font><br>
Interestingly enough, I now sometimes get dd transfer
rates of 2.2GiB/s. Plotting the
.../node?/meminfo[FilePages] value versus time for the 2
cpunodes shows that the<br>
data is now mostly placed on node0. Unfortunately, the
variable rates still remain, as one would expect if it is
an OSS caching issue, but the poor performance is also
better.<br>
<br>
Resulting plot with all <b><font face="Courier New,
Courier, monospace">numactl --cpunodebind=0</font></b><br>
<img src="cid:part4.09831B82.F38EEA3A@iodoctors.com"
alt="" class=""><br>
<br>
Resulting plots with <b><font face="Courier New, Courier,
monospace">numactl --cpunodebind=x</font></b> where x
alternates between 0 and 1 for each subsequent dd pass.
And indeed, the file pages migrate between cpunodes.<br>
<br>
<img src="cid:part5.6B3BCFBA.3D3416D4@iodoctors.com"
alt="" class="">
<div>
<div class="h5"><br>
<br>
<div class="m_-6103604654120049206moz-cite-prefix">On
4/5/2018 9:22 AM, Mohr Jr, Richard Frank (Rick Mohr)
wrote:<br>
</div>
<blockquote type="cite">
<pre>John,
I had a couple of thoughts (though not sure if they are directly relevant to your performance issue):
1) Do you know what caching settings are applied on the lustre servers? This could have an impact on performance, especially if your tests are being run while others are doing IO on the system.
2) It looks like there is a parameter called llite.<fsname>.max_cached_mb that controls how much client side data is cached. According to the manual, the default value is 3/4 of the host’s RAM (which would be 48GB in your case). I don’t know why the cache seems to be used unevenly between your 4 OSTs, but it might explain why the cache for some OSTs decrease when others increase.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
<a class="m_-6103604654120049206moz-txt-link-freetext" href="http://www.nics.tennessee.edu" target="_blank" moz-do-not-send="true">http://www.nics.tennessee.edu</a>
</pre>
<blockquote type="cite">
<pre>On Apr 2, 2018, at 8:06 PM, John Bauer <a class="m_-6103604654120049206moz-txt-link-rfc2396E" href="mailto:bauerj@iodoctors.com" target="_blank" moz-do-not-send="true"><bauerj@iodoctors.com></a> wrote:
I am running dd 10 times consecutively to read a 64GB file ( stripeCount=4 stripeSize=4M ) on a Lustre client(version 2.10.3) that has 64GB of memory.
The client node was dedicated.
for pass in 1 2 3 4 5 6 7 8 9 10
do
of=/dev/null if=${file} count=128000 bs=512K
done
Instrumentation of the I/O from dd reveals varying performance. In the plot below, the bottom frame has wall time
on the X axis, and file position of the dd reads on the Y axis, with a dot plotted at the wall time and starting file position of every read.
The slopes of the lines indicate the data transfer rate, which vary from 475MB/s to 1.5GB/s. The last 2 passes have sharp breaks
in the performance, one with increasing performance, and one with decreasing performance.
The top frame indicates the amount of memory used by each of the file's 4 OSCs over the course of the 10 dd runs. Nothing terribly odd here except that
one of the OSC's eventually has its entire stripe ( 16GB ) cached and then never gives any up.
I should mention that the file system has 320 OSTs. I found LU-6370 which eventually started discussing LRU management issues on systems with high
numbers of OST's leading to reduced RPC sizes.
Any explanations for the varying performance?
Thanks,
John
<johbmffmkkegkbkh.png>
--
I/O Doctors, LLC
507-766-0378
<a class="m_-6103604654120049206moz-txt-link-abbreviated" href="mailto:bauerj@iodoctors.com" target="_blank" moz-do-not-send="true">bauerj@iodoctors.com</a>
______________________________<wbr>_________________
lustre-discuss mailing list
<a class="m_-6103604654120049206moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org" target="_blank" moz-do-not-send="true">lustre-discuss@lists.lustre.<wbr>org</a>
<a class="m_-6103604654120049206moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank" moz-do-not-send="true">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a>
</pre>
</blockquote>
</blockquote>
<br>
<pre class="m_-6103604654120049206moz-signature" cols="72">--
I/O Doctors, LLC
507-766-0378
<a class="m_-6103604654120049206moz-txt-link-abbreviated" href="mailto:bauerj@iodoctors.com" target="_blank" moz-do-not-send="true">bauerj@iodoctors.com</a></pre>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org"
moz-do-not-send="true">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
I/O Doctors, LLC
507-766-0378
<a class="moz-txt-link-abbreviated" href="mailto:bauerj@iodoctors.com">bauerj@iodoctors.com</a></pre>
</body>
</html>