<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mv="http://macVmlSchemaUri" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New",serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier",serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">This sounds exactly like what we ran into when we upgraded to 2.9 (and is still present in 2.10). See these:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2017-May/014524.html">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2017-May/014524.html</a><o:p></o:p></p>
<p class="MsoNormal"><a href="https://jira.hpdd.intel.com/browse/LU-9574">https://jira.hpdd.intel.com/browse/LU-9574</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The mailing list thread describes our problem a little more and gives a workaround (reverting a specific commit that seems to cause the problem). In the LU, Jinshan uploaded a patch about 1.5 weeks ago that seems to fix this for us. It
would be good to know if this helps your situation too. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of John Bauer <bauerj@iodoctors.com><br>
<b>Date: </b>Thursday, August 31, 2017 at 7:52 PM<br>
<b>To: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[lustre-discuss] sudden read performance drop on sequential forward read.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">All,<br>
<br>
I have an application that writes a 100GB file forwards, and then begins a sequence of reading a 70 GB section of the file forwards and backwards. At some point in the run,
<br>
not always at the same point, the read performance degrades significantly. The initial forward reads are about 1.3 GB/s. The backwards reads about 300 MB/s. In an instant,
<br>
the forward read performance drops to 2.8 MB/s. From about 250 seconds on, this is the only file that is being read or written by the application, running on a dedicated client node.<br>
The file has a stripe count of 4, and stripe size of 512KB. If the stripe count is changed to 1, this behavior does not present itself. The cpu usage is minimal during the period of degraded performance.
<br>
The LNET traffic is also about 2.8 MB/s during the period of degraded performance. The system has 64GB of memory, meaning Lustre can not cache the entire 70GB active set of the file that is being read.
<br>
The Lustre client version is 2.9.0.<br>
<br>
Any ideas what could be causing this? What should I be watching in the /proc/fs/lustre file system to find some clues?<br>
<br>
The behavior is depicted in the image below, which shows the file position as a function of wall clock time. The writes and reads are of size 512KB.<br>
<br>
Thanks,<br>
<br>
John<br>
<br>
<br>
<br>
<img border="0" width="1506" height="868" id="_x0000_i1025" src="cid:image001.png@01D32297.312AF020"><o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>I/O Doctors, LLC<o:p></o:p></pre>
<pre>507-766-0378<o:p></o:p></pre>
<pre><a href="mailto:bauerj@iodoctors.com">bauerj@iodoctors.com</a><o:p></o:p></pre>
</div>
</body>
</html>