<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:2 0 5 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi everyone,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We recently saw some extremely high stat loads on our lustre FS. Output from “llstat -i 1 mdt” looked like:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">[root@hpfs-fsl-mds0 lustre]#
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier;color:black">/proc/fs/lustre/mds/MDS/mdt/stats @ 1530642446.366015124</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier;color:black">Name Cur.Count Cur.Rate #Events Unit last min avg max stddev</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier;color:black">req_waittime 182102 182102 22343734858[usec] 3951261 2 38.94 3027235 897.47 </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier;color:black">req_qdepth 182103 182103 22343734859[reqs] 12528 0 0.08 571 0.29 </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier;color:black">req_active 182103 182103 22343734859[reqs] 484211 1 2.88 99 1.50 </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier;color:black">req_timeout 182104 182104 22343734860[sec] 182104 1 9.32 36 13.44 </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:Courier;color:black">reqbuf_avail 437980 437980 55509906321[bufs] 27996863 32 63.89 65 0.47</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This was driving the load on our MDS up into the 100 to 200 range. Surprisingly, the MDS and the LFS from a client were still generally responsive. The numbers in the “Cur.Count” column are normally in the
100’s to 1000’s for our file system (we have ~600 lustre clients). The kiblnd_sd_* and ldlm_* processes were driving up the load.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We’ve tracked down the users causing this. There were two different workloads that we identified there were causing the problems. One of them was fairly common, the other is fairly infrequent. There are
a couple of things I wanted input on from the wider community. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">First, since one of the workloads is common for our lab and we haven’t seen this issue before (at least not to this extent), we think this might be related specifically to 2.10.4, which recently updated to.
We didn’t see anything in the changelog that was obviously related but if there are any other known issues or groups seeing this, that would be good to know. We are using ZFS on both the MDT and OST’s.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Also, the ldlm processes lead us to looking at flock vs localflock. On previous generations of our LFS’s, we used localflock. But on the current LFS, we decided to try flock instead. This LFS has been in
production for a couple years with no obvious problems due to flock but we decided to drop back to localflock as a precaution for now. We need to do a more controlled test but this does seem to help. What are other sites using for locking parameters?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Darby<o:p></o:p></span></p>
</div>
</body>
</html>