<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mv="http://macVmlSchemaUri" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Apple Color Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.emailquote, li.emailquote, div.emailquote
{mso-style-name:emailquote;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:1.0pt;
margin-bottom:.0001pt;
border:none;
padding:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">This issue is tracked under <a href="https://jira.hpdd.intel.com/browse/LU-4533">
https://jira.hpdd.intel.com/browse/LU-4533</a>.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We submitted, but then abandoned a patch because of memory usage. Andreas proposed to do dynamic allocation of the obd_histogram buckets, but that work isn’t completed yet. Although, it shouldn’t be too hard. I just added the ‘easy’
label to LU-4533 <span style="font-family:"Apple Color Emoji"">☺</span>.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Cory<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Patrick Farrell <paf@cray.com><br>
<b>Date: </b>Monday, July 3, 2017 at 8:29 AM<br>
<b>To: </b>Andreas Dilger <andreas.dilger@intel.com>, Reinoud Bokhorst <rbokhorst@astron.nl><br>
<b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] Problem with raising osc.*.max_rpcs_in_flight<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div id="x_divtagdefaultwrapper">
<p><span style="font-size:12.0pt;color:black">It definitely is limited to 32 buckets. We've toyed with raising that limit (and Cray did so internally), but it does use some memory, etc.<o:p></o:p></span></p>
<p><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p><span style="font-size:12.0pt;color:black">So that's almost certainly the issue you're seeing, Reinoud. RPCs larger than the largest size appear as the largest size.<o:p></o:p></span></p>
<p><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p><span style="font-size:12.0pt;color:black">- Patrick<o:p></o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Dilger, Andreas <andreas.dilger@intel.com><br>
<b>Sent:</b> Sunday, July 2, 2017 3:45:03 AM<br>
<b>To:</b> Reinoud Bokhorst<br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Problem with raising osc.*.max_rpcs_in_flight</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt">It may also be that this histogram is limited to 32 buckets?<br>
<br>
Cheers, Andreas<br>
<br>
> On Jun 30, 2017, at 03:03, Reinoud Bokhorst <rbokhorst@astron.nl> wrote:<br>
> <br>
> Hi all,<br>
> <br>
> I have a problem with raising the osc.*.max_rpcs_in_flight client<br>
> setting on our Lustre 2.7.0. I am trying the increase the setting from<br>
> 32 to 64 but according to osc.*.rpc_stats it isn't being used. The<br>
> statistics still stop at 31 rpcs with high write request numbers, e.g.<br>
> <br>
> read write<br>
> rpcs in flight rpcs % cum % | rpcs % cum %<br>
> 0: 0 0 0 | 0 0 0<br>
> 1: 7293 38 38 | 2231 16 16<br>
> 2: 3872 20 59 | 1196 8 25<br>
> 3: 1851 9 69 | 935 6 31<br>
> --SNIP--<br>
> 28: 0 0 100 | 89 0 87<br>
> 29: 0 0 100 | 90 0 87<br>
> 30: 0 0 100 | 94 0 88<br>
> 31: 0 0 100 | 1573 11 100<br>
> <br>
> I have modified some ko2iblnd driver parameters in an attempt to get it<br>
> working:<br>
> <br>
> options ko2iblnd peer_credits=128 peer_credits_hiw=128 credits=2048<br>
> concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048<br>
> fmr_flush_trigger=512 fmr_cache=1<br>
> <br>
> Specifically I raised peer_credits_hiw to 128 as I've understood that it<br>
> must be twice the value of max_rpcs_in_flight. Checking the module<br>
> parameters that were actually loaded, I noticed that it was set to 127.<br>
> So apparently it must be smaller than peers_credits. After noticing this<br>
> I tried setting max_rpcs_in_flight to 60 but that didn't help either.<br>
> Are there any other parameters affecting the max rpcs? Do all settings<br>
> have to be powers of 2?<br>
> <br>
> Related question; documentation on the driver parameters and how it all<br>
> hangs together is rather scarce on the internet. Does anyone have some<br>
> good pointers?<br>
> <br>
> Thanks,<br>
> Reinoud Bokhorst<br>
> <br>
> <br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> lustre-discuss@lists.lustre.org<br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></span></p>
</div>
</div>
</body>
</html>