<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Thanks for your answers. I should check the JIRA first next time
:) It is unfortunate not to have the histogram above 32, are
there other means to get insight in that data?</p>
<p><br>
</p>
<p>In the mean time I found in the code that peer_credits_hiw must
be between peer_credits/2 and (peer_credits-1).That clarifies that
part assuming it doesn't affect rpc limits.<br>
</p>
<br>
Regards Reinoud<br>
<br>
<br>
<div class="moz-cite-prefix">On 07/03/17 16:52, Cory Spitz wrote:<br>
</div>
<blockquote type="cite"
cite="mid:46DD992C-8390-40B2-8D6E-B27B2038BFE2@cray.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Apple Color Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.emailquote, li.emailquote, div.emailquote
{mso-style-name:emailquote;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:1.0pt;
margin-bottom:.0001pt;
border:none;
padding:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal">This issue is tracked under <a
href="https://jira.hpdd.intel.com/browse/LU-4533"
moz-do-not-send="true">
https://jira.hpdd.intel.com/browse/LU-4533</a>.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We submitted, but then abandoned a patch
because of memory usage. Andreas proposed to do dynamic
allocation of the obd_histogram buckets, but that work isn’t
completed yet. Although, it shouldn’t be too hard. I just
added the ‘easy’ label to LU-4533 <span
style="font-family:"Apple Color Emoji"">☺</span>.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Cory<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:12.0pt;color:black">From: </span></b><span
style="font-size:12.0pt;color:black">lustre-discuss
<a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss-bounces@lists.lustre.org"><lustre-discuss-bounces@lists.lustre.org></a> on behalf
of Patrick Farrell <a class="moz-txt-link-rfc2396E" href="mailto:paf@cray.com"><paf@cray.com></a><br>
<b>Date: </b>Monday, July 3, 2017 at 8:29 AM<br>
<b>To: </b>Andreas Dilger
<a class="moz-txt-link-rfc2396E" href="mailto:andreas.dilger@intel.com"><andreas.dilger@intel.com></a>, Reinoud Bokhorst
<a class="moz-txt-link-rfc2396E" href="mailto:rbokhorst@astron.nl"><rbokhorst@astron.nl></a><br>
<b>Cc: </b><a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss@lists.lustre.org">"lustre-discuss@lists.lustre.org"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss@lists.lustre.org"><lustre-discuss@lists.lustre.org></a><br>
<b>Subject: </b>Re: [lustre-discuss] Problem with raising
osc.*.max_rpcs_in_flight<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div id="x_divtagdefaultwrapper">
<p><span style="font-size:12.0pt;color:black">It definitely
is limited to 32 buckets. We've toyed with raising that
limit (and Cray did so internally), but it does use some
memory, etc.<o:p></o:p></span></p>
<p><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p><span style="font-size:12.0pt;color:black">So that's
almost certainly the issue you're seeing, Reinoud. RPCs
larger than the largest size appear as the largest size.<o:p></o:p></span></p>
<p><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p><span style="font-size:12.0pt;color:black">- Patrick<o:p></o:p></span></p>
</div>
<div class="MsoNormal" style="text-align:center"
align="center">
<hr size="2" align="center" width="98%">
</div>
<div id="x_divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span
style="color:black"> lustre-discuss
<a class="moz-txt-link-rfc2396E" href="mailto:lustre-discuss-bounces@lists.lustre.org"><lustre-discuss-bounces@lists.lustre.org></a> on
behalf of Dilger, Andreas
<a class="moz-txt-link-rfc2396E" href="mailto:andreas.dilger@intel.com"><andreas.dilger@intel.com></a><br>
<b>Sent:</b> Sunday, July 2, 2017 3:45:03 AM<br>
<b>To:</b> Reinoud Bokhorst<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> Re: [lustre-discuss] Problem with
raising osc.*.max_rpcs_in_flight</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt">It may
also be that this histogram is limited to 32 buckets?<br>
<br>
Cheers, Andreas<br>
<br>
> On Jun 30, 2017, at 03:03, Reinoud Bokhorst
<a class="moz-txt-link-rfc2396E" href="mailto:rbokhorst@astron.nl"><rbokhorst@astron.nl></a> wrote:<br>
> <br>
> Hi all,<br>
> <br>
> I have a problem with raising the
osc.*.max_rpcs_in_flight client<br>
> setting on our Lustre 2.7.0. I am trying the increase
the setting from<br>
> 32 to 64 but according to osc.*.rpc_stats it isn't
being used. The<br>
> statistics still stop at 31 rpcs with high write
request numbers, e.g.<br>
> <br>
> read write<br>
> rpcs in flight rpcs % cum % | rpcs %
cum %<br>
> 0: 0 0 0 | 0
0 0<br>
> 1: 7293 38 38 | 2231
16 16<br>
> 2: 3872 20 59 | 1196
8 25<br>
> 3: 1851 9 69 | 935
6 31<br>
> --SNIP--<br>
> 28: 0 0 100 | 89
0 87<br>
> 29: 0 0 100 | 90
0 87<br>
> 30: 0 0 100 | 94
0 88<br>
> 31: 0 0 100 | 1573 11
100<br>
> <br>
> I have modified some ko2iblnd driver parameters in an
attempt to get it<br>
> working:<br>
> <br>
> options ko2iblnd peer_credits=128
peer_credits_hiw=128 credits=2048<br>
> concurrent_sends=256 ntx=2048 map_on_demand=32
fmr_pool_size=2048<br>
> fmr_flush_trigger=512 fmr_cache=1<br>
> <br>
> Specifically I raised peer_credits_hiw to 128 as I've
understood that it<br>
> must be twice the value of max_rpcs_in_flight.
Checking the module<br>
> parameters that were actually loaded, I noticed that
it was set to 127.<br>
> So apparently it must be smaller than peers_credits.
After noticing this<br>
> I tried setting max_rpcs_in_flight to 60 but that
didn't help either.<br>
> Are there any other parameters affecting the max
rpcs? Do all settings<br>
> have to be powers of 2?<br>
> <br>
> Related question; documentation on the driver
parameters and how it all<br>
> hangs together is rather scarce on the internet. Does
anyone have some<br>
> good pointers?<br>
> <br>
> Thanks,<br>
> Reinoud Bokhorst<br>
> <br>
> <br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> <a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
> <a
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></span></p>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
<br>
</body>
</html>