<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p>It definitely is limited to 32 buckets. We've toyed with raising that limit (and Cray did so internally), but it does use some memory, etc.</p>
<p><br>
</p>
<p>So that's almost certainly the issue you're seeing, Reinoud. RPCs larger than the largest size appear as the largest size.</p>
<p><br>
</p>
<p>- Patrick</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Dilger, Andreas <andreas.dilger@intel.com><br>
<b>Sent:</b> Sunday, July 2, 2017 3:45:03 AM<br>
<b>To:</b> Reinoud Bokhorst<br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Problem with raising osc.*.max_rpcs_in_flight</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">It may also be that this histogram is limited to 32 buckets?<br>
<br>
Cheers, Andreas<br>
<br>
> On Jun 30, 2017, at 03:03, Reinoud Bokhorst <rbokhorst@astron.nl> wrote:<br>
> <br>
> Hi all,<br>
> <br>
> I have a problem with raising the osc.*.max_rpcs_in_flight client<br>
> setting on our Lustre 2.7.0. I am trying the increase the setting from<br>
> 32 to 64 but according to osc.*.rpc_stats it isn't being used. The<br>
> statistics still stop at 31 rpcs with high write request numbers, e.g.<br>
> <br>
> read write<br>
> rpcs in flight rpcs % cum % | rpcs % cum %<br>
> 0: 0 0 0 | 0 0 0<br>
> 1: 7293 38 38 | 2231 16 16<br>
> 2: 3872 20 59 | 1196 8 25<br>
> 3: 1851 9 69 | 935 6 31<br>
> --SNIP--<br>
> 28: 0 0 100 | 89 0 87<br>
> 29: 0 0 100 | 90 0 87<br>
> 30: 0 0 100 | 94 0 88<br>
> 31: 0 0 100 | 1573 11 100<br>
> <br>
> I have modified some ko2iblnd driver parameters in an attempt to get it<br>
> working:<br>
> <br>
> options ko2iblnd peer_credits=128 peer_credits_hiw=128 credits=2048<br>
> concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048<br>
> fmr_flush_trigger=512 fmr_cache=1<br>
> <br>
> Specifically I raised peer_credits_hiw to 128 as I've understood that it<br>
> must be twice the value of max_rpcs_in_flight. Checking the module<br>
> parameters that were actually loaded, I noticed that it was set to 127.<br>
> So apparently it must be smaller than peers_credits. After noticing this<br>
> I tried setting max_rpcs_in_flight to 60 but that didn't help either.<br>
> Are there any other parameters affecting the max rpcs? Do all settings<br>
> have to be powers of 2?<br>
> <br>
> Related question; documentation on the driver parameters and how it all<br>
> hangs together is rather scarce on the internet. Does anyone have some<br>
> good pointers?<br>
> <br>
> Thanks,<br>
> Reinoud Bokhorst<br>
> <br>
> <br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> lustre-discuss@lists.lustre.org<br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</div>
</span></font>
</body>
</html>