[lustre-discuss] Problem with raising osc.*.max_rpcs_in_flight

[Dilger, Andreas]

It may also be that this histogram is limited to 32 buckets?

Cheers, Andreas

> On Jun 30, 2017, at 03:03, Reinoud Bokhorst <rbokhorst at astron.nl> wrote:
> 
> Hi all,
> 
> I have a problem with raising the osc.*.max_rpcs_in_flight client
> setting on our Lustre 2.7.0. I am trying the increase the setting from
> 32 to 64 but according to osc.*.rpc_stats it isn't being used. The
> statistics still stop at 31 rpcs with high write request numbers, e.g.
> 
>                        read                    write
> rpcs in flight        rpcs   % cum % |       rpcs   % cum %
> 0:                       0   0   0   |          0   0   0
> 1:                    7293  38  38   |       2231  16  16
> 2:                    3872  20  59   |       1196   8  25
> 3:                    1851   9  69   |        935   6  31
> --SNIP--
> 28:                      0   0 100   |         89   0  87
> 29:                      0   0 100   |         90   0  87
> 30:                      0   0 100   |         94   0  88
> 31:                      0   0 100   |       1573  11 100
> 
> I have modified some ko2iblnd driver parameters in an attempt to get it
> working:
> 
> options ko2iblnd peer_credits=128 peer_credits_hiw=128 credits=2048
> concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048
> fmr_flush_trigger=512 fmr_cache=1
> 
> Specifically I raised peer_credits_hiw to 128 as I've understood that it
> must be twice the value of max_rpcs_in_flight. Checking the module
> parameters that were actually loaded, I noticed that it was set to 127.
> So apparently it must be smaller than peers_credits. After noticing this
> I tried setting max_rpcs_in_flight to 60 but that didn't help either.
> Are there any other parameters affecting the max rpcs? Do all settings
> have to be powers of 2?
> 
> Related question; documentation on the driver parameters and how it all
> hangs together is rather scarce on the internet. Does anyone have some
> good pointers?
> 
> Thanks,
> Reinoud Bokhorst
> 
> 
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org