[lustre-discuss] Problem with raising osc.*.max_rpcs_in_flight

[Reinoud Bokhorst]

Hi all,

I have a problem with raising the osc.*.max_rpcs_in_flight client
setting on our Lustre 2.7.0. I am trying the increase the setting from
32 to 64 but according to osc.*.rpc_stats it isn't being used. The
statistics still stop at 31 rpcs with high write request numbers, e.g.

                        read                    write
rpcs in flight        rpcs   % cum % |       rpcs   % cum %
0:                       0   0   0   |          0   0   0
1:                    7293  38  38   |       2231  16  16
2:                    3872  20  59   |       1196   8  25
3:                    1851   9  69   |        935   6  31
--SNIP--
28:                      0   0 100   |         89   0  87
29:                      0   0 100   |         90   0  87
30:                      0   0 100   |         94   0  88
31:                      0   0 100   |       1573  11 100

I have modified some ko2iblnd driver parameters in an attempt to get it
working:

options ko2iblnd peer_credits=128 peer_credits_hiw=128 credits=2048
concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048
fmr_flush_trigger=512 fmr_cache=1

Specifically I raised peer_credits_hiw to 128 as I've understood that it
must be twice the value of max_rpcs_in_flight. Checking the module
parameters that were actually loaded, I noticed that it was set to 127.
So apparently it must be smaller than peers_credits. After noticing this
I tried setting max_rpcs_in_flight to 60 but that didn't help either.
Are there any other parameters affecting the max rpcs? Do all settings
have to be powers of 2?

Related question; documentation on the driver parameters and how it all
hangs together is rather scarce on the internet. Does anyone have some
good pointers?

Thanks,
Reinoud Bokhorst