[lustre-discuss] Lustre routing help needed

[Mohr Jr, Richard Frank (Rick Mohr)]

> On Oct 30, 2017, at 8:47 AM, Kevin M. Hildebrand <kevin at umd.edu> wrote:
> 
> All of the hosts (client, server, router) have the following in ko2iblnd.conf:
> 
> alias ko2iblnd-opa ko2iblnd
> options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4
> 
> install ko2iblnd /usr/sbin/ko2iblnd-probe

Those parameters will only get applied to omnipath interfaces (which you don’t have), so everything you have should just be running with default parameters.  Since your lnet routers have a different version of lustre than your servers/clients, it might be possible that the default values for the ko2iblnd parameters are different between the two versions.  You can always check this by looking at the values in the files under /sys/module/ko2iblnd/parameters.  It might be worthwhile to compare those values on the lnet routers to the values on the servers to see if maybe there is a difference that could affect the behavior.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu