[Lustre-discuss] Disabling RDMA on an IB interface

[Liang Zhen]

On Feb 24, 2011, at 10:45 AM, Jeremy Filizetti wrote:

> As Chris mentioned your talking about two very different methods.  I think you can use netem with IPoIB but I have never tried it.  If you use connected mode I think your still technically doing RDMA but the maximum size (MTU)  is around 64k which isn't sufficient for higher latencies.  In the first few slides of my LUG presentation last year I have some graphs that show how RDMA performance is affected by latency and need to be increased to compensate for the bandwidth delay product (BDP).  If you do what to use IPoIB you can add a line similar the following in your /etc/modprobe.conf or a fille in /etc/modprobe.d directory:
> 
>   options lnet networks=tcp(ib0)
> 
> If you want to use RC QPs as ko2iblnd does, we use the following kernel parameters:
> 
>    options lnet networks=o2ib(ib0)
>    options ko2iblnd map_on_demand=2 peer_credits=128 credits=256 concurrent_sends=256 ntx=512 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1


If you have peer_credits=128, then I would suggest increase credits=1024 and ntx=2048, otherwise a couple of clients could consume all NI credits, concurrent_sends is not necessary here because o2iblnd will estimate proper value for it. 

Thanks
Liang