[Lustre-discuss] Disabling RDMA on an IB interface

[Alvaro Aguilera]

Thanks for all your suggestions. TCPoIB seems to be doing the job.

Regards,
Alvaro.



On Thu, Feb 24, 2011 at 4:16 AM, Liang Zhen <liang at whamcloud.com> wrote:

>
> On Feb 24, 2011, at 10:45 AM, Jeremy Filizetti wrote:
>
> > As Chris mentioned your talking about two very different methods.  I
> think you can use netem with IPoIB but I have never tried it.  If you use
> connected mode I think your still technically doing RDMA but the maximum
> size (MTU)  is around 64k which isn't sufficient for higher latencies.  In
> the first few slides of my LUG presentation last year I have some graphs
> that show how RDMA performance is affected by latency and need to be
> increased to compensate for the bandwidth delay product (BDP).  If you do
> what to use IPoIB you can add a line similar the following in your
> /etc/modprobe.conf or a fille in /etc/modprobe.d directory:
> >
> >   options lnet networks=tcp(ib0)
> >
> > If you want to use RC QPs as ko2iblnd does, we use the following kernel
> parameters:
> >
> >    options lnet networks=o2ib(ib0)
> >    options ko2iblnd map_on_demand=2 peer_credits=128 credits=256
> concurrent_sends=256 ntx=512 fmr_pool_size=2048 fmr_flush_trigger=512
> fmr_cache=1
>
>
> If you have peer_credits=128, then I would suggest increase credits=1024
> and ntx=2048, otherwise a couple of clients could consume all NI credits,
> concurrent_sends is not necessary here because o2iblnd will estimate proper
> value for it.
>
> Thanks
> Liang
>
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20110225/a650a399/attachment.htm>