[Lustre-discuss] Disabling RDMA on an IB interface

[Jeremy Filizetti]

As Chris mentioned your talking about two very different methods.  I think
you can use netem with IPoIB but I have never tried it.  If you use
connected mode I think your still technically doing RDMA but the maximum
size (MTU)  is around 64k which isn't sufficient for higher latencies.  In
the first few slides of my LUG presentation last year I have some graphs
that show how RDMA performance is affected by latency and need to be
increased to compensate for the bandwidth delay product (BDP).  If you do
what to use IPoIB you can add a line similar the following in your
/etc/modprobe.conf or a fille in /etc/modprobe.d directory:

  options lnet networks=tcp(ib0)

If you want to use RC QPs as ko2iblnd does, we use the following kernel
parameters:

   options lnet networks=o2ib(ib0)
   options ko2iblnd map_on_demand=2 peer_credits=128 credits=256
concurrent_sends=256 ntx=512 fmr_pool_size=2048 fmr_flush_trigger=512
fmr_cache=1

I'm not aware of any patches that add delay to the OFED stack.  At NRL we
use real delay between sites and hardware simulated delay using Obsidian
Longbow XRs (http://www.obsidianresearch.com).  They support a single SDR
connection and can add delay to the IB channel up to 1 second.

Jeremy


On Wed, Feb 23, 2011 at 5:19 PM, Andreas Dilger <adilger at whamcloud.com>wrote:

> On 2011-02-23, at 8:44 AM, Alvaro Aguilera wrote:
> > I want to do some tests injecting artificial latency to LNET using netem
> over an InfiniBand network. This doesn't work out of the box, since LNET's
> RDMA transfers aren't affected by the kernel latency. Does anyway know if
> there is a way to disable LNET's RDMA support and force its packets through
> the kernel? Other ideas are also welcomed.
>
> Depending on what you are trying to measure, you may want to contact some
> of the folks at NRL, who have done a bunch of IB WAN latency testing.  They
> inject latency into the network using special hardware, so that probably
> won't help you, but maybe they already measured what you are looking at :-).
>
> Cheers, Andreas
> --
> Andreas Dilger
> Principal Engineer
> Whamcloud, Inc.
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20110223/634381a9/attachment.htm>