[Lustre-discuss] Lustre with 10GbE or Infiniband?

[Charles Taylor]

On Feb 11, 2009, at 4:35 PM, Scott Atchley wrote:

> To add to Brian's comments, IB 4X SDR is limited to about 700-750 MB/s
> by the fabric. O2IBLND cannot go faster than minimum of either the
> fabric or PCI-E connection allow.

Hmmm.   I can agree with the second part of that statement but I  
question the first.   We've measured much closer to the 1GByte/sec  
wire rate of IB using several different tools.  750 GBytes/sec   
corresponds to roughly 6 GBits/sec.   You lose 2 of the 10 Gbits to  
encoding (8B10) so line rate is really 8GBits/sec or 1 GByte/sec.      
Yes, you'll lose some more to protocol and swtiching overhead but it  
is not anywhere near an additional 2 GBits/sec - in our experience.

Just ran a quick IMB (formerly Pallas) between a couple of our SDR  
nodes and got 860 MBytes/sec (ping-pong, 4MB).   So I don't think  
there is anything inherent in SDR IB that limits you to 750 MBytes/ 
sec.   However, running IPoIB will  probably limit you to something  
even less than that which is why you should use the O2IBLND if you  
want the real benefit of IB.

Just our experience,

Charlie Taylor
UF HPC Center