[Lustre-discuss] Lustre with 10GbE or Infiniband?

Scott Atchley atchley at myri.com
Thu Feb 12 05:28:13 PST 2009


On Feb 11, 2009, at 8:46 PM, Kevin Van Maren wrote:

> Charles Taylor wrote:
>> On Feb 11, 2009, at 4:35 PM, Scott Atchley wrote:
>>
>>> To add to Brian's comments, IB 4X SDR is limited to about 700-750  
>>> MB/s
>>> by the fabric. O2IBLND cannot go faster than minimum of either the
>>> fabric or PCI-E connection allow.
>>
>> Hmmm.   I can agree with the second part of that statement but I
>> question the first.   We've measured much closer to the 1GByte/sec
>> wire rate of IB using several different tools.  750 GBytes/sec
>> corresponds to roughly 6 GBits/sec.   You lose 2 of the 10 Gbits to
>> encoding (8B10) so line rate is really 8GBits/sec or 1 GByte/sec.
>> Yes, you'll lose some more to protocol and swtiching overhead but it
>> is not anywhere near an additional 2 GBits/sec - in our experience.
>
> Correct.  Infinipath SDR was getting ~980 MB/s, and DDR HCAs in SDR  
> mode
> can also do quite well in an x8 PCIe slot.
>
> The PCI-X HCAs were limited to around 850MB/s by the bus, and PCIe  
> HCAs
> _are_ likewise limited to around 700-750MB/s -- but only in a PCIe  
> x4 slot.
>
> DDR IB (unless using a PCIe gen2 connectX card, or a x16 Infinipath
> card) are also
> limited to around 1450-1600 MB/s by the PCIe x8 bus, with a wire speed
> of 2000 MB/s.
>
> QDR IB, in a Gen2 x8 PCIe slot, are also going to be limited to <<
> 4000MB/s line rate
> (should expect around twice the BW of the gen1 PCIe slots).
>
> The IB headers are very small, compared to a 2KB or 4KB packet size,  
> but
> the PCIe
> headers (and eg flow-control overhead) are quite large compared to a
> typical 256B packet size.
>
> To clarify one point: IB advertises the "signaling" rate, so the 10Gb
> includes the overhead
> bits, as 8 bits are encoded in a 10 bit representation for
> transmission.  So 10Gb/s = 1GB/s,
> with 10-bit bytes.  Ethernet, on the other hand, always advertises the
> "data" rate, so 10Gb
> Ethernet is 1.25GB/s (12.5Gb/s signaling rate), as there are 8 bits  
> in a
> byte.  Ethernet packet
> headers are also effectively a bit larger than for IB (with IFG,
> preamble, etc).
>
> Kevin

Thanks everyone! I get so confused by IB performance claims. :-)

Scott



More information about the lustre-discuss mailing list