[Lustre-discuss] Lustre with 10GbE or Infiniband?
Scott Atchley
atchley at myri.com
Thu Feb 12 05:28:13 PST 2009
On Feb 11, 2009, at 8:46 PM, Kevin Van Maren wrote:
> Charles Taylor wrote:
>> On Feb 11, 2009, at 4:35 PM, Scott Atchley wrote:
>>
>>> To add to Brian's comments, IB 4X SDR is limited to about 700-750
>>> MB/s
>>> by the fabric. O2IBLND cannot go faster than minimum of either the
>>> fabric or PCI-E connection allow.
>>
>> Hmmm. I can agree with the second part of that statement but I
>> question the first. We've measured much closer to the 1GByte/sec
>> wire rate of IB using several different tools. 750 GBytes/sec
>> corresponds to roughly 6 GBits/sec. You lose 2 of the 10 Gbits to
>> encoding (8B10) so line rate is really 8GBits/sec or 1 GByte/sec.
>> Yes, you'll lose some more to protocol and swtiching overhead but it
>> is not anywhere near an additional 2 GBits/sec - in our experience.
>
> Correct. Infinipath SDR was getting ~980 MB/s, and DDR HCAs in SDR
> mode
> can also do quite well in an x8 PCIe slot.
>
> The PCI-X HCAs were limited to around 850MB/s by the bus, and PCIe
> HCAs
> _are_ likewise limited to around 700-750MB/s -- but only in a PCIe
> x4 slot.
>
> DDR IB (unless using a PCIe gen2 connectX card, or a x16 Infinipath
> card) are also
> limited to around 1450-1600 MB/s by the PCIe x8 bus, with a wire speed
> of 2000 MB/s.
>
> QDR IB, in a Gen2 x8 PCIe slot, are also going to be limited to <<
> 4000MB/s line rate
> (should expect around twice the BW of the gen1 PCIe slots).
>
> The IB headers are very small, compared to a 2KB or 4KB packet size,
> but
> the PCIe
> headers (and eg flow-control overhead) are quite large compared to a
> typical 256B packet size.
>
> To clarify one point: IB advertises the "signaling" rate, so the 10Gb
> includes the overhead
> bits, as 8 bits are encoded in a 10 bit representation for
> transmission. So 10Gb/s = 1GB/s,
> with 10-bit bytes. Ethernet, on the other hand, always advertises the
> "data" rate, so 10Gb
> Ethernet is 1.25GB/s (12.5Gb/s signaling rate), as there are 8 bits
> in a
> byte. Ethernet packet
> headers are also effectively a bit larger than for IB (with IFG,
> preamble, etc).
>
> Kevin
Thanks everyone! I get so confused by IB performance claims. :-)
Scott
More information about the lustre-discuss
mailing list