[Lustre-discuss] Infiniband question...

[Kevin Van Maren]

Corey Kovacs wrote:
> I am in the early planning stages of putting together a lustre system
> and when I look at IB speeds it seems most if not all the 40Gb IB
> cards are running over PCIe with 8 lanes. If PCIe lanes are good for
> about 200MB/s that means I can only get 1600MB's tops from one of
> these cards, but 40Gb xlates to around 4GB/s. Am I reasoning this
> correctly or is have the potential of the card wasted?
>   

PCIe "gen1" x8 will peak at around 1400-1600 MB/s, while "gen2" will 
peak around 3.2GB/s.
So you need to make sure the server supports the new gen2 (5GT/s) PCIe 
bus with 8 lanes.
For example, the Oracle Sun Fire X4170/X4270 servers support the faster 
PCIe bus.

[Note that the raw BW is the same (PCIe x8 and DDR 4x IB; and PCIe x8 gen2
and QDR IB4x), but PCIe implementations have smaller maximum packet sizes so
the effective BW is a bit lower due to higher overhead.]

> I can see using 40Gb switches to get the backplane speed and reduce
> contention on the switch but what are people using on the hosts?
>
>
> Corey
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>