[Lustre-discuss] Switch for lustre clients

Mike Berg mike.berg at sun.com
Tue Aug 26 21:55:50 PDT 2008


Hi,

There are important considerations on how you put your IB fabric  
together otherwise you can have a congested IB fabric which could  
result in overall cluster stability and performance issues.

If you need or want to maximize the bandwidth and latency  
characteristics of Infiniband, then you need to consider a full CLOS  
fabric. In short a full CLOS topology, also called "fat tree", is a  
fully connected, non-blocking topology, which provides the same  
latency and bandwidth to any endpoints of the fabric. If you Google  
for "CLOS fabric" you will see many references.

So, a 288 port switch is typically 24 switch blades. Each blade is  
really a 24 port switch, with 12 client ports and 12 ports plugging  
into a 2 stage internal IB fabric. So a 288 port switch is a 3 stage  
switch. This same switch can be created by using individual stand  
alone 24 port switches, with the same cross section bandwidth and  
latency characteristics as the 288 port switch. The problem with  
building a 288 port fabric in this manner is the two stages that  
normally would be part of the internal workings of a 288 port switch  
are now external with many connectors and cables. This adds complexity  
for both the physical implementation as well as trouble shooting bad  
cables. In this case, from a cost perspective you are likely better  
off using the ISR 9288 switch. However, a Voltaire representative  
should be able to go over the details and trade offs with you.

You can also create a CLOS fabric that has a blocking factor, which  
when done correctly can reduce the number of switches required, and  
still provide great bandwidth and consistent latency. This is also  
something your Voltaire representative should be able to go over in  
detail with you.

For the Sun Data Center Switch 3x24, consider this switch as it is  
described, three 24 port switches but with the advantage of reducing  
the number of cable connections when compared to traditional 24 port  
switches. This reduction in number of cables can ease building the 288  
port fabric I describe above using conventional 24 port switches. So  
when building your fabric consideration needs to be taken with how  
connections are made to maintain the blocking factor you choose. Be  
sure to work out the details with your Sun representative, they should  
be able to formulate the fabric layout that meets your performance  
requirements.

Regards,
Mike

On Aug 26, 2008, at 8:46 PM, Minh Hien wrote:

> Dear all,
> I'll have around 230 lustre clients for 30TB on infinity band. I  
> wonder what kind of switch should be used to maintain performance  
> and reliability to that scale of clients.
>
> Currently, I have 4 choices. The first 3 choices are using 288-port  
> switch, Voltaire Grid Director Switch ISR 9288 (10GB Infinity  
> band) , Grid Director ISR 2012 (20GB Infinity band), or Sun Data  
> Centre Switch 3x24.
>
> The last choice is we could use smaller switch and connect them up,  
> such as 96-port Voltaire Grid Director ISR 9096. It seems a cheaper  
> solution, however, i suspect it could raise performance issue.
>
> With your experience, could you give me some guidelines on how to  
> design the lustre client in this case? Thank you very much.
>
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss




More information about the lustre-discuss mailing list