[Lustre-discuss] MDS network traffic question

[Kevin Van Maren]

I would replace the 1GigE with the 10GigE: have all Ethernet traffic go over the 10GigE links, rather than add another "tcp1" network for Lustre.  This will keep your configuration much simpler, and make the migration as painless as possible (just move the IP address to the 10GigE port on the servers).

The MDS traffic _volume_ is much lower than it is for the OSS nodes.  The big win from 10GigE would be the lower latency: if you approach 100MB/s of MDS traffic, you have much bigger problems than a 10GigE NIC can solve.

Kevin


On Oct 11, 2011, at 3:54 PM, James Robnett wrote:

> 
>   We have a small lustre install consisting of an MDS and 5 OSS servers.
> Historically the MDS and OSS servers had both a 1Gbit ethernet interface
> (tcp0) to workstations and a QDR IB interface (ib0) to our cluster.
> 
>   We're planning on adding a MTU 9000 10Gbit ethernet (tcp1) interface
> to the MDS and OSS nodes and workstations for faster access.  Our
> software has a pretty high IO to CPU component.
> 
>   I just discovered that our MDS can't in fact take another PCIe 8x
> card but it does have a spare GigE port.  The 10gbit Ethernet switch
> can support 1gbit and 10gbit interfaces.
> 
>   We'd then have 3 networks
> tcp0 at 1gbit to slow clients
> tcp1 at 10gbit to faster clients
> ib0 to cluster
> 
>   My question is:
> 
>   Is there a risk of congestion or overrunning that 2nd GigE MDS 
> interface if our workstations and OSS servers communicate over tcp1 at
> 10gbit but the MDS tcp1 is connected at 1Gbit.  The bulk of our traffic
> will continue to be between the cluster and lustre over IB but the
> workstations can trivially over run ethernet hence the desire for
> 10gbit between them and the OSSes.
> 
>   My gut feeling is it should be fine, particularly with the larger MTU,
> there's not that much traffic to the MDS but I'd easily believe it if
> somebody said it's risky thing to do.
> 
>   The alternative is to buy a new MDS and swap disks into it.
> 
> James Robnett
> National Radio Astronomy Observatory
> Array Operations Center
> 


Confidentiality Notice: This e-mail message, its contents and any attachments to it are confidential to the intended recipient, and may contain information that is privileged and/or exempt from disclosure under applicable law. If you are not the intended recipient, please immediately notify the sender and destroy the original e-mail message and any attachments (and any copies that may have been made) from your system or otherwise. Any unauthorized use, copying, disclosure or distribution of this information is strictly prohibited.