[Lustre-discuss] Multiple IB ports

[Atul Vidwansa]

Hi Brian,

With one 4x QDR IB port, you can achieve 2 GB/Sec on single client, multi-threaded workload provided that you have right storage (with enough bandwidth) at other end.  We have tested this multiple times at DDN.

I have seen sites that do IB-bonding across 2 ports but mostly in failover configuration. To get 10GB/Sec to a single node requires aggregating 5 QDR IB ports. You will need to confirm from your IB vendor (Mellanox? ), OS vendor (SGI/RedHat/Novell) and Lustre vendor whether they support aggregating so many links.  I think the challenge you will have is to find a Lustre client node that has enough x8 PCIe slots to sustain 3 dual-port Infiniband adapters at full rate (think multiple such nodes in a typical Lustre filesystem, not so economical). Other alternative is to find a server that can support 8X or 12X QDR IB port on the motherboard to get more bandwidth.

With a typical Lustre client memory of 24-64GB and memory to CPU bandwidth of 10GB/Sec (with standard DDR3-1333MHz  DIMMS), it is not possible to fit dataset larger than 2/3rd  of memory. If you still want to achieve 10GB/Sec of bandwidth between storage and memory, there are clever alternatives. You will have to stage your data into memory beforehand and keep memory pages locked and continue feeding data as these pages are consumed. It is lot harder than it seems on the paper.

Cheers,
-Atul


From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Brian O'Connor
Sent: Monday, 21 March 2011 9:23 AM
To: lustre-discuss at lists.lustre.org
Subject: [Lustre-discuss] Multiple IB ports

Hi,
    Any body actually using multiple IB ports on a client for an aggregated connection?

Ie. Many oss with one qdr IB each. Clients with 4 qdr IB ports. Assuming the normal
issues with bus bandwidth etc, what sort of perf can I expect

qdr ~ 3-4Gbytes/Sec

I'm trying to size a cluster and clients to get ~10GBytes/Sec on *one*
client node.

If I can aggregate IB linearly the next step will be to try and figure out
How to get 10Gigabytes/s to local storage :(


Some times customers are crazy.......



Brian O'Connor

-------------------------------------------------

SGI Consulting

Email: briano at sgi.com<mailto:briano at sgi.com>, Mobile +61 417 746 452

Phone: +61 3 9963 1900, Fax: +61 3 9963 1902

357 Camberwell Road, Camberwell, Victoria, 3124

AUSTRALIA http://www.sgi.com/support/services

-------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20110321/4d779bbd/attachment.htm>