[Lustre-discuss] Multiple IB ports

Paul Nowoczynski pauln at psc.edu
Mon Mar 21 08:28:35 PDT 2011


Hi Brian,
I don't think it's crazy to strive for that rate, especially when there 
are machines on the market which can accommodate multiple TB's of 
memory. Assuming my math is mostly correct, to load or unload a 16TB 
data set into a single machine (with 16TB of memory) would take about an 
hour and a half with a single QDR interface:

(16*1024^4)/(3.4*1000^3)/60 == 86 mins

The ratio of memory capacity to I/O bandwidth is a critical issue for 
most large machines. Typically in HPC, we'd like to dump all of memory 
in 5 to 10 minutes.
thanks,
paul


Brian O'Connor wrote:
>
> Hi,
>
> Any body actually using multiple IB ports on a client for an 
> aggregated connection?
>
> Ie. Many oss with one qdr IB each. Clients with 4 qdr IB ports. 
> Assuming the normal
>
> issues with bus bandwidth etc, what sort of perf can I expect
>
> qdr ~ 3-4Gbytes/Sec
>
> I’m trying to size a cluster and clients to get ~10GBytes/Sec on **one**
>
> client node.
>
> If I can aggregate IB linearly the next step will be to try and figure out
>
> How to get 10Gigabytes/s to local storage L
>
> Some times customers are crazy…….
>
> Brian O'Connor
>
> -------------------------------------------------
>
> SGI Consulting
>
> Email: briano at sgi.com <mailto:briano at sgi.com>, Mobile +61 417 746 452
>
> Phone: +61 3 9963 1900, Fax: +61 3 9963 1902
>
> 357 Camberwell Road, Camberwell, Victoria, 3124
>
> AUSTRALIA http://www.sgi.com/support/services
>
> -------------------------------------------------
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss




More information about the lustre-discuss mailing list