[Lustre-discuss] Multiple IB ports

[Andreas Dilger]

On 2011-03-21, at 10:18 AM, Sebastien Piechurski wrote: 
> From my understanding, but confirmation from more skilled people on the list would be welcomed, using multiple IB ports with a lustre client will be difficult to manage, and will probably not bring any performance improvements.
> I was told by a colleague that there were currently too many internal locks in the clients to sustain a big throughput. Lustre is designed for global throughput on many clients, but not on individual clients.
> I can observe this on my site, where I have enough storage and servers to reach 21GB/s globally, but am unable to get more than 300MB/s on a single client even though the DDR IB network would sustain +800MB/s ...

There must be something wrong with your configuration or the code has some bug, because we have had single clients doing 2GB/s in the past.  What version of Lustre did you test on?

Is this a single-threaded write?  With single-threaded IO the bottleneck often happens in the kernel copy_{to,from}_user() that is copying data to/from userspace in order to do data caching in the client.  Having multiple threads doing the IO allows multiple cores to do the data copying.

Is the lustre debugging disabled?  "lctl set_param debug=0" if this helps.

Is the Lustre network checksum disabled?  "lctl set_param osc.*.checksums=0"  There is a patch to allow hardware-assisted checksums, but it needs some debugging before it can be landed into the production release.


> From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Brian O'Connor
> Sent: lundi 21 mars 2011 04:53
> To: lustre-discuss at lists.lustre.org
> Subject: [Lustre-discuss] Multiple IB ports
> 
> Hi,
>     Any body actually using multiple IB ports on a client for an aggregated connection?
>  
> Ie. Many oss with one qdr IB each. Clients with 4 qdr IB ports. Assuming the normal
> issues with bus bandwidth etc, what sort of perf can I expect
>  
> qdr ~ 3-4Gbytes/Sec
>  
> I’m trying to size a cluster and clients to get ~10GBytes/Sec on *one*
> client node.
>  
> If I can aggregate IB linearly the next step will be to try and figure out
> How to get 10Gigabytes/s to local storage L
>  
>  
> Some times customers are crazy…….
>  
>  
>  
> Brian O'Connor
>  
> -------------------------------------------------
>  
> SGI Consulting
>  
> Email: briano at sgi.com, Mobile +61 417 746 452
>  
> Phone: +61 3 9963 1900, Fax: +61 3 9963 1902
>  
> 357 Camberwell Road, Camberwell, Victoria, 3124
>  
> AUSTRALIA http://www.sgi.com/support/services
>  
> -------------------------------------------------
>  
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss


Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.