[Lustre-discuss] Multiple IB ports

[Sebastien Piechurski]

Thanks for the correction.

I guess I need to redo some benchs, and go through the tunables ....

> -----Original Message-----
> From: Andreas Dilger [mailto:adilger at whamcloud.com] 
> Sent: lundi 21 mars 2011 12:38
> To: Sebastien Piechurski
> Cc: Brian O'Connor; lustre-discuss at lists.lustre.org
> Subject: Re: [Lustre-discuss] Multiple IB ports
> 
> On 2011-03-21, at 10:18 AM, Sebastien Piechurski wrote: 
> > From my understanding, but confirmation from more skilled 
> people on the list would be welcomed, using multiple IB ports 
> with a lustre client will be difficult to manage, and will 
> probably not bring any performance improvements.
> > I was told by a colleague that there were currently too 
> many internal locks in the clients to sustain a big 
> throughput. Lustre is designed for global throughput on many 
> clients, but not on individual clients.
> > I can observe this on my site, where I have enough storage 
> and servers to reach 21GB/s globally, but am unable to get 
> more than 300MB/s on a single client even though the DDR IB 
> network would sustain +800MB/s ...
> 
> There must be something wrong with your configuration or the 
> code has some bug, because we have had single clients doing 
> 2GB/s in the past.  What version of Lustre did you test on?
> 
> Is this a single-threaded write?  With single-threaded IO the 
> bottleneck often happens in the kernel copy_{to,from}_user() 
> that is copying data to/from userspace in order to do data 
> caching in the client.  Having multiple threads doing the IO 
> allows multiple cores to do the data copying.
> 
> Is the lustre debugging disabled?  "lctl set_param debug=0" 
> if this helps.
> 
> Is the Lustre network checksum disabled?  "lctl set_param 
> osc.*.checksums=0"  There is a patch to allow 
> hardware-assisted checksums, but it needs some debugging 
> before it can be landed into the production release.
> 
> 
> > From: lustre-discuss-bounces at lists.lustre.org 
> [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of 
> Brian O'Connor
> > Sent: lundi 21 mars 2011 04:53
> > To: lustre-discuss at lists.lustre.org
> > Subject: [Lustre-discuss] Multiple IB ports
> > 
> > Hi,
> >     Any body actually using multiple IB ports on a client 
> for an aggregated connection?
> >  
> > Ie. Many oss with one qdr IB each. Clients with 4 qdr IB 
> ports. Assuming the normal
> > issues with bus bandwidth etc, what sort of perf can I expect
> >  
> > qdr ~ 3-4Gbytes/Sec
> >  
> > I'm trying to size a cluster and clients to get 
> ~10GBytes/Sec on *one*
> > client node.
> >  
> > If I can aggregate IB linearly the next step will be to try 
> and figure out
> > How to get 10Gigabytes/s to local storage L
> >  
> >  
> > Some times customers are crazy.......
> >  
> >  
> >  
> > Brian O'Connor
> >  
> > -------------------------------------------------
> >  
> > SGI Consulting
> >  
> > Email: briano at sgi.com, Mobile +61 417 746 452
> >  
> > Phone: +61 3 9963 1900, Fax: +61 3 9963 1902
> >  
> > 357 Camberwell Road, Camberwell, Victoria, 3124
> >  
> > AUSTRALIA http://www.sgi.com/support/services
> >  
> > -------------------------------------------------
> >  
> > _______________________________________________
> > Lustre-discuss mailing list
> > Lustre-discuss at lists.lustre.org
> > http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
> 
> Cheers, Andreas
> --
> Andreas Dilger 
> Principal Engineer
> Whamcloud, Inc.
> 
> 
> 
>