[Lustre-discuss] Benchmarking small file performance

[Andreas Dilger]

On May 17, 2008  13:47 -0300, Balagopal Pillai wrote:
> On Sat, 17 May 2008, Ragnar Kjørstad wrote:
>      Postmark is a good benchmark for small file peformance. I have tried 
> it with gfs and lustre before for comparison. Lustre doesn't shine well in 
> that benchmark. But when the file size is set a little high, lustre does pick 
> up quite a bit.

Do you have numbers at what file size Lustre does well with postmark?
Given HTML email with large attachments, the days that average email
size for many users is getting quite large.

> > On Thu, May 15, 2008 at 04:21:56PM -0400, Rick Friedman wrote:
> > > We build a Lustre appliance that comes fully configured and tuned. A
> > > number of customers have asked us about small file performance. We are
> > > looking for feedback on how people "quantify" this type of performance
> > > with their Lustre based systems. In most cases we are looking at
> > > applications that are reading and writing files in the 1K to 20M size
> > > range. 
> > 
> > One tool commonly used to compare file system operation performance for
> > local filesystems and NFS clients is bonnie++
> > (http://www.coker.com.au/bonnie++/). It's not really designed for
> > paralell filesystems but at least it will quantify single-client lustre
> > performance in a way that your customers can relate to. 
> > 
> > Note that since bonnie++ only runs on a single machine at the time you
> > need some kind of wrapper to run it on a multi-client lustre cluster.
> > There is a very primitive one available at
> > http://ragnark.vestdata.no/download/dbonnie++.txt, but what you should
> > do is make a version of bonnie++ that use MPI for syncronization instead
> > of semaphores, so that it can easily be run on clusters.


Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.