[Lustre-discuss] performance tuning w/ dbench, bonnie++

[Andreas Dilger]

On 2009-12-21, at 10:57, Jay Christopherson wrote:
> However, no matter what our configuration, dbench never shows any  
> improvement.  It never shows more than 25MB/sec, with 5 clients.  I  
> *KNOW* we are getting better throughput, but I need to be able to  
> prove it.

Dbench should not be considered a performance benchmark, but rather  
only a load testing tool.

The main reason dbench is not a good benchmark tool is that it creates  
and deletes files fairly rapidly, and if the file has never been  
written to disk before it is deleted, it wills still count this IO in  
the "bandwidth" number even though no bytes hit the disk.  In real  
world usage it is fairly uncommon that files are created and deleted  
in such a short time.  Compilers used to be the common usage example  
for this, but today they dump temp files to tmpfs (RAM-based) and not  
to disk filesystems.

Lustre is an "eager writer" in that it submits data to the server as  
soon as it has a full RPC worth of data.  Local filesystems often  
delay writing between 5-30s, which is fine because they control the VM  
directly in conjunction with the disk IO.  For Lustre, which sometimes  
has 10000 clients, delaying the writes by 30s would mean interfering  
with application traffic on the network, and would also waste  
30*{server bandwidth} of data that could already have been written to  
disk.

In the end, the most important metric you need to look at is whether  
your application has seen improvement now that you restructured your  
Lustre config.  If it has, then end of story.  If it hasn't, then your  
application is doing something that Lustre doesn't like very much  
(e.g. small IO, concurrent appends to the same file).  It might be  
possible to tune Lustre further, depending on what it is your  
application is doing, but it is pointless to optimize for dbench,  
since it is unlikely to be doing exactly what your application is doing.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.