[Lustre-discuss] Single client performance

[Andreas Dilger]

On 2010-03-11, at 21:55, David Simas wrote:
> I'm using Lustre 1.8.2-ext4 on RHEL5 x86_64.  I've got four OSSs each
> with one OST, in hardware RAID 6.  Lustre runs on a 10 Gb network
> between the servers and clients.  For the OSSs, iozone tells me that
> I can write into ext4 on the RAID arrays I configured at 750 MB/s.
> Indeed, the write peformance of ext4 vs. ext3 is one reason I'm using
> 1.8.2-ext4.

Note that the "ext3" used by Lustre for ldiskfs is more like ext4.   
Also note that the userspace IO performance of ldiskfs-ext3 is poor,  
but Lustre uses its own IO submission from within the kernel to avoid  
performance bottlenecks of the VFS.  I still expect ldiskfs-ext4 to be  
as fast/faster than ldiskfs-ext3, but probably not by the margin that  
you are seeing from userspace testing.

>  I can also confirm (using ttcp) that the 10 Gb network
> between the clients and severs can do 750 MB/s.  So I don't think that
> the network or disk/controller/RAID configurations are holding me
> back.

What is the CPU usage of the client & server?  Use "top", then press  
"1" to get per-CPU stats.  For single-threaded IO, TCP + user->kernel  
data copy overhead can saturate a single core, leaving other cores  
idle.  Running with multiple IO threads, and using an RDMA-capable  
network (IB is the most popular) will definitely avoid the CPU  
bottleneck.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.