[lustre-discuss] poor performance on reading small files

Mohr Jr, Richard Frank (Rick Mohr) rmohr at utk.edu
Wed Aug 3 08:32:09 PDT 2016


Do you have the Lustre read caching feature enabled?  I think it should be on by default, but you might want to check.  If the files are only 20 KB, then I would think the Lustre OSS nodes could keep them in memory most of the time to speed up access (unless of course this is a metadata bottleneck as Oliver suggested.)  Do your OSS nodes have a lot of memory?  Do you know what your typical memory usage is on the OSS nodes?

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu


> On Jul 28, 2016, at 10:19 PM, Riccardo Veraldi <Riccardo.Veraldi at cnaf.infn.it> wrote:
> 
> Hello,
> 
> I have a lustre cluster on rhel7, 6 OSS each of them has 3 OSTs and 1 MDS.
> 
> I am using lustre on ZFS.
> While write performances are excellent also on smaller files, I find there is a drop down in performance
> on reading 20KB files. Performance can go as low as 200MB/sec or even less.
> I am talking about random reads and random stride reads.
> I did the following to try to improve things:
> 	• disabled lnet debug messages:
> 		• sysctl -w lnet.debug=0 
> 	• increased dirty cache
> 		• lctl set_param osc.lutrefs\*.max_dirty_mb=256
> 	• increased number of RPC in flight
> 		• for i in `ls  /proc/fs/lustre/osc/lustrefs-OST00*/max_rpcs_in_flight`; do echo 32 > $i; done
> it did not improve reading 20KB file performances.
> I have to say in advance I did not set up any striping because I will have no more than 6 concurrent reads and writes,
> so striping is not that much important for me.
> Here the problem is that one single random read  of a 20KB file is around 190MB/s and this is really disappointing.
> I am open to any suggestion.
> I made sure it is not a ZFS problem, on the OSSs ZFS is performing like a charm locally.
> thank you
> 
> 
> Riccardo
> 
> 
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org




More information about the lustre-discuss mailing list