[lustre-discuss] poor performance on reading small files

[Ben Evans]

I thought read caching was disabled by default, as the kernel's default
handling of pages was better.  It's been awhile since I looked at those
test results.

On 8/3/16, 11:32 AM, "lustre-discuss on behalf of Mohr Jr, Richard Frank
(Rick Mohr)" <lustre-discuss-bounces at lists.lustre.org on behalf of
rmohr at utk.edu> wrote:

>Do you have the Lustre read caching feature enabled?  I think it should
>be on by default, but you might want to check.  If the files are only 20
>KB, then I would think the Lustre OSS nodes could keep them in memory
>most of the time to speed up access (unless of course this is a metadata
>bottleneck as Oliver suggested.)  Do your OSS nodes have a lot of memory?
> Do you know what your typical memory usage is on the OSS nodes?
>
>--
>Rick Mohr
>Senior HPC System Administrator
>National Institute for Computational Sciences
>http://www.nics.tennessee.edu
>
>
>> On Jul 28, 2016, at 10:19 PM, Riccardo Veraldi
>><Riccardo.Veraldi at cnaf.infn.it> wrote:
>> 
>> Hello,
>> 
>> I have a lustre cluster on rhel7, 6 OSS each of them has 3 OSTs and 1
>>MDS.
>> 
>> I am using lustre on ZFS.
>> While write performances are excellent also on smaller files, I find
>>there is a drop down in performance
>> on reading 20KB files. Performance can go as low as 200MB/sec or even
>>less.
>> I am talking about random reads and random stride reads.
>> I did the following to try to improve things:
>> 	€ disabled lnet debug messages:
>> 		€ sysctl -w lnet.debug=0
>> 	€ increased dirty cache
>> 		€ lctl set_param osc.lutrefs\*.max_dirty_mb=256
>> 	€ increased number of RPC in flight
>> 		€ for i in `ls
>>/proc/fs/lustre/osc/lustrefs-OST00*/max_rpcs_in_flight`; do echo 32 >
>>$i; done
>> it did not improve reading 20KB file performances.
>> I have to say in advance I did not set up any striping because I will
>>have no more than 6 concurrent reads and writes,
>> so striping is not that much important for me.
>> Here the problem is that one single random read  of a 20KB file is
>>around 190MB/s and this is really disappointing.
>> I am open to any suggestion.
>> I made sure it is not a ZFS problem, on the OSSs ZFS is performing like
>>a charm locally.
>> thank you
>> 
>> 
>> Riccardo
>> 
>> 
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
>_______________________________________________
>lustre-discuss mailing list
>lustre-discuss at lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org