[Lustre-discuss] lustre 2.2.0 statfs performance

[Yevheniy Demchenko]

On 04/06/2012 11:08 PM, Andreas Dilger wrote:
> On 2012-04-06, at 2:09 PM, Yevheniy Demchenko wrote:
>> On 04/06/2012 07:43 PM, Andreas Dilger wrote:
>>> On 2012-04-06, at 10:33 AM, Yevheniy Demchenko wrote:
>>>> On 04/06/2012 06:40 AM, Andreas Dilger wrote:
>>>>> On 2012-04-05, at 2:40 PM, Yevheniy Demchenko wrote:
>>>>>> We are experimenting with a new lustre-2.2.0 and surprisingly not
>>>>>> getting any statfs performance improvements. It's generally as poor as in previous versions. Also, statahead does not seem to make any
>>>>>> difference to statfs performance.
>>>>>>
>>>>>> #strace -r ls -l
>>>>>> ......
>>>>>>   0.000041 lstat("77193", {st_mode=S_IFREG|0644, st_size=4096, ...}) = 0
>>>>>>   0.000238 lgetxattr("77193", "security.selinux", 0x129d800, 255) =
>>>>>> -1 ENODATA (No data available)
>>>>> This is selinux trying to fetch xattrs.  A plague on selinux, IMHO, but we can't get rid of it.  The prefetching and caching of existing and negative  (i.e. non-existent) xattrs on the client is being discussed in http://bugs.whamcloud.com/browse/LU-549.
>>>> Thanks for reply, things are starting to be clearer. It seems,
>>>> that problem is not in selinux itself, but in not-caching xattrs
>>>> in lustre in general. if i understand correctly, statahead tries
>>>> to read ahead file attributes and put them in cache, so that client
>>>> would not contact MDS for getting them. But if we don't cache
>>>> xattrs, every tool like ls or rsync will contact mds for every
>>>> file as it tries to get acls and selinux attributes, which can not
>>>> be cached. Wouldn't it be better to catch the case when FS is
>>>> mounted with -o noacl and selinux is disabled on the client side
>>>> and immediately return EOPNOTSUPP without contacting MDS? We would
>>>> be happy to trade acls and xattrs for a better statfs performance.
>>> Yes, that makes sense.  It would probably be a simple patch in ll_getxattr_common() if you wanted to give it a try.
>> Some new results and patch:
>> 1. run (immediately after fs mount):
>> #time ls -l
>> real    0m33.170s
>> user    0m0.945s
>> sys     0m15.861s
>> 2. and subsequent runs:
>> #time ls -l
>> real    0m12.134s
>> user    0m0.819s
>> sys     0m10.491s
> Saving 10s for 100k files is quite respectable for such a simple patch.
> Could you please submit it to Gerrit for inclusion:
>
> http://wiki.whamcloud.com/display/PUB/Submitting+Changes
> http://wiki.whamcloud.com/display/PUB/Using+Gerrit
>
> You can use LU-549 for this change.
>
Submitted. Who should i add as a reviewer for this patch?
Thanks,

Ing. Yevheniy Demchenko
Senior Linux Administrator
UVT s.r.o.