[Lustre-discuss] lustre 2.2.0 statfs performance

[Yevheniy Demchenko]

On 04/06/2012 06:40 AM, Andreas Dilger wrote:
> On 2012-04-05, at 2:40 PM, Yevheniy Demchenko wrote:
>> We are experimenting with a new lustre-2.2.0 and surprisingly not
>> getting any statfs performance improvements. It's generally as poor as
>> in previous versions. Also, statahead does not seem to make any
>> difference to statfs performance.
>>
>> Simple ls -l test for a 100000 files directory was done on the node23
>> with statahead on and off:
>> -----statahead off-----
>> #time ls -l
>> real    0m52.751s
>> user    0m1.153s
>> sys     0m20.101s
>>
>> #time ls -l
>> real    0m21.280s
>> user    0m1.086s
>> sys     0m11.973s
>> All subsequent runs complete in virtually the same time (21-24s).
>>
>> -----statahead on-----
>> #time ls -l
>> real    0m43.846s
>> user    0m1.242s
>> sys     0m20.444s
>>
>> #time ls -l
>> real    0m24.000s
>> user    0m1.104s
>> sys     0m14.125s
>> All subsequent runs complete in virtually the same time (21-24s).
> It looks to me like a 25% improvement was seen - 53s to 44s for first run.  After that, the directories/inodes/locks/attributes are all in the client cache, so statahead is no longer active (no RPCs need to be sent).
>
>> #strace -r ls -l
>> ......
>>     0.000041 lstat("77193", {st_mode=S_IFREG|0644, st_size=4096, ...}) = 0
>>     0.000238 lgetxattr("77193", "security.selinux", 0x129d800, 255) =
>> -1 ENODATA (No data available)
> This is selinux trying to fetch xattrs.  A plague on selinux, IMHO, but we can't get rid of it.  The prefetching and caching of existing and negative  (i.e. non-existent) xattrs on the client is being discussed in http://bugs.whamcloud.com/browse/LU-549.
Thanks for reply, things are starting to be clearer. It seems, that
problem is not in selinux itself, but in not-caching xattrs in lustre in
general. if i understand correctly, statahead tries to read ahead file
attributes and put them in cache, so that client would not contact mds
for getting them. But if we don't cache xattrs, every tool like ls or
rsync will contact mds for every file as it tries to get acls and
selinux attributes, which can not be cached. Wouldn't it be better to
catch the case when FS is mounted with -o noacl and selinux is disabled
on the client side and immediately return EOPNOTSUPP without contacting
mds? We would be happy to trade acls and xattrs for a better statfs
performance.
>
> Cheers, Andreas
> --
> Andreas Dilger                       Whamcloud, Inc.
> Principal Lustre Engineer            http://www.whamcloud.com/
>
>
>
>
Ing. Yevheniy Demchenko
Senior Linux Administrator
UVT s.r.o.