[Lustre-discuss] lustre 2.2.0 statfs performance

[Andreas Dilger]

On 2012-04-06, at 2:09 PM, Yevheniy Demchenko wrote:
> On 04/06/2012 07:43 PM, Andreas Dilger wrote:
>> On 2012-04-06, at 10:33 AM, Yevheniy Demchenko wrote:
>>> On 04/06/2012 06:40 AM, Andreas Dilger wrote:
>>>> On 2012-04-05, at 2:40 PM, Yevheniy Demchenko wrote:
>>>>> We are experimenting with a new lustre-2.2.0 and surprisingly not
>>>>> getting any statfs performance improvements. It's generally as poor as in previous versions. Also, statahead does not seem to make any
>>>>> difference to statfs performance.
>>>>> 
>>>>> #strace -r ls -l
>>>>> ......
>>>>>   0.000041 lstat("77193", {st_mode=S_IFREG|0644, st_size=4096, ...}) = 0
>>>>>   0.000238 lgetxattr("77193", "security.selinux", 0x129d800, 255) =
>>>>> -1 ENODATA (No data available)
>>>> 
>>>> This is selinux trying to fetch xattrs.  A plague on selinux, IMHO, but we can't get rid of it.  The prefetching and caching of existing and negative  (i.e. non-existent) xattrs on the client is being discussed in http://bugs.whamcloud.com/browse/LU-549.
>>> 
>>> Thanks for reply, things are starting to be clearer. It seems,
>>> that problem is not in selinux itself, but in not-caching xattrs
>>> in lustre in general. if i understand correctly, statahead tries
>>> to read ahead file attributes and put them in cache, so that client
>>> would not contact MDS for getting them. But if we don't cache
>>> xattrs, every tool like ls or rsync will contact mds for every
>>> file as it tries to get acls and selinux attributes, which can not
>>> be cached. Wouldn't it be better to catch the case when FS is
>>> mounted with -o noacl and selinux is disabled on the client side
>>> and immediately return EOPNOTSUPP without contacting MDS? We would
>>> be happy to trade acls and xattrs for a better statfs performance.
>> 
>> Yes, that makes sense.  It would probably be a simple patch in ll_getxattr_common() if you wanted to give it a try.
> 
> Some new results and patch:
> 1. run (immediately after fs mount):
> #time ls -l
> real    0m33.170s
> user    0m0.945s
> sys     0m15.861s
> 2. and subsequent runs:
> #time ls -l
> real    0m12.134s
> user    0m0.819s
> sys     0m10.491s

Saving 10s for 100k files is quite respectable for such a simple patch.
Could you please submit it to Gerrit for inclusion:

http://wiki.whamcloud.com/display/PUB/Submitting+Changes
http://wiki.whamcloud.com/display/PUB/Using+Gerrit

You can use LU-549 for this change.

> --- ./lustre-2.2.50-master/lustre/llite/xattr.c 2012-03-16
> 08:20:59.000000000 +0100
> +++ ./lustre-2.2.50/lustre/llite/xattr.c  2012-04-06 21:48:19.642232233
> +0200
> @@ -40,6 +40,7 @@
> #include <linux/sched.h>
> #include <linux/mm.h>
> #include <linux/smp_lock.h>
> +#include <linux/selinux.h>
> 
> #define DEBUG_SUBSYSTEM S_LLITE
> 
> @@ -94,7 +95,8 @@
>              xattr_type == XATTR_ACL_DEFAULT_T) &&
>            !(sbi->ll_flags & LL_SBI_ACL))
>                 return -EOPNOTSUPP;
> -
> +       if (xattr_type == XATTR_SECURITY_T && !selinux_is_enabled())
> +               return -ENODATA;
>         if (xattr_type == XATTR_USER_T && !(sbi->ll_flags &
> LL_SBI_USER_XATTR))
>                 return -EOPNOTSUPP;
>         if (xattr_type == XATTR_TRUSTED_T &&
> !cfs_capable(CFS_CAP_SYS_ADMIN))
> 
> Indeed, selinux doesn't do any good for lustre statfs. It seems that acl
> part is already filtered out in xattr.c.
> I'll try to find out where lustre client spends 12 "sys" seconds in
> subsequent runs.



Cheers, Andreas
--
Andreas Dilger                       Whamcloud, Inc.
Principal Lustre Engineer            http://www.whamcloud.com/