[Lustre-discuss] lustre 2.2.0 statfs performance

[Yevheniy Demchenko]

On 04/06/2012 07:43 PM, Andreas Dilger wrote:
> On 2012-04-06, at 10:33 AM, Yevheniy Demchenko wrote:
>> On 04/06/2012 06:40 AM, Andreas Dilger wrote:
>>> On 2012-04-05, at 2:40 PM, Yevheniy Demchenko wrote:
>>>> We are experimenting with a new lustre-2.2.0 and surprisingly not
>>>> getting any statfs performance improvements. It's generally as poor as in previous versions. Also, statahead does not seem to make any
>>>> difference to statfs performance.
>>>>
>>>> #strace -r ls -l
>>>> ......
>>>>    0.000041 lstat("77193", {st_mode=S_IFREG|0644, st_size=4096, ...}) = 0
>>>>    0.000238 lgetxattr("77193", "security.selinux", 0x129d800, 255) =
>>>> -1 ENODATA (No data available)
>>> This is selinux trying to fetch xattrs.  A plague on selinux, IMHO, but we can't get rid of it.  The prefetching and caching of existing and negative  (i.e. non-existent) xattrs on the client is being discussed in http://bugs.whamcloud.com/browse/LU-549.
>>>
>> Thanks for reply, things are starting to be clearer. It seems, that
>> problem is not in selinux itself, but in not-caching xattrs in lustre in
>> general. if i understand correctly, statahead tries to read ahead file
>> attributes and put them in cache, so that client would not contact mds
>> for getting them. But if we don't cache xattrs, every tool like ls or
>> rsync will contact mds for every file as it tries to get acls and
>> selinux attributes, which can not be cached. Wouldn't it be better to
>> catch the case when FS is mounted with -o noacl and selinux is disabled
>> on the client side and immediately return EOPNOTSUPP without contacting
>> mds? We would be happy to trade acls and xattrs for a better statfs
>> performance.
> Yes, that makes sense.  It would probably be a simple patch in ll_getxattr_common() if you wanted to give it a try.
>
> Cheers, Andreas
> --
> Andreas Dilger                       Whamcloud, Inc.
> Principal Lustre Engineer            http://www.whamcloud.com/
Some new results and patch:
1. run (immediately after fs mount):
#time ls -l
real    0m33.170s
user    0m0.945s
sys     0m15.861s
2. and subsequent runs:
#time ls -l
real    0m12.134s
user    0m0.819s
sys     0m10.491s

--- ./lustre-2.2.50-master/lustre/llite/xattr.c 2012-03-16
08:20:59.000000000 +0100
+++ ./lustre-2.2.50/lustre/llite/xattr.c  2012-04-06 21:48:19.642232233
+0200
@@ -40,6 +40,7 @@
 #include <linux/sched.h>
 #include <linux/mm.h>
 #include <linux/smp_lock.h>
+#include <linux/selinux.h>
 
 #define DEBUG_SUBSYSTEM S_LLITE
 
@@ -94,7 +95,8 @@
              xattr_type == XATTR_ACL_DEFAULT_T) &&
            !(sbi->ll_flags & LL_SBI_ACL))
                 return -EOPNOTSUPP;
-
+       if (xattr_type == XATTR_SECURITY_T && !selinux_is_enabled())
+               return -ENODATA;
         if (xattr_type == XATTR_USER_T && !(sbi->ll_flags &
LL_SBI_USER_XATTR))
                 return -EOPNOTSUPP;
         if (xattr_type == XATTR_TRUSTED_T &&
!cfs_capable(CFS_CAP_SYS_ADMIN))

Indeed, selinux doesn't do any good for lustre statfs. It seems that acl
part is already filtered out in xattr.c.
I'll try to find out where lustre client spends 12 "sys" seconds in
subsequent runs.

Ing. Yevheniy Demchenko
Senior Linux Administrator
UVT s.r.o.