[Lustre-discuss] lustre 2.2.0 statfs performance

[Andreas Dilger]

On 2012-04-05, at 2:40 PM, Yevheniy Demchenko wrote:
> We are experimenting with a new lustre-2.2.0 and surprisingly not
> getting any statfs performance improvements. It's generally as poor as
> in previous versions. Also, statahead does not seem to make any
> difference to statfs performance.
> 
> Simple ls -l test for a 100000 files directory was done on the node23
> with statahead on and off:

> -----statahead off-----
> #time ls -l
> real    0m52.751s
> user    0m1.153s
> sys     0m20.101s
> 
> #time ls -l
> real    0m21.280s
> user    0m1.086s
> sys     0m11.973s
> All subsequent runs complete in virtually the same time (21-24s).
> 
> -----statahead on-----
> #time ls -l
> real    0m43.846s
> user    0m1.242s
> sys     0m20.444s
> 
> #time ls -l
> real    0m24.000s
> user    0m1.104s
> sys     0m14.125s
> All subsequent runs complete in virtually the same time (21-24s).

It looks to me like a 25% improvement was seen - 53s to 44s for first run.  After that, the directories/inodes/locks/attributes are all in the client cache, so statahead is no longer active (no RPCs need to be sent).

> #strace -r ls -l
> ......
>     0.000041 lstat("77193", {st_mode=S_IFREG|0644, st_size=4096, ...}) = 0
>     0.000238 lgetxattr("77193", "security.selinux", 0x129d800, 255) =
> -1 ENODATA (No data available)

This is selinux trying to fetch xattrs.  A plague on selinux, IMHO, but we can't get rid of it.  The prefetching and caching of existing and negative  (i.e. non-existent) xattrs on the client is being discussed in http://bugs.whamcloud.com/browse/LU-549.

Cheers, Andreas
--
Andreas Dilger                       Whamcloud, Inc.
Principal Lustre Engineer            http://www.whamcloud.com/