[Lustre-discuss] exceedingly slow lstats

[Robin Humble]

On Fri, Jan 20, 2012 at 02:35:19PM -0800, John White wrote:
>Well, I was reading the strace wrong anyway:
>lstat("../403/a323", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.134326>
>getxattr("../403/a323", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported) <0.000018>
>lstat("../403/a330", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.158898>
>getxattr("../403/a330", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported) <0.000019>
>lstat("../403/a331", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.239466>
>getxattr("../403/a331", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported) <0.000012>
>lstat("../403/a332", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.130146>
>getxattr("../403/a332", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported) <0.000012>
>
>The getxattr takes an incredibly short amount of time, it's the lstat itself that's taking 0.1+s.  

it used to be that weird slowdowns and high load could be caused by
kernel zone_reclaim confusion, so firstly I'd suggest checking that
vm.zone_reclaim_mode=0 everywhere (clients and servers).

after that see if turning off read & write_through caches on OSS's
helps metadata rates. there's a fair chance that streaming i/o to OSS's
is filling OSS ram and pushing inodes/dentries out of OSS vfs cache
causing big metadata slowdowns - the more streaming i/o the greater the
slowdown.

if turning off the data caches fixes the problem for you (ie. it's not
faulty hardware or an old lustre version or something else) then there
are couple of different methods that could let you get both data
caching and good metadata rates, but first things first...

cheers,
robin
--
Dr Robin Humble, HPC Systems Analyst, NCI National Facility