[Lustre-discuss] exceedingly slow lstats

[John White]

Hello,
	We've started getting complaints from users about exceedingly slow lstats on directories of any size.  The back-end storage is not overloaded (MDT is all but completely unloaded, the OSTs aren't doing much beyond 500MB/s (aggregate, DDN 9900s).  The load on the OSSs is also inexplicably high (consistently above 50).  Strace is showing the majority of time being spent on failing getxattrs.  We've alleviated much of the impact of this slowness by unaliasing color ls's, but that's merely a work-around.  Any ideas at first glance?

     0.001222 lstat("/global/scratch/jwhite/backuptest/highcount/3/a2", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
     0.664400 getxattr("/global/scratch/jwhite/backuptest/highcount/3/a2", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
     0.000040 lstat("/global/scratch/jwhite/backuptest/highcount/3/a5", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
     0.916908 getxattr("/global/scratch/jwhite/backuptest/highcount/3/a5", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
     0.000040 lstat("/global/scratch/jwhite/backuptest/highcount/3/a10", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
     0.387650 getxattr("/global/scratch/jwhite/backuptest/highcount/3/a10", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
     0.000040 lstat("/global/scratch/jwhite/backuptest/highcount/3/a11", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
     0.376490 getxattr("/global/scratch/jwhite/backuptest/highcount/3/a11", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
     0.000049 lstat("/global/scratch/jwhite/backuptest/highcount/3/a22", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
     1.134287 getxattr("/global/scratch/jwhite/backuptest/highcount/3/a22", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
     0.000045 lstat("/global/scratch/jwhite/backuptest/highcount/3/a28", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
     0.540547 getxattr("/global/scratch/jwhite/backuptest/highcount/3/a28", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
     0.000046 lstat("/global/scratch/jwhite/backuptest/highcount/3/a29", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
     0.572334 getxattr("/global/scratch/jwhite/backuptest/highcount/3/a29", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
     0.000040 lstat("/global/scratch/jwhite/backuptest/highcount/3/a36", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
     0.392290 getxattr("/global/scratch/jwhite/backuptest/highcount/3/a36", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
----------------
John White
HPC Systems Engineer
(510) 486-7307
One Cyclotron Rd, MS: 50C-3396
Lawrence Berkeley National Lab
Berkeley, CA 94720