[Lustre-discuss] very slow directory access (lstat taking >20s to return)

[Frederik Ferner]

Thanks both for your replies.

In the mean time we've upgraded our file system to 1.8.3-ddn3.3, which 
kept us busy, so sorry for the late reply. In our initial testing we've 
not seen this problem but more testing is in progress.

Andreas Dilger wrote:
> On 2010-10-29, at 21:20, Daniel Kobras wrote:
>> On Fri, Oct 29, 2010 at 09:40:33AM +0100, Frederik Ferner wrote:
>>> Doing a 'strace -T -e file ls -n' on one directory with about 750
>>> files, while users were seeing the hanging ls, showed lstat calls
>>> taking seconds, up to 23s.
>> The (l)stat() calls determine the exact size of all files in the
>> displayed directory. This means that each OSTs needs to revoke
>> client write locks for all these files, ie. client-side write
>> caches for all files in the directory are flushed before the
>> (l)stat() returns. This can easily take several seconds if there is
>> heavy write activity on the file.

As far as I was able to see at that time, the actual files were not 
written to anymore but additional files in the directory or one 
directory below were still created.

> Actually, unlike most other cluster filesystems Lustre does not need
> to revoke the OST write locks in order to determine the file size.
> The OST extent locks are conditionally revoked if the client is no
> longer using them, but if they are in use the clients holding those
> locks only return a "glimpse" of the current file size to the OST,
> which in turn returns the size to the client doing the (l)stat()
> call.

I assume on a busy OST this could still take a while, though?

We'll monitor the situation after the upgrade and if it happens again, 
we'll try more debugging again.

Thanks,
Frederik

-- 
Frederik Ferner
Computer Systems Administrator		phone: +44 1235 77 8624
Diamond Light Source Ltd.		mob:   +44 7917 08 5110
(Apologies in advance for the lines below. Some bits are a legal
requirement and I have no control over them.)