[Lustre-discuss] Client directory entry caching

[Daire Byrne]

On Thu, Jul 29, 2010 at 8:54 PM, Andreas Dilger
<andreas.dilger at oracle.com> wrote:
> On 2010-07-29, at 04:47, Daire Byrne wrote:
>> I was wondering if it is possible to have the client completely cache
>> a recursive listing of a lustre filesystem such that on a second run
>> it doesn't have to talk to the MDT again? Taking the simplest case
>> where I only have one client that is browsing a million file tree
>> (say), I would expect that once the ldlm has cached the locks
>> (lru_size) then a second recursive scan (find, ls -R) shouldn't need
>> to talk to the MDT/OST again. But this is not the case probably
>> because a recursive scan needs to do a open() and getdents() on each
>> directory it finds.
>
> The getdents() calls can be returned from the client-side cache, it is only the open() that needs to go to the MDS.  Lustre actually does support client-side open cache, but it is currently only used by NFS servers (which, sadly, opens and closes the file for every single write operation on a file).

Ah yes... that makes sense. I recall the opencache gave a big boost in
performance for NFS exporting but I wasn't sure if it had become the
default. I haven't been keeping up to date with Lustre developments.

> I know Oleg has at times discussed enabling the open cache on the client for regular filesystem access, but I don't know the tweak needed for this offhand.  I know in the past we didn't do this because there was extra DLM locking overhead for cancelling the open lock, but with the DLM lock cancel batching that may not be as big a performance hit.
>
> It wouldn't be a bad idea to start with a /proc tuneable or "-o openlock" mount option that selectively allows open cache per client mount, so that performance testing can be done.  After that we can decide whether this is only good for specific workloads and bad for others, or if it is an improvement for most workloads and should be enabled by default.

So just as a quick and dirty check of the opencache I did a "find" on
a remote client directly using the lustre client and also through an
NFS gateway client (in this case running on the MDS).

  find /mnt/lustre (not cached) = 51 secs
  find /mnt/lustre (cached = 22 secs
  find /mnt/nfs (not cached) = 127 secs
  find /mnt/nfs (cached) = 15 secs.

So even with the metadata going over NFS the opencache in the client
seems to make quite a difference (I'm not sure how much the NFS client
caches though). As expected I see no mdt activity for the NFS export
once cached. I think it would be really nice to be able to enable the
opencache on any lustre client. A couple of potential workloads that I
can think of that would benefit are WAN clients and clients that need
to do mainly metadata (e.g. scanning the filesystem, rsync --link-dest
hardlink snapshot backups). For the WAN case I'd be quite interested
to see what the overhead of the lock cancellation would be like for a
busy filesystem. I suppose we can already test that by doing an NFS
export. I don't suppose you know if CEA's "ganesha" userspace NFS
server has access to the opencache? It can cache data to disk too
which is also good for WAN applications.

>> If I just stat all the files without doing a recursive scan then it gets everything from the client cache as expected without the MDS chatter - e.g.
>>
>>  find /mnt/lustre -type f > /tmp/files.txt
>>  cat /tmp/files.txt | xargs ls -l
>>
>> Is there any way to improve the browsing speed and cache directory
>> contents - especially for the case where I only have a single client
>> accessing an entire tree? As an aside I also noticed that a "ls -l"
>> does a getxattr - does that get cached by the client too? I can
>> imagine it might cause quite a bit of MDS chatter.
>
> So far, Lustre doesn't cache any xattr on the client beyond the file layout ("lustre.lov" xattr), but it is something I've been thinking about.  The security.capability attribute is special-cased in the 1.8.4 client to not return any data, and beyond that there aren't any attributes that I'm aware of that are widely used, so I don't think there is a pressing demand for this, but if a case can be made for this we'll definitely look at it more seriously.

Yea I doubt it makes much difference I just noticed that "ls -l" did
it and was wondering what Lustre made of it.

Thanks for the insightful reply as usual Andreas,

Daire