[Lustre-devel] please ignore previous mail, it was sent accidentially Re: subtree locks and path re-validation avoidance

[Vladimir V. Saveliev]

Oops. Sorry, I did not complete the mail



On Thu, 2008-02-28 at 21:02 +0200, Vladimir V. Saveliev wrote:
> Hello
> 
> On Fri, 2008-02-22 at 20:15 -0700, Peter J Braam wrote:
> > I'd like to make a suggestion to perhaps immediately find the right 
> > primitives for getcwd to return a reasonably correct pathname in 
> > Lustre.  I believe this is the simplest case where I have seen pathname 
> > revalidation being important.  In the context of that example the 
> > subtree lock discussion might gain more clarity.
> 
> Can we have something like th
> 
> When a client cancels a lock on an object on CWD, server sets NULL mode
> lock for the object. That NULL mode lock indicates that the object is 
> 
> > 
> > I would also like to note that I had a discussion with Linus at one of 
> > the kernel workshops in Ottawa maybe almost 4-5 years ago.  First Linus 
> > attacked the idea of using file identifiers - he suggested that doing 
> > everything with pathnames was better (which is what InterMezzo did).   
> > When we explained to him that this requires locking all parents he began 
> > to see the problems we had with this and understood the locking at the 
> > fid/name level that we use in Lustre.  I found little resistance when I 
> > mentioned to him that for this model the VFS does not have a correct 
> > implementation of getcwd, unless the dcache is kept current.
> > 
> > UCSC has received funding from the National Labs and now been turned 
> > into a peta-scale I/O institute I believe did more results on file 
> > systems implemented with pathnames.  Some things are beautiful and easy 
> > with pathnames, but others get really ugly, and so far I don't see this 
> > displacing fid ideas that govern NFS, AFS and Lustre.
> > 
> > - Peter -
> > 
> > Alex Zhuravlev wrote:
> > > Hi,
> > >
> > > couple comments inline ...
> > >
> > > Vladimir V. Saveliev wrote:
> > >   
> > >> The example shows the details:
> > >>
> > >> 1. A client C1 holds ordinary lock on an object O1 (it did
> > >> chdir(/a/b/c/d/e), O1 is inode of /a/b/c/d/e). C1 is idle now.
> > >>     
> > >
> > > chdir doesn't return any lock. should it?
> > >
> > >   
> > >> 2. Another client C2 does ls -ld /a/b/c/d/e, MD server sends a BAST to
> > >> C1 and C1 cancels the lock of O1.
> > >>
> > >> 3. C2 is not interested anymore in O1, so it drops the lock. 
> > >>
> > >> 4. Yet another client C3 acquires subtree lock on /a/b and caches and
> > >> possibly changes (if under WBC) objects under /a/b including /a/b/c/d/e
> > >> (the object O1). The key issue is that MDS neither remembers about O1 on
> > >> C1 nor keeps information about objects cached by a client under a
> > >> subtree lock.
> > >>
> > >> 5. Now C1 continues with stat(``.''). It sees that the lock on O1 is
> > >> canceled, so it goes to MD server and acquires the lock on O1.
> > >>
> > >> Now we have:
> > >> uptodate O1 is on C3;
> > >> MDS has a request for O1 from C1 and MDS can not easily deterimine
> > >> whether O1 is under any subtree lock. In order to find whether the lock
> > >> conflict exists we need to have a special procedure. It is referred to
> > >> as path re-validation.
> > >>
> > >> The main thing to be done on path re-validation is to look for above
> > >> subtree lock. While it is probably doable, the path re-validation is not
> > >> going to be very efficient (especially in case of CMD). I can provide
> > >> more details if necessary.
> > >>
> > >>
> > >> However, it looks like it is possible to avoid having to do path
> > >> re-validation completely.
> > >>
> > >>
> > >> The problem appears when clients request locks on objects directly,
> > >> without doing downward lookup through a directory structure.
> > >> This happens, for example, when clients access directly components of
> > >> current working directories (CWDs).
> > >> If a client cancels locks on such objects (either due to a BAST or
> > >> voluntary) - it has to go through the path re-validation later.
> > >>
> > >> Objects to which a client may access directly appear in result of normal
> > >> downward lookup. Therefore, they were locked, and their locks can be
> > >> canceled. That is the point where we can take care about future accesses
> > >> without re-validation.
> > >> On canceling a lock of directly accessible object we have to inform DLM
> > >> that the ordinary locking has to be used for that object. That will
> > >> prevent the object from getting cached under a subtree lock.
> > >>     
> > >
> > > 1) there may be thousands of such objects (many processes on many nodes)
> > > 2) it's not clear when to enable this back
> > >
> > >   
> > >> The problem with this schema is to determine which objects are directly
> > >> accessible. But wouldn't solving it be worth doing given that it may
> > >> help to avoid path re-validation deal.
> > >>
> > >> Any comments are welcome.
> > >>     
> > >
> > > thanks, Alex
> > > _______________________________________________
> > > Lustre-devel mailing list
> > > Lustre-devel at lists.lustre.org
> > > http://lists.lustre.org/mailman/listinfo/lustre-devel
> > >