From shay at opensfs.org Mon Jan 9 23:05:28 2012 From: shay at opensfs.org (Shay Seager) Date: Mon, 9 Jan 2012 15:05:28 -0800 Subject: [Lustre-devel] Call for Participation - LUG 2012 Message-ID: <5A25D756-1FC4-421B-80B4-EF72366136ED@opensfs.org> Request for Participation The LUG program committee would like to invite members of the Lustre community to submit presentation abstracts for inclusion in this year's meeting. It is not necessary to submit a technical paper, just an abstract of your proposed talk no more than a page in length. Talks should target a half an hour in length and reflect the work that you or your site are doing with Lustre. The deadline to submit presentation abstracts is February 14, 2012. For questions or to submit a presentation abstract, send mail to submissions at lists.opensfs.org or contact the program committee chair, Stephen Simms (ssimms at indiana.edu). Thank you, Shay - - - Shay Seager Open SFS Secretary www.opensfs.org (925) 290-7641 shay at opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From shay at opensfs.org Mon Jan 9 23:48:18 2012 From: shay at opensfs.org (Shay Seager) Date: Mon, 9 Jan 2012 15:48:18 -0800 Subject: [Lustre-devel] LUG 2012 Registration is open Message-ID: Hello! Happy New Year! We here at OpenSFS hope that you have had a wonderful start to your 2011. OpenSFS would like to invite you to enjoy spring in beautiful Austin Tx - LUG 2012 will be held at the Hyatt Regency Austin, Downtown on the Lake, Monday, April 23rd 2012 through Wednesday, April 25th 2012. This 2 1/2 day event is the primary venue for discussion and seminars on open source parallel file system technologies with a unique focus on the Lustre™ parallel file system. To Register please follow this link http://www.opensfs.org/lug/registration LUG is a user-led event whose organizing committee is made up of representatives from Indiana University (IU), Naval Research Laboratory (NRL), Oak Ridge National Laboratory (ORNL), Sandia National Laboratories (SNL), Lawrence Livermore National Laboratory (LLNL), and Texas Advanced Computing Center (TACC). Feel free to contact me with any questions. Best, Shay - - - Shay Seager Open SFS Secretary www.opensfs.org (925) 290-7641 shay at opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From di.wang at whamcloud.com Tue Jan 10 23:11:18 2012 From: di.wang at whamcloud.com (wangdi) Date: Tue, 10 Jan 2012 15:11:18 -0800 Subject: [Lustre-devel] FID update (reserve a bit in FID to identify it refers Metadata object or data object) Message-ID: <4F0CC596.4020409@whamcloud.com> Hello Here is an update about FID in Lustre. We will implement FID on OST in lustre 2.x, so both data and metadata objects will be referred by FID in lustre. But different with the FID on MDT, OST FID will not use OI to map the real inode.(OI is container for FID->inode mapping). Instead it will still store objects under /O structure (like the current 1.8 implementation), so they can have an internal reference from the namespace to avoid problems with fsck. Then 1. for metadata object, it will be stored under /root and referenced by OI, i.e. the target must lookup OI by FID to get the inode. 2. for data object, it will be stored under /O and referenced by the /O structure itself, i.e. the target can lookup /O by some kind of FID hash to get the inode. So the target(MDT or OST) needs to know it is meta_data object or data object, to figure out where to get the object by FID. We thought to reserve the highest bit of OID to identify whether it is a metadata or data object, so the maximum OID for each sequence will shrink from 4 billion to 2 billion. For FID related information, you can referhttp://wiki.lustre.org/index.php/Architecture_-_Interoperability_fids_zfs Thanks WangDi From yong.fan at whamcloud.com Sun Jan 22 16:11:38 2012 From: yong.fan at whamcloud.com (Fan Yong) Date: Mon, 23 Jan 2012 00:11:38 +0800 Subject: [Lustre-devel] Lustre dcache clean up and distributed locking In-Reply-To: <8A50205D-3704-4969-9997-3D6157C252D0@whamcloud.com> References: <8449503C-66B3-4CDC-89BF-5B52E6D29579@whamcloud.com> <5F3A45681A0F5944B3194CF8C8FB905B66FA4DD4F9@MX17A.corp.emc.com> <4F19A01B.4070405@whamcloud.com> <8A50205D-3704-4969-9997-3D6157C252D0@whamcloud.com> Message-ID: <4F1C353A.9080101@whamcloud.com> On 1/21/12 2:06 AM, Andreas Dilger wrote: > Hi all, please enjoy your New Years holiday. This issue can wait until your return. > > Fan Yong, > About DNE locking (I write this now only to avoid forgetting it, no immediate answer is needed) - in order to handle the split name/permission issue, the current plan is for the client to get the LOOKUP lock for the FID on both the master MDT and the remote MDT. Hi Andreas, Why two LOOKUP locks on two MDTs, why not one LOOKUP lock on inode holder MDT and the other name lock (new) on name entry holder MDT? Any special advantage? I think both methods need client-side support (for protocol changes). Cheers, Fan Yong > > If the link is removed from the master MDT, then the FID LOOKUP lock will be cancelled there, but if another hard link is remove from a different MDT then the only the LOOKUP from the other MDT needs to be cancelled. > > This will be similar to Tao's proposal for a separate lock bit for the name, in the DNE case where the name is remote from the inode. It still suffers from some inefficiency in case of multiple hard links from the same master MDT to a remote inode (canceling the FID LOOKUP lock when unlinking one name will force it to be refetched for the other links), but since hard links are so rare this should not significantly impact performance. > > Cheers, Andreas > > On 2012-01-20, at 10:10, Fan Yong wrote: >> Excellent work. I just added some comments inside your document. Please >> check. >> >> >> Best Regards, >> Fan Yong >> Whamcloud, Inc. >> >> On 1/20/12 9:34 AM, haiying.tang at emc.com wrote: >>> Hi Andreas, >>> >>> Tao is on vacation now. It might be difficult for him to check emails due to limited internet access at hometown. >>> For urgent issue, you folks could send email to his gmail account bergwolf at gmail.com. >>> >>> Thanks, >>> Haiying >>> >>> -----Original Message----- >>> From: Andreas Dilger [mailto:adilger at whamcloud.com] >>> Sent: 2012年1月20日 1:39 >>> To: Peng, Tao >>> Cc: faibish, sorin; Tang, Haiying; Liu, Xuezhao; laisiyao at whamcloud.com; yong.fan at whamcloud.com; green at whamcloud.com; eeb at whamcloud.com >>> Subject: Re: Lustre dcache clean up and distributed locking >>> >>> On 2012-01-17, at 3:21 AM, wrote: >>>> Thanks Siyao and Oleg for answering my dcache revalidation question on lustre mailing list. I updated the design to reflect it. >>>> >>>> Please see attachment. >>> Tao, >>> Fan Yong is also taking a more detailed look at your document and will >>> hopefully have a chance to reply before the New Year holidays. >>> >>> Also, we are just working on landing the patches to add support for Linux >>> 2.6.38 for the Lustre client. One of the patches relates to the lockless >>> dcache changes that were introduced in that kernel. If you are interested >>> to review this patch, and become more familiar with the Lustre development >>> process, you should visit http://review.whamcloud.com/1865 for the patch. >>> >>> You need to create an account in Gerrit using OpenID (Google, mostly), and >>> an account in our bug tracking system (http://jira.whamcloud.com) if you >>> haven't already. >>> >>>>> -----Original Message----- >>>>> From: Andreas Dilger [mailto:adilger at whamcloud.com] >>>>> Sent: Tuesday, January 17, 2012 4:16 PM >>>>> To: Peng, Tao >>>>> Cc: faibish, sorin; Tang, Haiying; Liu, Xuezhao; Lai Siyao; Fan Yong; Oleg Drokin; Eric Barton >>>>> Subject: Re: Lustre dcache clean up and distributed locking >>>>> >>>>> On 2012-01-16, at 9:25 PM, wrote: >>>>>> I finally started to work on Lustre dcache cleanup and locking. After reading Lustre, ocfs2 and VFS >>>>> dcache related code, I came to a design for cleaning up Lustre dcache code and doing distributed >>>>> locking under dcache. For distributed locking, the main idea is to add a new inode bitlock DENTRY lock >>>>> to just protect valid dentry, instead of letting LOOKUP lock handle multiple purpose, which makes >>>>> client unable to know whether file is deleted or not when server cancels LOOKUP lock. Instead, client >>>>> holds PR mode DENTRY lock on all valid denties and when server cancels it, client knows the file is >>>>> deleted. >>>>>> For details, please see the attachments (I attached both word and pdf versions as I am not sure if >>>>> which is more convenient for you:). And please help to review and comment. Thank you! >>>>> >>>>> Hi Tao, >>>>> I'm passing this on to the engineers that are most familiar with the DLM and client dcache code in >>>>> Lustre. After a quick first read through your document, your investigation of the client code is very >>>>> insightful and looks like it will be able to remove some of the significant complexity that has grown >>>>> over time in the llite code. >>>>> >>>>> Cheers, Andreas >>>>> -- >>>>> Andreas Dilger Whamcloud, Inc. >>>>> Principal Engineer http://www.whamcloud.com/ >>>>> >>>>> >>>> >>> Cheers, Andreas >>> -- >>> Andreas Dilger Whamcloud, Inc. >>> Principal Engineer http://www.whamcloud.com/ >>> >>> >>> >>> >>> >> From adilger at whamcloud.com Sun Jan 22 17:33:42 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Sun, 22 Jan 2012 10:33:42 -0700 Subject: [Lustre-devel] Lustre dcache clean up and distributed locking In-Reply-To: <4F1C353A.9080101@whamcloud.com> References: <8449503C-66B3-4CDC-89BF-5B52E6D29579@whamcloud.com> <5F3A45681A0F5944B3194CF8C8FB905B66FA4DD4F9@MX17A.corp.emc.com> <4F19A01B.4070405@whamcloud.com> <8A50205D-3704-4969-9997-3D6157C252D0@whamcloud.com> <4F1C353A.9080101@whamcloud.com> Message-ID: <81008C6C-8D65-4FBF-A3C0-B01D0683DE9A@whamcloud.com> On 2012-01-22, at 9:11, Fan Yong wrote: > On 1/21/12 2:06 AM, Andreas Dilger wrote: >> >> Fan Yong, >> About DNE locking - in order to handle the split name/permission issue, the current plan is for the client to get the LOOKUP lock for the FID on both the master MDT and the remote MDT. > Hi Andreas, > > Why two LOOKUP locks on two MDTs, why not one LOOKUP lock on inode holder MDT and the other name lock (new) on name entry holder MDT? Any special advantage? I think both methods need client-side support (for protocol changes). The difference between the two approaches is relatively small in terms of the functional outcome, but in terms of implementation there is a bigger difference. Two reasons for this: - we already have LOOKUP locks, so less code changes are needed - having a single lock per FID is easier to manage compared to one lock per name Consider that chmod/chgrp/chown/ACL operations are more common in practice than hard links, then having a single LOOKUP lock is more efficient. Even with hard links, if the client does a lookup on a link on the master, it can find the FID from the dirent and may have the inode data cached from the remote MDT. I agree that name locks may have advantages in some cases, but I don't know if that is worth the extra complexity. Cheers, Andreas >> If the link is removed from the master MDT, then the FID LOOKUP lock will be cancelled there, but if another hard link is remove from a different MDT then the only the LOOKUP from the other MDT needs to be cancelled. >> >> This will be similar to Tao's proposal for a separate lock bit for the name, in the DNE case where the name is remote from the inode. It still suffers from some inefficiency in case of multiple hard links from the same master MDT to a remote inode (canceling the FID LOOKUP lock when unlinking one name will force it to be refetched for the other links), but since hard links are so rare this should not significantly impact performance. >> >> Cheers, Andreas >> >> On 2012-01-20, at 10:10, Fan Yong wrote: >>> Excellent work. I just added some comments inside your document. Please >>> check. >>> >>> >>> Best Regards, >>> Fan Yong >>> Whamcloud, Inc. >>> >>> On 1/20/12 9:34 AM, haiying.tang at emc.com wrote: >>>> Hi Andreas, >>>> >>>> Tao is on vacation now. It might be difficult for him to check emails due to limited internet access at hometown. >>>> For urgent issue, you folks could send email to his gmail account bergwolf at gmail.com. >>>> >>>> Thanks, >>>> Haiying >>>> >>>> -----Original Message----- >>>> From: Andreas Dilger [mailto:adilger at whamcloud.com] >>>> Sent: 2012年1月20日 1:39 >>>> To: Peng, Tao >>>> Cc: faibish, sorin; Tang, Haiying; Liu, Xuezhao; laisiyao at whamcloud.com; yong.fan at whamcloud.com; green at whamcloud.com; eeb at whamcloud.com >>>> Subject: Re: Lustre dcache clean up and distributed locking >>>> >>>> On 2012-01-17, at 3:21 AM, wrote: >>>>> Thanks Siyao and Oleg for answering my dcache revalidation question on lustre mailing list. I updated the design to reflect it. >>>>> >>>>> Please see attachment. >>>> Tao, >>>> Fan Yong is also taking a more detailed look at your document and will >>>> hopefully have a chance to reply before the New Year holidays. >>>> >>>> Also, we are just working on landing the patches to add support for Linux >>>> 2.6.38 for the Lustre client. One of the patches relates to the lockless >>>> dcache changes that were introduced in that kernel. If you are interested >>>> to review this patch, and become more familiar with the Lustre development >>>> process, you should visit http://review.whamcloud.com/1865 for the patch. >>>> >>>> You need to create an account in Gerrit using OpenID (Google, mostly), and >>>> an account in our bug tracking system (http://jira.whamcloud.com) if you >>>> haven't already. >>>> >>>>>> -----Original Message----- >>>>>> From: Andreas Dilger [mailto:adilger at whamcloud.com] >>>>>> Sent: Tuesday, January 17, 2012 4:16 PM >>>>>> To: Peng, Tao >>>>>> Cc: faibish, sorin; Tang, Haiying; Liu, Xuezhao; Lai Siyao; Fan Yong; Oleg Drokin; Eric Barton >>>>>> Subject: Re: Lustre dcache clean up and distributed locking >>>>>> >>>>>> On 2012-01-16, at 9:25 PM, wrote: >>>>>>> I finally started to work on Lustre dcache cleanup and locking. After reading Lustre, ocfs2 and VFS >>>>>> dcache related code, I came to a design for cleaning up Lustre dcache code and doing distributed >>>>>> locking under dcache. For distributed locking, the main idea is to add a new inode bitlock DENTRY lock >>>>>> to just protect valid dentry, instead of letting LOOKUP lock handle multiple purpose, which makes >>>>>> client unable to know whether file is deleted or not when server cancels LOOKUP lock. Instead, client >>>>>> holds PR mode DENTRY lock on all valid denties and when server cancels it, client knows the file is >>>>>> deleted. >>>>>>> For details, please see the attachments (I attached both word and pdf versions as I am not sure if >>>>>> which is more convenient for you:). And please help to review and comment. Thank you! >>>>>> >>>>>> Hi Tao, >>>>>> I'm passing this on to the engineers that are most familiar with the DLM and client dcache code in >>>>>> Lustre. After a quick first read through your document, your investigation of the client code is very >>>>>> insightful and looks like it will be able to remove some of the significant complexity that has grown >>>>>> over time in the llite code. >>>>>> >>>>>> Cheers, Andreas >>>>>> -- >>>>>> Andreas Dilger Whamcloud, Inc. >>>>>> Principal Engineer http://www.whamcloud.com/ >>>>>> >>>>>> >>>>> >>>> Cheers, Andreas >>>> -- >>>> Andreas Dilger Whamcloud, Inc. >>>> Principal Engineer http://www.whamcloud.com/ >>>> >>>> >>>> >>>> >>>> >>> > From tao.peng at emc.com Mon Jan 30 07:44:52 2012 From: tao.peng at emc.com (tao.peng at emc.com) Date: Mon, 30 Jan 2012 02:44:52 -0500 Subject: [Lustre-devel] Lustre dcache clean up and distributed locking In-Reply-To: References: <28CD3A7C-52EE-4E75-B00F-4280D60CE17D@whamcloud.com> Message-ID: Hi Siyao, Yes, ll_lookup_lock should be able to be removed. And looks like it will be handled by Yang Sheng in ‘LU-506 kernel: FC15 - Dcache RCU-walk changes’ as commented by Fan Yong (http://review.whamcloud.com/#patch,sidebyside,1865,2,lustre/include/linux/lustre_patchless_compat.h). I will leave it as is to let Sheng handle it. Thanks, Tao From: Lai Siyao [mailto:laisiyao at whamcloud.com] Sent: Friday, January 20, 2012 10:56 AM To: Oleg Drokin Cc: Andreas Dilger; Peng, Tao; faibish, sorin; Tang, Haiying; Liu, Xuezhao; Fan Yong; Eric Barton Subject: Re: Lustre dcache clean up and distributed locking Hi Tao, The reason why ll_lookup_lock is introduced is because in some kernels (eg. 2.6.32 for RHEL6), d_rehash_cond() doesn't exist and only d_rehash can be used, which needs unlock dcache_lock before entry, therefore there is a race here, and a similar global lock like dcache_lock is needed. The improvement can be done here is when d_rehash_cond() is exported, ll_lookup_lock lock/unlock can be a null operation, and in recent kernels, dcache_lock is removed, ll_lookup_lock should be able to be removed too. I like most of the cleanups, but I don't see much benefits introducing a DENTRY bit lock, because a dentry without LOOKUP lock can hardly be used, and inodebit lock coverage upgrade/downgrade is more useful here. Bye, - Lai On Fri, Jan 20, 2012 at 1:11 AM, Oleg Drokin > wrote: Hello! On Jan 17, 2012, at 3:16 AM, Andreas Dilger wrote: > On 2012-01-16, at 9:25 PM, > wrote: >> I finally started to work on Lustre dcache cleanup and locking. After reading Lustre, ocfs2 and VFS dcache related code, I came to a design for cleaning up Lustre dcache code and doing distributed locking under dcache. For distributed locking, the main idea is to add a new inode bitlock DENTRY lock to just protect valid dentry, instead of letting LOOKUP lock handle multiple purpose, which makes client unable to know whether file is deleted or not when server cancels LOOKUP lock. Instead, client holds PR mode DENTRY lock on all valid denties and when server cancels it, client knows the file is deleted. >> >> For details, please see the attachments (I attached both word and pdf versions as I am not sure if which is more convenient for you:). And please help to review and comment. Thank you! Thanks for looking into this. It is indeed possible to add another bit for just the dentry, but the problem is it would need to be granted in conjunction with the LOOKUP bit anyway (along with permission information), if we don't do this, then we would need to send an additional RPC to get this information very shortly after lookup in a lot of times and that would be pretty slow. Of course there is no way to get different lock mode for every bit, so in this scheme the lock mode swould really be the same. And the (currently) big problem with multiple bit locks is we don't have a way to cancel just subset of a lock (or degrade a lock coverage I should say). This was planned for quite a while ago, but somehow never ended up at the top of the priority list so was never implemented after all. There are minor comments on the logic flow after that too. E.g. the locks in lustre are unlocked right away on the client side, but just placed in the LRU and remain on the client in this sort of unreferenced state. When there is a conflict, the server sends the AST RPC and then the lock is actually released (cancelled) by client. Currently the AST is not delivered to a lock that has non-zero usage counters. Additionally due to the way we implement locks, it's not possible to have a separate lock for every name of the file. For your cleanup list, I think that is good. Note that item 'c' was already implemented some time ago, you can see it here: https://bugzilla.lustre.org/show_bug.cgi?id=16417 the patch is this: https://bugzilla.lustre.org/attachment.cgi?id=24200 It's probably somewhat stale now. It did not work correctly back at the time due to some other logic, but should work now. You can read the comments in that bug for some more info. Bye, Oleg -- Oleg Drokin Senior Software Engineer Whamcloud, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tao.peng at emc.com Mon Jan 30 07:46:14 2012 From: tao.peng at emc.com (tao.peng at emc.com) Date: Mon, 30 Jan 2012 02:46:14 -0500 Subject: [Lustre-devel] Lustre dcache clean up and distributed locking In-Reply-To: <4F19A01B.4070405@whamcloud.com> References: <8449503C-66B3-4CDC-89BF-5B52E6D29579@whamcloud.com> <5F3A45681A0F5944B3194CF8C8FB905B66FA4DD4F9@MX17A.corp.emc.com> <4F19A01B.4070405@whamcloud.com> Message-ID: Hi Fan Yong, Sorry for the late reply. I got your email earlier but didn't have chance to read the document due to very limited internet access. Thank you very much for taking detailed look at the document and giving so many valuable comments. About adding new inode bitlock, I must admit that I missed hardlink case. Although the lock mode can be changed to be the same as LOOKUP lock, I agree that inode bitlock cannot sufficiently protect every name/dentry separately for hardlinked files. And I agree, as you suggested in the document, adding a name entry based lock can solve it. IIUC, to add a name entry based lock, both client and server should be modified, and LDLM need to be changed to manage the new lock information. Like new inode bitlock, the name entry based lock will as well introduce interoperability issues as a result of on wire protocol change. And in DNE environment as Andreas explained, the original plan is to get LOOKUP on both master MDT and remote MDT. If we introduce name entry based lock, DNE case may need some change as well. I will take a closer look at the approach. In the meanwhile, could you please confirm if it is the right way to go? The benefit of above complexity seems to be better distributed dentry locking under dcache framework. What do others think? Andreas? Thanks and Best Regards, Tao > -----Original Message----- > From: Fan Yong [mailto:yong.fan at whamcloud.com] > Sent: Saturday, January 21, 2012 1:11 AM > To: Peng, Tao; bergwolf at gmail.com > Cc: Tang, Haiying; adilger at whamcloud.com; faibish, sorin; Liu, Xuezhao; laisiyao at whamcloud.com; > green at whamcloud.com; eeb at whamcloud.com; Bryon Neitzel > Subject: Re: Lustre dcache clean up and distributed locking > > Hi Tao, > > Excellent work. I just added some comments inside your document. Please check. > > > Best Regards, > Fan Yong > Whamcloud, Inc. > > On 1/20/12 9:34 AM, haiying.tang at emc.com wrote: > > Hi Andreas, > > > > Tao is on vacation now. It might be difficult for him to check emails due to limited internet access > at hometown. > > For urgent issue, you folks could send email to his gmail account bergwolf at gmail.com. > > > > Thanks, > > Haiying > > > > -----Original Message----- > > From: Andreas Dilger [mailto:adilger at whamcloud.com] > > Sent: 2012年1月20日 1:39 > > To: Peng, Tao > > Cc: faibish, sorin; Tang, Haiying; Liu, Xuezhao; > > laisiyao at whamcloud.com; yong.fan at whamcloud.com; green at whamcloud.com; > > eeb at whamcloud.com > > Subject: Re: Lustre dcache clean up and distributed locking > > > > On 2012-01-17, at 3:21 AM, wrote: > >> Thanks Siyao and Oleg for answering my dcache revalidation question on lustre mailing list. I > updated the design to reflect it. > >> > >> Please see attachment. > > Tao, > > Fan Yong is also taking a more detailed look at your document and will > > hopefully have a chance to reply before the New Year holidays. > > > > Also, we are just working on landing the patches to add support for > > Linux > > 2.6.38 for the Lustre client. One of the patches relates to the > > lockless dcache changes that were introduced in that kernel. If you > > are interested to review this patch, and become more familiar with the > > Lustre development process, you should visit http://review.whamcloud.com/1865 for the patch. > > > > You need to create an account in Gerrit using OpenID (Google, mostly), > > and an account in our bug tracking system (http://jira.whamcloud.com) > > if you haven't already. > > > >>> -----Original Message----- > >>> From: Andreas Dilger [mailto:adilger at whamcloud.com] > >>> Sent: Tuesday, January 17, 2012 4:16 PM > >>> To: Peng, Tao > >>> Cc: faibish, sorin; Tang, Haiying; Liu, Xuezhao; Lai Siyao; Fan > >>> Yong; Oleg Drokin; Eric Barton > >>> Subject: Re: Lustre dcache clean up and distributed locking > >>> > >>> On 2012-01-16, at 9:25 PM, wrote: > >>>> I finally started to work on Lustre dcache cleanup and locking. > >>>> After reading Lustre, ocfs2 and VFS > >>> dcache related code, I came to a design for cleaning up Lustre > >>> dcache code and doing distributed locking under dcache. For > >>> distributed locking, the main idea is to add a new inode bitlock > >>> DENTRY lock to just protect valid dentry, instead of letting LOOKUP > >>> lock handle multiple purpose, which makes client unable to know > >>> whether file is deleted or not when server cancels LOOKUP lock. Instead, client holds PR mode > DENTRY lock on all valid denties and when server cancels it, client knows the file is deleted. > >>>> For details, please see the attachments (I attached both word and > >>>> pdf versions as I am not sure if > >>> which is more convenient for you:). And please help to review and comment. Thank you! > >>> > >>> Hi Tao, > >>> I'm passing this on to the engineers that are most familiar with the > >>> DLM and client dcache code in Lustre. After a quick first read > >>> through your document, your investigation of the client code is very > >>> insightful and looks like it will be able to remove some of the significant complexity that has > grown over time in the llite code. > >>> > >>> Cheers, Andreas > >>> -- > >>> Andreas Dilger Whamcloud, Inc. > >>> Principal Engineer http://www.whamcloud.com/ > >>> > >>> > >> > > > > Cheers, Andreas > > -- > > Andreas Dilger Whamcloud, Inc. > > Principal Engineer http://www.whamcloud.com/ > > > > > > > > > > From adilger at whamcloud.com Mon Jan 30 23:27:52 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Mon, 30 Jan 2012 16:27:52 -0700 Subject: [Lustre-devel] Lustre dcache clean up and distributed locking In-Reply-To: References: <8449503C-66B3-4CDC-89BF-5B52E6D29579@whamcloud.com> <5F3A45681A0F5944B3194CF8C8FB905B66FA4DD4F9@MX17A.corp.emc.com> <4F19A01B.4070405@whamcloud.com> Message-ID: On 2012-01-30, at 12:46 AM, wrote: > Sorry for the late reply. I got your email earlier but didn't have chance to read the document due to very limited internet access. > > Thank you very much for taking detailed look at the document and giving so many valuable comments. > > About adding new inode bitlock, I must admit that I missed hardlink case. Although the lock mode can be changed to be the same as LOOKUP lock, I agree that inode bitlock cannot sufficiently protect every name/dentry separately for hardlinked files. And I agree, as you suggested in the document, adding a name entry based lock can solve it. > > IIUC, to add a name entry based lock, both client and server should be modified, and LDLM need to be changed to manage the new lock information. Like new inode bitlock, the name entry based lock will as well introduce interoperability issues as a result of on wire protocol change. And in DNE environment as Andreas explained, the original plan is to get LOOKUP on both master MDT and remote MDT. If we introduce name entry based lock, DNE case may need some change as well. > > I will take a closer look at the approach. In the meanwhile, could you please confirm if it is the right way to go? The benefit of above complexity seems to be better distributed dentry locking under dcache framework. > > What do others think? Andreas? Tao, before we go down the road of implementing such a change, there would have to be clear benefits to the performance and/or kernel integration before we could accept the interoperability issues and added code complexity. The benefits of having a per-name lock, as I see them, are: - can be cancelled separately from other name locks - can be changed independently from the inode permissions (LOOKUP) - potential ease of integration with the Linux dcache The drawbacks of introducing a per-name lock: - only useful for hard-linked files (uncommon case) - added code complexity for a new lock type - incompatibility with existing client/server - additional code to handle incompatibility transparently - lock enqueue may need extra RPC for each name lock (different resource) - lock cancellation may need an extra RPC for each resource The way I see it is that the DENTRY lock would need to be a different resource for each name on a file. The only way this is going to be useful AFAICS is if the client enqueues each DENTRY lock on a separate resource, so that it can be granted and cancelled independently of the LOOKUP lock (which is on the FID and would be common for all names that reference this inode). That would either need an extra RPC for every filename lookup (to enqueue both the DENTRY and LOOKUP locks) or a new LDLM_ENQUEUE RPC that can request and be granted two different locks (FID [seq, oid, ver, 0], and FID+name hash [seq, oid, ver, hash]) in a single RPC. So, the main question is how much better the client dcache/VFS integration will be with the DCACHE lock compared to the existing mechanism with the LOOKUP (FID) lock? Is it impossible (or very difficult) to use the existing mechanism for the new lockless dcache code? Or is it a matter of this being a somewhat cleaner implementation, but not critical for moving forward? I'm aware that our dcache code has been too complex for some time, but I haven't looked recently to see how much of this is due to using/supporting old kernel APIs, and how much of it is due to the use of the LOOKUP lock on the FID protecting too much of the state on the client. Cheers, Andreas >> -----Original Message----- >> From: Fan Yong [mailto:yong.fan at whamcloud.com] >> Sent: Saturday, January 21, 2012 1:11 AM >> To: Peng, Tao; bergwolf at gmail.com >> Cc: Tang, Haiying; adilger at whamcloud.com; faibish, sorin; Liu, Xuezhao; laisiyao at whamcloud.com; >> green at whamcloud.com; eeb at whamcloud.com; Bryon Neitzel >> Subject: Re: Lustre dcache clean up and distributed locking >> >> Hi Tao, >> >> Excellent work. I just added some comments inside your document. Please check. >> >> >> Best Regards, >> Fan Yong >> Whamcloud, Inc. >> >> On 1/20/12 9:34 AM, haiying.tang at emc.com wrote: >>> Hi Andreas, >>> >>> Tao is on vacation now. It might be difficult for him to check emails due to limited internet access >> at hometown. >>> For urgent issue, you folks could send email to his gmail account bergwolf at gmail.com. >>> >>> Thanks, >>> Haiying >>> >>> -----Original Message----- >>> From: Andreas Dilger [mailto:adilger at whamcloud.com] >>> Sent: 2012年1月20日 1:39 >>> To: Peng, Tao >>> Cc: faibish, sorin; Tang, Haiying; Liu, Xuezhao; >>> laisiyao at whamcloud.com; yong.fan at whamcloud.com; green at whamcloud.com; >>> eeb at whamcloud.com >>> Subject: Re: Lustre dcache clean up and distributed locking >>> >>> On 2012-01-17, at 3:21 AM, wrote: >>>> Thanks Siyao and Oleg for answering my dcache revalidation question on lustre mailing list. I >> updated the design to reflect it. >>>> >>>> Please see attachment. >>> Tao, >>> Fan Yong is also taking a more detailed look at your document and will >>> hopefully have a chance to reply before the New Year holidays. >>> >>> Also, we are just working on landing the patches to add support for >>> Linux >>> 2.6.38 for the Lustre client. One of the patches relates to the >>> lockless dcache changes that were introduced in that kernel. If you >>> are interested to review this patch, and become more familiar with the >>> Lustre development process, you should visit http://review.whamcloud.com/1865 for the patch. >>> >>> You need to create an account in Gerrit using OpenID (Google, mostly), >>> and an account in our bug tracking system (http://jira.whamcloud.com) >>> if you haven't already. >>> >>>>> -----Original Message----- >>>>> From: Andreas Dilger [mailto:adilger at whamcloud.com] >>>>> Sent: Tuesday, January 17, 2012 4:16 PM >>>>> To: Peng, Tao >>>>> Cc: faibish, sorin; Tang, Haiying; Liu, Xuezhao; Lai Siyao; Fan >>>>> Yong; Oleg Drokin; Eric Barton >>>>> Subject: Re: Lustre dcache clean up and distributed locking >>>>> >>>>> On 2012-01-16, at 9:25 PM, wrote: >>>>>> I finally started to work on Lustre dcache cleanup and locking. >>>>>> After reading Lustre, ocfs2 and VFS >>>>> dcache related code, I came to a design for cleaning up Lustre >>>>> dcache code and doing distributed locking under dcache. For >>>>> distributed locking, the main idea is to add a new inode bitlock >>>>> DENTRY lock to just protect valid dentry, instead of letting LOOKUP >>>>> lock handle multiple purpose, which makes client unable to know >>>>> whether file is deleted or not when server cancels LOOKUP lock. Instead, client holds PR mode >> DENTRY lock on all valid denties and when server cancels it, client knows the file is deleted. >>>>>> For details, please see the attachments (I attached both word and >>>>>> pdf versions as I am not sure if >>>>> which is more convenient for you:). And please help to review and comment. Thank you! >>>>> >>>>> Hi Tao, >>>>> I'm passing this on to the engineers that are most familiar with the >>>>> DLM and client dcache code in Lustre. After a quick first read >>>>> through your document, your investigation of the client code is very >>>>> insightful and looks like it will be able to remove some of the significant complexity that has >> grown over time in the llite code. >>>>> >>>>> Cheers, Andreas >>>>> -- >>>>> Andreas Dilger Whamcloud, Inc. >>>>> Principal Engineer http://www.whamcloud.com/ >>>>> >>>>> >>>> >>> >>> Cheers, Andreas >>> -- >>> Andreas Dilger Whamcloud, Inc. >>> Principal Engineer http://www.whamcloud.com/ >>> >>> >>> >>> >>> > Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Engineer http://www.whamcloud.com/ From yong.fan at whamcloud.com Tue Jan 31 12:02:14 2012 From: yong.fan at whamcloud.com (Fan Yong) Date: Tue, 31 Jan 2012 20:02:14 +0800 Subject: [Lustre-devel] Lustre dcache clean up and distributed locking In-Reply-To: References: <8449503C-66B3-4CDC-89BF-5B52E6D29579@whamcloud.com> <5F3A45681A0F5944B3194CF8C8FB905B66FA4DD4F9@MX17A.corp.emc.com> <4F19A01B.4070405@whamcloud.com> Message-ID: <4F27D846.6010504@whamcloud.com> On 1/31/12 7:27 AM, Andreas Dilger wrote: > On 2012-01-30, at 12:46 AM, wrote: >> Sorry for the late reply. I got your email earlier but didn't have chance to read the document due to very limited internet access. >> >> Thank you very much for taking detailed look at the document and giving so many valuable comments. >> >> About adding new inode bitlock, I must admit that I missed hardlink case. Although the lock mode can be changed to be the same as LOOKUP lock, I agree that inode bitlock cannot sufficiently protect every name/dentry separately for hardlinked files. And I agree, as you suggested in the document, adding a name entry based lock can solve it. >> >> IIUC, to add a name entry based lock, both client and server should be modified, and LDLM need to be changed to manage the new lock information. Like new inode bitlock, the name entry based lock will as well introduce interoperability issues as a result of on wire protocol change. And in DNE environment as Andreas explained, the original plan is to get LOOKUP on both master MDT and remote MDT. If we introduce name entry based lock, DNE case may need some change as well. >> >> I will take a closer look at the approach. In the meanwhile, could you please confirm if it is the right way to go? The benefit of above complexity seems to be better distributed dentry locking under dcache framework. >> >> What do others think? Andreas? > Tao, before we go down the road of implementing such a change, there would have to be clear benefits to the performance and/or kernel integration before we could accept the interoperability issues and added code complexity. > > The benefits of having a per-name lock, as I see them, are: > - can be cancelled separately from other name locks > - can be changed independently from the inode permissions (LOOKUP) > - potential ease of integration with the Linux dcache > > The drawbacks of introducing a per-name lock: > - only useful for hard-linked files (uncommon case) > - added code complexity for a new lock type > - incompatibility with existing client/server > - additional code to handle incompatibility transparently > - lock enqueue may need extra RPC for each name lock (different resource) > - lock cancellation may need an extra RPC for each resource > > The way I see it is that the DENTRY lock would need to be a different resource for each name on a file. The only way this is going to be useful AFAICS is if the client enqueues each DENTRY lock on a separate resource, so that it can be granted and cancelled independently of the LOOKUP lock (which is on the FID and would be common for all names that reference this inode). That would either need an extra RPC for every filename lookup (to enqueue both the DENTRY and LOOKUP locks) or a new LDLM_ENQUEUE RPC that can request and be granted two different locks (FID [seq, oid, ver, 0], and FID+name hash [seq, oid, ver, hash]) in a single RPC. > > So, the main question is how much better the client dcache/VFS integration will be with the DCACHE lock compared to the existing mechanism with the LOOKUP (FID) lock? Is it impossible (or very difficult) to use the existing mechanism for the new lockless dcache code? Or is it a matter of this being a somewhat cleaner implementation, but not critical for moving forward? > > I'm aware that our dcache code has been too complex for some time, but I haven't looked recently to see how much of this is due to using/supporting old kernel APIs, and how much of it is due to the use of the LOOKUP lock on the FID protecting too much of the state on the client. > > Cheers, Andreas Hi Tao, To introduce either per-name lock or DENTRY lock is mainly for removing Lustre special flag "DCACHE_LUSTRE_INVALID" to simply Lustre code. But if such simplification causes more complex ldlm logic, we have to consider whether it is worth to remove "DCACHE_LUSTRE_INVALID". It is not impossible to make "DCACHE_LUSTRE_INVALID" accepted by linux kernel. On the other hand, "DCACHE_LUSTRE_INVALID" can work together with lockless dcache in new linux kernel also. So to push the project move forward, we can divide it as two parts (according to your document): Lustre dcache code cleanup, and some new ldlm logic to remove "DCACHE_LUSTRE_INVALID". Before we reach an agreement on the second part, we can start from the first part to cleanup Lustre dcache code. How do you think? Cheers, -- Nasf > > >>> -----Original Message----- >>> From: Fan Yong [mailto:yong.fan at whamcloud.com] >>> Sent: Saturday, January 21, 2012 1:11 AM >>> To: Peng, Tao; bergwolf at gmail.com >>> Cc: Tang, Haiying; adilger at whamcloud.com; faibish, sorin; Liu, Xuezhao; laisiyao at whamcloud.com; >>> green at whamcloud.com; eeb at whamcloud.com; Bryon Neitzel >>> Subject: Re: Lustre dcache clean up and distributed locking >>> >>> Hi Tao, >>> >>> Excellent work. I just added some comments inside your document. Please check. >>> >>> >>> Best Regards, >>> Fan Yong >>> Whamcloud, Inc. >>> >>> On 1/20/12 9:34 AM, haiying.tang at emc.com wrote: >>>> Hi Andreas, >>>> >>>> Tao is on vacation now. It might be difficult for him to check emails due to limited internet access >>> at hometown. >>>> For urgent issue, you folks could send email to his gmail account bergwolf at gmail.com. >>>> >>>> Thanks, >>>> Haiying >>>> >>>> -----Original Message----- >>>> From: Andreas Dilger [mailto:adilger at whamcloud.com] >>>> Sent: 2012年1月20日 1:39 >>>> To: Peng, Tao >>>> Cc: faibish, sorin; Tang, Haiying; Liu, Xuezhao; >>>> laisiyao at whamcloud.com; yong.fan at whamcloud.com; green at whamcloud.com; >>>> eeb at whamcloud.com >>>> Subject: Re: Lustre dcache clean up and distributed locking >>>> >>>> On 2012-01-17, at 3:21 AM, wrote: >>>>> Thanks Siyao and Oleg for answering my dcache revalidation question on lustre mailing list. I >>> updated the design to reflect it. >>>>> Please see attachment. >>>> Tao, >>>> Fan Yong is also taking a more detailed look at your document and will >>>> hopefully have a chance to reply before the New Year holidays. >>>> >>>> Also, we are just working on landing the patches to add support for >>>> Linux >>>> 2.6.38 for the Lustre client. One of the patches relates to the >>>> lockless dcache changes that were introduced in that kernel. If you >>>> are interested to review this patch, and become more familiar with the >>>> Lustre development process, you should visit http://review.whamcloud.com/1865 for the patch. >>>> >>>> You need to create an account in Gerrit using OpenID (Google, mostly), >>>> and an account in our bug tracking system (http://jira.whamcloud.com) >>>> if you haven't already. >>>> >>>>>> -----Original Message----- >>>>>> From: Andreas Dilger [mailto:adilger at whamcloud.com] >>>>>> Sent: Tuesday, January 17, 2012 4:16 PM >>>>>> To: Peng, Tao >>>>>> Cc: faibish, sorin; Tang, Haiying; Liu, Xuezhao; Lai Siyao; Fan >>>>>> Yong; Oleg Drokin; Eric Barton >>>>>> Subject: Re: Lustre dcache clean up and distributed locking >>>>>> >>>>>> On 2012-01-16, at 9:25 PM, wrote: >>>>>>> I finally started to work on Lustre dcache cleanup and locking. >>>>>>> After reading Lustre, ocfs2 and VFS >>>>>> dcache related code, I came to a design for cleaning up Lustre >>>>>> dcache code and doing distributed locking under dcache. For >>>>>> distributed locking, the main idea is to add a new inode bitlock >>>>>> DENTRY lock to just protect valid dentry, instead of letting LOOKUP >>>>>> lock handle multiple purpose, which makes client unable to know >>>>>> whether file is deleted or not when server cancels LOOKUP lock. Instead, client holds PR mode >>> DENTRY lock on all valid denties and when server cancels it, client knows the file is deleted. >>>>>>> For details, please see the attachments (I attached both word and >>>>>>> pdf versions as I am not sure if >>>>>> which is more convenient for you:). And please help to review and comment. Thank you! >>>>>> >>>>>> Hi Tao, >>>>>> I'm passing this on to the engineers that are most familiar with the >>>>>> DLM and client dcache code in Lustre. After a quick first read >>>>>> through your document, your investigation of the client code is very >>>>>> insightful and looks like it will be able to remove some of the significant complexity that has >>> grown over time in the llite code. >>>>>> Cheers, Andreas >>>>>> -- >>>>>> Andreas Dilger Whamcloud, Inc. >>>>>> Principal Engineer http://www.whamcloud.com/ >>>>>> >>>>>> >>>>> >>>> Cheers, Andreas >>>> -- >>>> Andreas Dilger Whamcloud, Inc. >>>> Principal Engineer http://www.whamcloud.com/ >>>> >>>> >>>> >>>> >>>> > > Cheers, Andreas > -- > Andreas Dilger Whamcloud, Inc. > Principal Engineer http://www.whamcloud.com/ > > > >