From Nathan.Rutman at Sun.COM Fri Feb 1 21:01:08 2008 From: Nathan.Rutman at Sun.COM (Nathaniel Rutman) Date: Fri, 01 Feb 2008 13:01:08 -0800 Subject: [Lustre-devel] Feed API and Changelog HLD Message-ID: <47A38894.3020002@sun.com> The Feed API is a subset of the Changelog HLD. Changelog HLD is not complete, but hopefully decently fleshed out at this point. -------------- next part -------------- A non-text attachment was scrubbed... Name: feed_api.pdf Type: application/pdf Size: 93001 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: changelog-hld.pdf Type: application/pdf Size: 152806 bytes Desc: not available URL: From adilger at sun.com Fri Feb 1 23:11:02 2008 From: adilger at sun.com (Andreas Dilger) Date: Fri, 01 Feb 2008 16:11:02 -0700 Subject: [Lustre-devel] [Fwd: feed api, rev2] In-Reply-To: <47A16787.4030402@sun.com> References: <47A16787.4030402@sun.com> Message-ID: <20080201231102.GA21120@webber.adilger.int> On Jan 30, 2008 22:15 -0800, Nathaniel Rutman wrote: > hm, I tried to send this before, but it's not showing up in my mailbox. > Anyhow, fixed it up a little more... In general, it would be nicer to export as text for the mailing list unless there are graphics or something - makes it easier to comment on. struct feed entry { __u32 fe_len; total record length __u32 fe_ttype; transaction type __u64 fe_seq; local feed sequence number __u64 fe_cookie; synchronization cookie (for distributed events) __u64 fe_time; event time, server-local __u32 fe_result; return code (0=success) void *fe_data; transaction type-specific struct } struct feed_entry_open { ll_fid fid; (see lustre idl.h) ll_fid parent_fid; nid_t clientnid; __u32 fsuid; __u32 fsgid; __u32 cap; __u32 flags; __u32 mode; __u32 filename_len; char *filename; } Do you think it makes more sense to add the client nid to the feed_entry instead of the feed_entry_open? I'd think the client (or server) that performed the operation is common to every operation. For abort_size - I suspect that the default 1000 records is much too small for practical use. I'd think some millions of records would be reasonable, if there are many operations going on, so that a service that died had any chance of reconnecting before the records are lost. Alternately, it is probably safer to specify some timeout (e.g. 1h) instead of a record count. Of course, it is better if this is specified by the application, but I suspect the apps won't have a good idea of what an appropriate default value is, as that is highly dependent upon the filesystem size and usage. With the change to force the "ONESHOT" behaviour, this hurts the ability of the userspace process to pipeline operations. No matter how large a read the user process does, it will eventually have to finish processing ALL of the records before it can read any more, or they will be lost on the next read and any application failure will result in lost records if they are not completely processed. I preferred the ability to explicitly cancel records. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Oleg.Drokin at Sun.COM Sat Feb 2 06:26:35 2008 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Sat, 02 Feb 2008 01:26:35 -0500 Subject: [Lustre-devel] Reducing amount of glimpses Message-ID: Hello! Doing some large scale testing at ORNL, interesting pattern came up. Suppose we are doing large-scale IOR testing on a shared file. Some unlucky client does its writing at highest offset (or, at the beginning, was unlucky enough to grab whole-object PW lock). As other clients do their writes, they would do glimpses first to find out file size. Now those glimpses turn into thousands of glimpse requests to that poor client. And many of them actually coming in parallel. So I was thinking - perhaps it would be nice for a filter_intent_policy() to check if there are any glimpse requests being in flight to that client already for that same lock, and if there are, just wait for the request to return and use data from there? Of course potential caveat here is that we have no way to tell if the request reached client by the time we started our processing or not, and so potentially we might get size data that is a bit stale, but I wonder if this is critical enough in our case? Any ideas? Bye, Oleg From eeb at sun.com Sat Feb 2 06:39:14 2008 From: eeb at sun.com (Eric Barton) Date: Sat, 02 Feb 2008 06:39:14 +0000 Subject: [Lustre-devel] Reducing amount of glimpses In-Reply-To: References: Message-ID: <00dc01c86566$54235390$0281a8c0@ebpc> Oleg, Reduction (as you describe) is an absolutely essential strategy to achieve scalability. So without having checked through all the details of your idea, it feels like absolutely the right solution. If your idea not only changes latencies but also significant orderings, I'll feel less secure. Have you thought it all through? Cheers, Eric > -----Original Message----- > From: lustre-devel-bounces at lists.lustre.org > [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of > Oleg Drokin > Sent: 02 February 2008 6:27 AM > To: lustre-devel at clusterfs.com > Subject: [Lustre-devel] Reducing amount of glimpses > > Hello! > > Doing some large scale testing at ORNL, interesting > pattern came up. > Suppose we are doing large-scale IOR testing on a shared file. > Some unlucky client does its writing at highest offset > (or, at the > beginning, > was unlucky enough to grab whole-object PW lock). > As other clients do their writes, they would do glimpses > first to > find out file > size. Now those glimpses turn into thousands of glimpse requests > to that poor > client. And many of them actually coming in parallel. > So I was thinking - perhaps it would be nice for a > filter_intent_policy() to check > if there are any glimpse requests being in flight to that client > already for that > same lock, and if there are, just wait for the request to return > and use data from > there? > Of course potential caveat here is that we have no way to > tell if > the request reached > client by the time we started our processing or not, and so > potentially we might get > size data that is a bit stale, but I wonder if this is critical > enough in our case? > > Any ideas? > > Bye, > Oleg > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Oleg.Drokin at Sun.COM Sat Feb 2 07:00:24 2008 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Sat, 02 Feb 2008 02:00:24 -0500 Subject: [Lustre-devel] Reducing amount of glimpses In-Reply-To: <00dc01c86566$54235390$0281a8c0@ebpc> References: <00dc01c86566$54235390$0281a8c0@ebpc> Message-ID: Hello! On Feb 2, 2008, at 1:39 AM, Eric Barton wrote: > Reduction (as you describe) is an absolutely essential strategy > to achieve scalability. So without having checked through all the > details of your idea, it feels like absolutely the right solution. Yes, I just afraid that a bit stale size data might be reported, and not yet sure how critical is that to us. > If your idea not only changes latencies but also significant > orderings, > I'll feel less secure. Have you thought it all through? There is no ordering changes other than I would think perfectly normal race. Let me explain with a control flow: client3 holds highest write lock. clients 1 and 2 do glimpse. Right now (racy scenario): client 1 sends glimpse->server on its behalf sends glimpse ast to client3 -> client3 receives ast, sends reply client 2 sends glimpse->server on its behalf sends glimpse ast to client3 -> client3 writes some more data glimpse on behalf of client2 arrives to client3 -> client3 sends in updated size server now sends different sizes back to client 1 and client2. If we consolidate, it will look like this: client 1 sends glimpse->server on its behalf sends glimpse ast to client3 -> client3 receives ast, sends reply client 2 sends glimpse->server notices there is glimpse ast in flight already to client3, and waits for it client3 writes some more data server receives reply and sends identical sizes to client1 and client2. I tend to think this is normal race (kind of like with 2 processes doing stat on a local file and 3rd writing to file constantly, it is possible that when two stats are done closely apart, they would come with identical size), but just want to check if there are no objections about it. Bye, Oleg From adilger at sun.com Sat Feb 2 07:40:54 2008 From: adilger at sun.com (Andreas Dilger) Date: Sat, 02 Feb 2008 00:40:54 -0700 Subject: [Lustre-devel] Reducing amount of glimpses In-Reply-To: References: <00dc01c86566$54235390$0281a8c0@ebpc> Message-ID: <20080202074054.GA23836@webber.adilger.int> On Feb 02, 2008 02:00 -0500, Oleg Drokin wrote: > On Feb 2, 2008, at 1:39 AM, Eric Barton wrote: > > Reduction (as you describe) is an absolutely essential strategy > > to achieve scalability. So without having checked through all the > > details of your idea, it feels like absolutely the right solution. > > Yes, I just afraid that a bit stale size data might be reported, and > not yet sure how critical is that to us. A glimpse enqueue is already inherently racy, so I don't think this is a serious problem. > > If your idea not only changes latencies but also significant > > orderings, I'll feel less secure. Have you thought it all through? > > There is no ordering changes other than I would think perfectly normal > race. Let me explain with a control flow: > > client3 holds highest write lock. clients 1 and 2 do glimpse. > Right now (racy scenario): > > client 1 sends glimpse->server on its behalf sends glimpse ast to > client3 -> client3 receives ast, sends reply > client 2 sends glimpse->server on its behalf sends glimpse ast to > client3 -> > client3 writes some more data > glimpse on behalf of client2 arrives to client3 -> client3 sends in > updated size > server now sends different sizes back to client 1 and client2. > > If we consolidate, it will look like this: > client 1 sends glimpse->server on its behalf sends glimpse ast to > client3 -> client3 receives ast, sends reply > client 2 sends glimpse->server notices there is glimpse ast in flight > already to client3, and waits for it > client3 writes some more data > server receives reply and sends identical sizes to client1 and client2. > > I tend to think this is normal race (kind of like with 2 processes > doing stat on a local file and 3rd writing to file constantly, > it is possible that when two stats are done closely apart, they would > come with identical size), but just want to check if > there are no objections about it. The uses of glimpse are only in a few cases: - ls -l (inherently racy) - read - write In the read and write cases the glimpse return is used to detemine if the file size is before or after the given client's lock. Since the client that is getting the glimpse is holding the EOF lock this can't change the relative positions of the locks. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Nikita.Danilov at Sun.COM Sat Feb 2 11:22:21 2008 From: Nikita.Danilov at Sun.COM (Nikita Danilov) Date: Sat, 02 Feb 2008 14:22:21 +0300 Subject: [Lustre-devel] Reducing amount of glimpses In-Reply-To: References: Message-ID: <18340.21101.635645.378808@gargle.gargle.HOWL> Oleg Drokin writes: > Hello! > > Doing some large scale testing at ORNL, interesting pattern came up. > Suppose we are doing large-scale IOR testing on a shared file. > Some unlucky client does its writing at highest offset (or, at the > beginning, > was unlucky enough to grab whole-object PW lock). > As other clients do their writes, they would do glimpses first to > find out file > size. Now those glimpses turn into thousands of glimpse requests > to that poor > client. And many of them actually coming in parallel. I think there is a way to further reduce the number of glimpses sent in the read/write paths. In a situation when client is doing read and the region being read ends in a certain stripe, it's often enough to send a glimpse request only to the OST where this stripe is located. Indeed, if stripe is not a hole, then size of this stripe alone is sufficient to determine whether we are in a short read situation or not. Of course if stripe happens to end in a hole, client needs to send more glimpses, but in the most common case of non-sparse file, 1 RPC seems to be enough. > > Bye, > Oleg Nikita. From Peter.Braam at Sun.COM Mon Feb 4 15:17:10 2008 From: Peter.Braam at Sun.COM (Peter J Braam) Date: Mon, 04 Feb 2008 08:17:10 -0700 Subject: [Lustre-devel] Feed API and Changelog HLD In-Reply-To: <47A38894.3020002@sun.com> References: <47A38894.3020002@sun.com> Message-ID: <47A72C76.7000206@sun.com> On page 10 of the changelog API: recursive behavior for directories in filesets must exclude directories that are mount point of other filesets (as implemented for the Path Forward a few years back). I didn't see where ABORT sizes are discussed, but as I mentioned before there are few cases where ditching records is possible. The storage of the records should be discussed. The llog headers may need changes and probably the API's for canceling records will change to indicate what consumer is canceling. It would be wise to discuss how disk full failures etc are handled in some detail. - peter - Nathaniel Rutman wrote: > The Feed API is a subset of the Changelog HLD. > Changelog HLD is not complete, but hopefully decently fleshed out at > this point. > > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Nathan.Rutman at Sun.COM Mon Feb 4 22:39:36 2008 From: Nathan.Rutman at Sun.COM (Nathaniel Rutman) Date: Mon, 04 Feb 2008 14:39:36 -0800 Subject: [Lustre-devel] [Fwd: feed api, rev2] In-Reply-To: <20080201231102.GA21120@webber.adilger.int> References: <47A16787.4030402@sun.com> <20080201231102.GA21120@webber.adilger.int> Message-ID: <47A79428.9060702@sun.com> Andreas Dilger wrote: > > With the change to force the "ONESHOT" behaviour, this hurts the ability > of the userspace process to pipeline operations. No matter how large a > read the user process does, it will eventually have to finish processing > ALL of the records before it can read any more, or they will be lost on > the next read and any application failure will result in lost records > if they are not completely processed. I preferred the ability to > explicitly cancel records. > There are arguments on both sides. I'd really like to stay away from having multiple cancellation mechanisms though - I think this will be extra work and extra confusion. With explicit cancel and a regular file we can have multiple readers all reading the same audit log, which is a big plus. (Record would be purged at first cancel.) This adds complication for the users though, which is a minus: they must now a) persistently keep track of their last-processed record ID, b) explicitly cancel records, c) map record numbers to file offsets. I think c) is a real hassle. With explicit cancel and a FIFO we have a) and b), but not c), but not multiple readers either. From Jinshan.Xiong at Sun.COM Tue Feb 5 04:34:13 2008 From: Jinshan.Xiong at Sun.COM (jay) Date: Tue, 05 Feb 2008 12:34:13 +0800 Subject: [Lustre-devel] raid5 r/w performance for rhel5, and a comparison to rhel4 kernel Message-ID: <47A7E745.1020607@sun.com> Please check the attachment. The rebuild time has been decreased dramatically after fixing a bug in md framework. Jay -------------- next part -------------- A non-text attachment was scrubbed... Name: Thumper-sgp_dd_rhel5.ods Type: application/vnd.oasis.opendocument.spreadsheet Size: 313189 bytes Desc: not available URL: From Peter.Braam at Sun.COM Tue Feb 5 13:43:48 2008 From: Peter.Braam at Sun.COM (Peter J Braam) Date: Tue, 05 Feb 2008 06:43:48 -0700 Subject: [Lustre-devel] Feed API and Changelog HLD In-Reply-To: <47A7B8C7.1040002@sun.com> References: <47A38894.3020002@sun.com> <47A72C76.7000206@sun.com> <47A7B8C7.1040002@sun.com> Message-ID: <47A86814.2000905@sun.com> Nathaniel Rutman wrote: > Peter J Braam wrote: >> On page 10 of the changelog API: recursive behavior for directories >> in filesets must exclude directories that are mount point of other >> filesets (as implemented for the Path Forward a few years back). >> >> I didn't see where ABORT sizes are discussed, but as I mentioned >> before there are few cases where ditching records is possible. >> > I don't know what you're talking about with ABORT - you mean e.g. HSM > should > abort copying an old file if the file is modified after the copy > starts? Or something else? > Hmm, this was my jetlagged head, sorry. I meant when you want to force a truncation of the log (and throw records away), how you do handle this? You need this mechanism, even though it will be used only rarely. While I am writing this, I also realize that there are some logs that may require re-compacting. This happens for the LRU logs (tracking most recent access to files for HSM migration management) I think. The idea for these logs is to (1) append a new record when an object is accessed, and (2) cancel the previous record for that object using a pointer to that record in the object. These records can be tiny (obj id / parent obj id), but the mechanisms causes llog files to be non-empty with holes. >> The storage of the records should be discussed. The llog headers may >> need changes and probably the API's for canceling records will change >> to indicate what consumer is canceling. >> >> It would be wise to discuss how disk full failures etc are handled in >> some detail. >> > ok. Does it seem to be on the right track in general now? At a 10,000 feet level yes. My concerns at this point are error handling and performance and multiple consumer management. That's not so clear yet. - Peter - From Peter.Bojanic at Sun.COM Wed Feb 6 22:01:29 2008 From: Peter.Bojanic at Sun.COM (Peter Bojanic) Date: Wed, 06 Feb 2008 17:01:29 -0500 Subject: [Lustre-devel] Lustre developer documentation now available Message-ID: <70997156-F0D6-4177-BB4F-6FC67D8C97D3@sun.com> Dear Lustre Community, The Lustre Group at Sun is pleased to announce the availability of detailed technical documentation for Lustre. This material has existed for some time but was closely guarded intellectual property by the former CFS. In the spirit of making Lustre continually more open, we have been encouraged by Sun execs to make this valuable information available to the Lustre community. Lustre High Level Designs (HLD): http://arch.lustre.org/index.php?title=LustreHLDs Lustre Detailed Level Designs (DLD): http://arch.lustre.org/index.php?title=LustreDLDs Lustre Internals: http://wiki.lustre.org/index.php?title=Lustre_Internals Lustre HLDs and DLDs are available for download from the CVS repository in their original Lyx format. The HLD and DLD wiki pages include CVS download instructions. Additionally, each wiki page has links for the most current design documents available in PDF format. For each PDF, the author and date are listed, along with a synopsis. Over the coming days, we'll continue to make more of the 2008, 2007 and 2006 HLDs and DLDs available as PDFs on the wiki pages. Note that some of this documentation may be out of date, incomplete, or not yet implemented in Lustre. But it still provides useful reference material for understanding Lustre. The Lustre Internals course was taught a few times in the past year by Peter Braam. We've decided that it represents more of a useful knowledge resource than course material. It is our hope that this documentation will contribute to nurturing a stronger and more vibrant development community for Lustre. Cheers, Bojanic From bs at q-leap.de Wed Feb 6 22:34:07 2008 From: bs at q-leap.de (Bernd Schubert) Date: Wed, 6 Feb 2008 23:34:07 +0100 Subject: [Lustre-devel] Lustre developer documentation now available In-Reply-To: <70997156-F0D6-4177-BB4F-6FC67D8C97D3@sun.com> References: <70997156-F0D6-4177-BB4F-6FC67D8C97D3@sun.com> Message-ID: <20080206223407.GA4569@lanczos.q-leap.de> Hello Peter, great, thanks a lot! Cheers, Bernd On Wed, Feb 06, 2008 at 05:01:29PM -0500, Peter Bojanic wrote: > Dear Lustre Community, > > The Lustre Group at Sun is pleased to announce the availability of > detailed technical documentation for Lustre. > > This material has existed for some time but was closely guarded > intellectual property by the former CFS. In the spirit of making > Lustre continually more open, we have been encouraged by Sun execs to > make this valuable information available to the Lustre community. > > Lustre High Level Designs (HLD): http://arch.lustre.org/index.php?title=LustreHLDs > > Lustre Detailed Level Designs (DLD): http://arch.lustre.org/index.php?title=LustreDLDs > > Lustre Internals: http://wiki.lustre.org/index.php?title=Lustre_Internals > > Lustre HLDs and DLDs are available for download from the CVS > repository in their original Lyx format. The HLD and DLD wiki pages > include CVS download instructions. Additionally, each wiki page has > links for the most current design documents available in PDF format. > For each PDF, the author and date are listed, along with a synopsis. > Over the coming days, we'll continue to make more of the 2008, 2007 > and 2006 HLDs and DLDs available as PDFs on the wiki pages. > > Note that some of this documentation may be out of date, incomplete, > or not yet implemented in Lustre. But it still provides useful > reference material for understanding Lustre. > > The Lustre Internals course was taught a few times in the past year by > Peter Braam. We've decided that it represents more of a useful > knowledge resource than course material. > > It is our hope that this documentation will contribute to nurturing a > stronger and more vibrant development community for Lustre. > > Cheers, > Bojanic > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From aurelien.degremont at cea.fr Thu Feb 7 10:52:55 2008 From: aurelien.degremont at cea.fr (DEGREMONT Aurelien) Date: Thu, 07 Feb 2008 11:52:55 +0100 Subject: [Lustre-devel] Lustre HSM HLD draft Message-ID: <47AAE307.9040305@cea.fr> Hello Here is a first draft for comments of the Lustre HSM HLD. It is intended to be a support for further analyzes and comments from CFS/Sun. The document covers the main parts of the HSM features but some elements are still lacking. The policy management and the space manager will be describe later. Let us know your comments and ideas about it. Regards, Aurelien Degremont CEA -------------- next part -------------- A non-text attachment was scrubbed... Name: hld_hsm.pdf Type: application/pdf Size: 159330 bytes Desc: not available URL: From Peter.Bojanic at Sun.COM Thu Feb 7 13:54:51 2008 From: Peter.Bojanic at Sun.COM (Peter Bojanic) Date: Thu, 07 Feb 2008 08:54:51 -0500 Subject: [Lustre-devel] Lustre-devel Digest, Vol 23, Issue 6 In-Reply-To: References: Message-ID: Aurélien, This is fantastic to see! Thanks so much for this contribution to Lustre. I expect you're going to get a lot of interest and feedback. You may consider logging a bug and encourage feedback through comments. This way, you can have a consolidated history of feedback. Someone like Johann or Andreas can offer you more detailed guidance. Cheers, Bojanic > Date: Thu, 07 Feb 2008 11:52:55 +0100 > From: DEGREMONT Aurelien > Subject: [Lustre-devel] Lustre HSM HLD draft > To: lustre-devel at lists.lustre.org > Message-ID: <47AAE307.9040305 at cea.fr> > Content-Type: text/plain; charset="utf-8" > > Hello > > Here is a first draft for comments of the Lustre HSM HLD. > It is intended to be a support for further analyzes and comments from > CFS/Sun. > > The document covers the main parts of the HSM features but some > elements > are still lacking. > The policy management and the space manager will be describe later. > > Let us know your comments and ideas about it. > > Regards, > > Aurelien Degremont > CEA From Richard.Matthews at Sun.COM Thu Feb 7 16:19:51 2008 From: Richard.Matthews at Sun.COM (Rick Matthews) Date: Thu, 07 Feb 2008 10:19:51 -0600 Subject: [Lustre-devel] Lustre HSM HLD draft Message-ID: <47AB2FA7.10205@Sun.COM> All, I'm new to this list, so I'll start with apologies. My Lustre background is also limited; a situation I hope to fix. As part of the Solaris Software Archiving group, I was asked to review the HSM HLD by my management. That review was sent to Peter Bojanic. He suggested I get involved in the community discussion. This is a posting of my original response, based on a copy of the HLD which seems to be the one posted. I've made a couple of minor corrections. Page 1, 1, Define coordinator (space coordinator?), define agent, (condense Part II intro, page 14) (for me, MDT, MGS and OST) Page 8, 3.8, "use" not "used" in second sentence Page 9, 3.8.2 et.al., "precised" (maybe, explicit or precise) Page 9, 3.8.4, Lustre ID "if" no path Page 10, 4.1, 1) When archived? (probably in Space Manager portion) SAM-QFS archives well ahead of space need. 4) External object reference must be unusable, until 5. 4.2, 2) Implies only one copy per "version"...bad idea Page 12, 5.3, Last Sentence, This enables, not This ables 6.1, 100,000 migrations make current migration list operations problematic (lets say want to move last migration to be next migration). Page 13, Lustre object mtime may not be good enough. There are several mechanisms (like touch) to manipulate mtime, which makes it unusable as a last written time. Page 15, a variant on 1.5, ask for/return last valid byte offset (perhaps within a range). Page 19, Special Path, does this boil down to invisible I/O? Page 23, 2.3 and 2.4, I'm assuming that lists of tuples can be processed in any order. Page 25, 1, Punch - becomes "sparse" not "spare" I think this spec needs to be more consistent with its use of data range. It is confusing as laid out. Page 26, 3.2 space will be exhausted, or space will be low, not space will be missing. Page 28, protection of Lustre extended attributes? Issues: The Space manager is likely the most important piece. There is no detail on it. This is where archive and other policy is enforced. The described HSM seems to follow the "copy out" when space needed, then purge, model. This function (a Space Manager function) is contrary to SAM, and a shortfall of many HSMs. File/object association is an important component of SAM. For example, if I access a file in a source tree, I'm likely to access the others as well. The purge (3.2, Space manager needs to make room) and 4.1 "needs to be atomic" is a complex operations. Sequencing is important. Coordination between agents seems important. For example, if agents requested new copy-outs on objects striped on 10 different stores, ordering them on tape seems difficult. What is the backup story for Lustre? How does that play with the HSM? -- --------------------------------------------------------------------- Rick Matthews email: Rick.Matthews at sun.com Sun Microsystems, Inc. phone:+1(651) 554-1518 1270 Eagan Industrial Road phone(internal): 54418 Suite 160 fax: +1(651) 554-1540 Eagan, MN 55121-1231 USA main: +1(651) 554-1500 --------------------------------------------------------------------- From JC.LAFOUCRIERE at CEA.FR Fri Feb 8 00:03:19 2008 From: JC.LAFOUCRIERE at CEA.FR (JC.LAFOUCRIERE at CEA.FR) Date: Fri, 8 Feb 2008 01:03:19 +0100 Subject: [Lustre-devel] Lustre HSM HLD draft References: <47AB2FA7.10205@Sun.COM> Message-ID: Hello thank you for your review, I add some comments in the following Page 1, 1, Define coordinator (space coordinator?), define agent, (condense Part II intro, page 14) (for me, MDT, MGS and OST) These are defined in the arch wiki pages Page 10, 4.2, 2) Implies only one copy per "version"...bad idea Different versions correspond to different files in the external storage. We take the more recent. Not sure I understand your remark Page 13, Lustre object mtime may not be good enough. There are several mechanisms (like touch) to manipulate mtime, which makes it unusable as a last written time. If a user make a touch in the past this change the mtime and can hide previous writes. If we want to keep real write time we need to add a new time field in Lustre backend (may be ZFS has it) Page 19, Special Path, does this boil down to invisible I/O? The path is /mnt_mount/.lustre/fid/FID_NUMBER. When a file is open through this path a flag is carried to the OSS to avoid copy in trigger (this used to fill the file) Page 23, 2.3 and 2.4, I'm assuming that lists of tuples can be processed in any order. yes Issues: The Space manager is likely the most important piece. There is no detail on it. This is where archive and other policy is enforced. The space manager is based on changelogs/feed Lustre feature which are very new (draft HLD has just been published). This is why it not described at this time. The described HSM seems to follow the "copy out" when space needed, then purge, model. This function (a Space Manager function) is contrary to SAM, and a shortfall of many HSMs. no spacemanger is doing pre-migration and when free space is needed, it only has to make punc Coordination between agents seems important. For example, if agents requested new copy-outs on objects striped on 10 different stores, ordering them on tape seems difficult. Tape access optimization has to be made by the archival system. We try to put as few external storage knowledge as possible in Lustre to be external storage independant. What is the backup story for Lustre? How does that play with the HSM? HSM do not backup the namespace. It has to be done with a separate tool like a MDT scannner. The copy tool can use the FID2PATH() function to save the object pathname with the file. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Richard.Matthews at Sun.COM Fri Feb 8 11:52:12 2008 From: Richard.Matthews at Sun.COM (Rick Matthews) Date: Fri, 08 Feb 2008 05:52:12 -0600 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: References: <47AB2FA7.10205@Sun.COM> Message-ID: <47AC426C.8060902@Sun.COM> JC.LAFOUCRIERE at CEA.FR wrote: Thanks for allowing me to participate. > Hello > > thank you for your review, I add some comments in the following > > Page 1, 1, Define coordinator (space coordinator?), > define agent, (condense Part II intro, page 14) > (for me, MDT, MGS and OST) > These are defined in the arch wiki pages > Thank you, I still haven't got to them yet...but plan to. > Page 10, > 4.2, 2) Implies only one copy per "version"...bad idea > Different versions correspond to different files in the external storage. We take the more recent. > Not sure I understand your remark > A basic mantra of SAM-QFS and other data retention systems is that one image of the data is vulnerable (a tape breaks, or is otherwise overwritten). While the archival system can be responsible for making multiple identical images, it can still represent a single point of failure. Note: I am using version to represent a point in time image of the files data, and copy to represent an image of that version. (See LOCKSS for additional references on copies). > Page 13, Lustre object mtime may not be good enough. There are several > mechanisms (like touch) to manipulate mtime, which makes it > unusable as a last written time. > If a user make a touch in the past this change the mtime and can hide previous writes. > If we want to keep real write time we need to add a new time field in Lustre backend > (may be ZFS has it) > What the archival system needs to know is that the copy previously made (or a first copy need to be made), which seems to be triggered by a user (not archive or other - like restore) write operation. > Page 19, Special Path, does this boil down to invisible I/O? > The path is /mnt_mount/.lustre/fid/FID_NUMBER. When a file is open through this path a > flag is carried to the OSS to avoid copy in trigger (this used to fill the file) > > Page 23, 2.3 and 2.4, I'm assuming that lists of tuples can be processed > in any order. > yes > > Issues: > The Space manager is likely the most important piece. There is no > detail on it. This is where archive and other policy is enforced. > The space manager is based on changelogs/feed Lustre feature which are very new (draft HLD has just been > published). This is why it not described at this time. > OK...also consider using change logs as a trigger for need of a new archive version (not copy). Alleviates the mtime issue above. > The described HSM seems to follow the "copy out" when space needed, > then purge, model. This function (a Space Manager function) is contrary > to SAM, and a shortfall of many HSMs. > no spacemanger is doing pre-migration and when free space is needed, it only has to make punc > OK, so who schedules the pre-migration to the archive system? > Coordination between agents seems important. For example, > if agents requested new copy-outs on objects striped on > 10 different stores, ordering them on tape seems difficult. > Tape access optimization has to be made by the archival system. We try to put as few external storage knowledge > as possible in Lustre to be external storage independant. > The isolation between archive system and file system is (to me) a good idea. I'd just like you to consider that the recall (stage-in) events can be optimized. At least, make sure the archive system is allowed to reorder as needed (hence the async - list of tuples in any order - question above). Think of other association between files to live storage as 1) a pre-stage operation, or 2) a disk cache pre-fetch operation. I hope I'm using understandable words ;>) > What is the backup story for Lustre? How does that play with > the HSM? > HSM do not backup the namespace. It has to be done with a separate tool like a MDT scannner. > The copy tool can use the FID2PATH() function to save the object pathname with the file. > > One point here is that an HSM + namespace/metadata backup + unarchived data capture can be used to be a nearly continuous backup operation with a relatively tiny backup window. -- --------------------------------------------------------------------- Rick Matthews email: Rick.Matthews at sun.com Sun Microsystems, Inc. phone:+1(651) 554-1518 1270 Eagan Industrial Road phone(internal): 54418 Suite 160 fax: +1(651) 554-1540 Eagan, MN 55121-1231 USA main: +1(651) 554-1500 --------------------------------------------------------------------- From ron.rex at gmail.com Fri Feb 8 15:21:51 2008 From: ron.rex at gmail.com (Ron) Date: Fri, 8 Feb 2008 07:21:51 -0800 (PST) Subject: [Lustre-devel] 2.6.23 kernel struct file_operations sendfile Message-ID: <243f2731-0369-4d16-9e1c-74ab010ea343@m34g2000hsb.googlegroups.com> Hi, In trying (just to see what happens) to build against patchless 2.6.23, I see that the file_operations structure no longer has the sendfile field. Has anyone in the development team thought of a way around this? Thanks, Ron From Alex.Zhuravlev at Sun.COM Fri Feb 8 15:31:47 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Fri, 08 Feb 2008 18:31:47 +0300 Subject: [Lustre-devel] [RFC] parallel enqueue Message-ID: <47AC75E3.7010901@sun.com> Hi, in some cases (truncate, append) we still use serialized enqueue when all locks have to be enqueued synchronously one by one. what if we could mark all locks issued by single client with some unique tag (timestamp + nid?), then enqueue them all and then, in case of conflict, in blocking ast handler compare tag of conflicting lock with own tag, cancel our granted locks if our tag is greater than tag of conflicting lock and enqueue them again? thanks, Alex From aurelien.degremont at cea.fr Fri Feb 8 15:55:33 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Fri, 08 Feb 2008 16:55:33 +0100 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47AB2FA7.10205@Sun.COM> References: <47AB2FA7.10205@Sun.COM> Message-ID: <47AC7B75.30603@cea.fr> Hello First of all, thanks for your remarks. Information explained in the architecture documents from the Arch Wiki have not been re-explained in the HLD. So some points could be unclear, but read or check the arch docs first. If the HLD must be self sufficent or more details are really needed, let me know. I will clarify some points anyway in the new document version. Rick Matthews a écrit : > Page 10, 4.1, 1) When archived? (probably in Space Manager portion) > SAM-QFS archives well ahead of space need. Concerning the archived copies vunlerability, I'm not sure this is Lustre responsability to manage several copies of each of its file versions into the HSM... > 6.1, 100,000 migrations make current migration list operations > problematic (lets say want to move last migration to > be next migration). You speak about pending migrations ? This is just pointer manipulation. I do not see a real problem at this level. This value is only algorithmic indications, not about resources (memory, ...) But we could decrease this value to 10,000. > Page 13, Lustre object mtime may not be good enough. There are several > mechanisms (like touch) to manipulate mtime, which makes it > unusable as a last written time. If fact, this value is only needed for user information, not for Lustre internals. Lustre will based is comparison on the FID version. The mtime field is used for listing the file copies in the HSM, and as the lustre fid version is not relevant for the user, will indicates the associated file date at this time. (just a quick example, not the final output) user$ list_hsm_copies ./foo Storage Date Size Version ============================================ HSM1 Feb 2 2006 1566162 1 HSM1 Jun 18 2007 1423540 2 HSM1 Jun 18 2007 1900051 54 But the touch could be problematic. Lustre gurus, is there another time field we could use instead ? Should we add a "last-modification-field-which-ignore-touch" ? Is this really a problem is we use display a "touched" time ? In this case, we display what the user set on the file, we suppose he did it in purpose. > Page 15, a variant on 1.5, ask for/return last valid byte offset > (perhaps within a range). Why not... But do you have use cases were the current "Data available" feature as explained in 1.5 is not sufficent ? > Page 28, protection of Lustre extended attributes? I do not see what you mean. > Issues: > The purge (3.2, Space manager needs to make room) and 4.1 > "needs to be atomic" is a complex operations. Sequencing is > important. Does "transactionnal" fit ? I will add a Bugzilla entry and a new updated version the HLD on it, next Monday. Regards, -- Aurelien Degremont CEA/DAM - DIF/DSSI/SISR From Alexey.Lyashkov at Sun.COM Fri Feb 8 17:54:32 2008 From: Alexey.Lyashkov at Sun.COM (Alex Lyashkov) Date: Fri, 08 Feb 2008 19:54:32 +0200 Subject: [Lustre-devel] 2.6.23 kernel struct file_operations sendfile In-Reply-To: <243f2731-0369-4d16-9e1c-74ab010ea343@m34g2000hsb.googlegroups.com> References: <243f2731-0369-4d16-9e1c-74ab010ea343@m34g2000hsb.googlegroups.com> Message-ID: <1202493272.6760.27.camel@bear.shadowland> Hi Ron, look into bugzilla bug 14250. This bug include patch for buuld patchless client against vanila 2.4.23/24.24 kernels. On Fri, 2008-02-08 at 07:21 -0800, Ron wrote: > Hi, > In trying (just to see what happens) to build against patchless > 2.6.23, I see that the > file_operations structure no longer has the sendfile field. Has > anyone in the development > team thought of a way around this? > Thanks, > Ron > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From ron.rex at gmail.com Fri Feb 8 19:03:26 2008 From: ron.rex at gmail.com (Ron) Date: Fri, 8 Feb 2008 11:03:26 -0800 (PST) Subject: [Lustre-devel] 2.6.23 kernel struct file_operations sendfile In-Reply-To: <1202493272.6760.27.camel@bear.shadowland> References: <243f2731-0369-4d16-9e1c-74ab010ea343@m34g2000hsb.googlegroups.com> <1202493272.6760.27.camel@bear.shadowland> Message-ID: <3b98d27b-5256-4adf-8e83-5efe1231a25b@s13g2000prd.googlegroups.com> Hi Alex, great!, Thanks. I'm having trouble finding a version of the source that the patch will apply cleaning to. I've tried releases 1.6.0.1, 1.6.3, 1.6.4.1, 1.6.4.2 and then, from cvs, a checkout of a few days ago. Could you please, perhaps give me a cvs tag that I can use? or some other way to get a source tree (to apply the patch to) and then build against patchless 2.6.23.12? Thanks. From Peter.Braam at Sun.COM Fri Feb 8 20:37:48 2008 From: Peter.Braam at Sun.COM (Peter J Braam) Date: Fri, 08 Feb 2008 13:37:48 -0700 Subject: [Lustre-devel] [RFC] parallel enqueue In-Reply-To: <47AC75E3.7010901@sun.com> References: <47AC75E3.7010901@sun.com> Message-ID: <47ACBD9C.3060402@sun.com> Yes this is a good idea, but probably not extremely urgent. - Peter - Alex Zhuravlev wrote: > Hi, > > in some cases (truncate, append) we still use serialized enqueue > when all locks have to be enqueued synchronously one by one. > > what if we could mark all locks issued by single client with some > unique tag (timestamp + nid?), then enqueue them all and then, in > case of conflict, in blocking ast handler compare tag of conflicting > lock with own tag, cancel our granted locks if our tag is greater > than tag of conflicting lock and enqueue them again? > > thanks, Alex > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Nathan.Rutman at Sun.COM Fri Feb 8 21:18:20 2008 From: Nathan.Rutman at Sun.COM (Nathaniel Rutman) Date: Fri, 08 Feb 2008 13:18:20 -0800 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47AAE307.9040305@cea.fr> References: <47AAE307.9040305@cea.fr> Message-ID: <47ACC71C.7050808@sun.com> DEGREMONT Aurelien wrote: > Hello > > Here is a first draft for comments of the Lustre HSM HLD. > It is intended to be a support for further analyzes and comments from > CFS/Sun. > > The document covers the main parts of the HSM features but some elements > are still lacking. > The policy management and the space manager will be describe later. > > Let us know your comments and ideas about it. > > Regards, 5.1 external storage list - is this to be stored on the MGS device or a separate device? If the coordinator lives on the MGS, why not it's storage as well? In any case, it should be possible to co-locate the coordinator on the MGS and used the MGS's storage device, in the same way that the MGS can currently co-locate with the MDT. 6.3 object ref should include version number. Also include checksum? How does the coordinator request activity from an agent? If the coordinator is the RPC server, then it's up to the agents to make requests; agents aren't listening for RPC requests themselves. 2.1Archiving one Lustre file There should not be a cache miss when archiving a lustre file; perhaps open-by-fid is intended to bypass atime updates so that the file isn't marked as "recently accessed"? 2.2Restoring a file "External ID" presumably contains all information required to retrieve the file - tape #, path name, etc? Once file is copied back, we should probably restore original ctime, mtime, atime - coordinator is storing this, correct? IV2 - why not multiple purged windows? Seems like if you're going to purge 1 object out of a file, you might want to purge more. Specifically, it will probably be a common case to purge every object of a file from a particular OST. This is not contiguous in a striped file. I don't see any reason to purge anything smaller than an entire object on an OST - is there good reason for this? If that's the case, then it the OST must keep track of purged objects, not ranges within an existing object. If the MDT is tracking purged areas also, then there's a good potential synergy here with a missing OST -- If the missing OST's objects are marked as purged, then we can potentially recover them automatically from HSM... 4.2 How is a purge request recovered? For example, MDT says purge obj1 from ost1, ost1 replies "ok", but then dies before it actually does the purge. Reboots, doesn't know anything about purge request now, but MDT has marked it as purged. Transparent access - should this avoid modification of atime/mtime? V2.1 How long does OST wait for completion? Is there a timeout? We probably need a "no timeout if progress is being made" kind of function - clients currently do this kind of thing with OSTs. V2.2 No need to copy-in purged data on full-object-size writes. Page 13, Lustre object mtime may not be good enough. There are several mechanisms (like touch) to manipulate mtime, which makes it unusable as a last written time. If a user make a touch in the past this change the mtime and can hide previous writes. If we want to keep real write time we need to add a new time field in Lustre backend (may be ZFS has it) If a user touches or otherwise modifies the mtime on purpose, they presumably know what they are doing. Besides, we're using the object version number, not the mtime, to determine whether a file is up to date. I think this can be ignored. From Alexey.Lyashkov at Sun.COM Sat Feb 9 05:14:38 2008 From: Alexey.Lyashkov at Sun.COM (Alex Lyashkov) Date: Sat, 09 Feb 2008 07:14:38 +0200 Subject: [Lustre-devel] 2.6.23 kernel struct file_operations sendfile In-Reply-To: <3b98d27b-5256-4adf-8e83-5efe1231a25b@s13g2000prd.googlegroups.com> References: <243f2731-0369-4d16-9e1c-74ab010ea343@m34g2000hsb.googlegroups.com> <1202493272.6760.27.camel@bear.shadowland> <3b98d27b-5256-4adf-8e83-5efe1231a25b@s13g2000prd.googlegroups.com> Message-ID: <1202534078.6760.42.camel@bear.shadowland> Hi Ron, Patch created against sources from cvs - branch b1_6 date 20080116. Patch tested for first with 2.6.23.8 and later with 2.6.24-rc8. if you found any problems with this patch, please submit info into bugzilla. I'm will be fix in next patch version. On Fri, 2008-02-08 at 11:03 -0800, Ron wrote: > Hi Alex, > great!, Thanks. > I'm having trouble finding a version of the source that the patch will > apply cleaning to. > I've tried releases 1.6.0.1, 1.6.3, 1.6.4.1, 1.6.4.2 and then, from > cvs, a checkout of a few days ago. > Could you please, perhaps give me a cvs tag that I can use? or some > other way to get a source tree (to apply the patch to) and then build > against patchless 2.6.23.12? > Thanks. > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From aurelien.degremont at cea.fr Mon Feb 11 14:59:06 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Mon, 11 Feb 2008 15:59:06 +0100 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47ACC71C.7050808@sun.com> References: <47AAE307.9040305@cea.fr> <47ACC71C.7050808@sun.com> Message-ID: <47B062BA.1070809@cea.fr> Nathaniel Rutman a écrit : > 5.1 external storage list - is this to be stored on the MGS device or a > separate device? If the coordinator lives on the MGS, why not it's > storage as well? In any case, it should be possible to co-locate the > coordinator on the MGS and used the MGS's storage device, in the same > way that the MGS can currently co-locate with the MDT. > How does the coordinator request activity from an agent? If the > coordinator is the RPC server, then it's up to the agents to make > requests; agents aren't listening for RPC requests themselves. Presently, it is never said that the coordinator will live on the MGS. The Coordinator constrains are: 1 - Must receive various migration requests from OST/MDT. 2 - Should be able to communicate with Agents and asks them migrations. 3 - Should store configuration and migration logs. I think #1 and #2 are two differents API. The coordinator is clearly a RPC server for the first one. How #2 should be implemented is not so clear. What would be be the "Lustre-way" here? For #3, the few logs that will be backed up here are not huge, and it surely could be colocated with another Target, but I'm not sure this should be mandatory. This device should be available to several servers, for failover like the other Targets. We could imagine having more than 1 coordinator at long term. I'm not sure it is a good idea to stick it to another target. > 6.3 object ref should include version number. Also include checksum? For data coherency? Should we add a explicit checksum for those values (stored in an EA) or used a possible backend feature (Can ZFS and ldiskfs detect EA value corruption by themselves?) ? > 2.1Archiving one Lustre file > There should not be a cache miss when archiving a lustre file; perhaps > open-by-fid is intended to bypass atime updates > so that the file isn't marked as "recently accessed"? > Transparent access - should this avoid modification of atime/mtime? I would say yes. > 2.2Restoring a file > "External ID" presumably contains all information required to retrieve > the file - tape #, path name, etc? > Once file is copied back, we should probably restore original ctime, > mtime, atime - coordinator is storing this, correct? External ID is an opaque value manage by the archiving tool. If the HSM can store a lot of metadata, only a ref is needed, if not, the tool is responsible for storing all the data it needs. Anyway, this is totally opaque for Lustre. I hope the HSMs will not need so many data in this field. HPSS does not need so many data, it uses its internal DB to store them. I suppose SAM also. > IV2 - why not multiple purged windows? Seems like if you're going to > purge 1 object out of a file, you might want to purge more. > Specifically, it will probably be a common case to purge every object of > a file from a particular OST. This is not contiguous in a > striped file. > I don't see any reason to purge anything smaller than an entire object > on an OST - is there good reason for this? Multiple purged window is subtle. If you permit this feature, you could technically have, in the worst case, one purged window per byte, and this could be very huge to store. Do you think you will do several holes in the same file? In which cases? In fact, the more common case is to totally purge a file which have been migrated on HSM, and it is only an optimisation to keep the start and the end of the file on disk, to avoid triggering tons of cache misses with commands like "file foo/*" or a tool like Nautilus or Windows Explorer browsing the directory. The purged window is stored by per object, OST object and MDT object. So, if several objects are purged, each object will store its own purged window. But the MDT object describing this file will store a special purged window which starts at the smallest unavailable bytes and ends at the first available one. The MDT purged window indicates "if you do I/O in this range, you're not sure the date are there." or "Outside of this area, I guarantee data are present." Maintain multiple purged windows will be an headache, with no real need I think. Moreover, people have asked for an OST-object based migration, even if I think whole file migration will be the most common case. > If that's the case, then it > the OST must keep track of purged objects, not ranges within an existing > object. Objects are not removed, only their datas. All metadata are kept. > If the MDT is tracking purged areas also, then there's a good potential > synergy here with a missing OST -- > If the missing OST's objects are marked as purged, then we can > potentially recover them automatically from > HSM... What do you call a "missing OST" ? A corrupt one ? A offline one? Unavailable? Where will you copy back the object data ? On another OST object ? With the purged window on each OST object and MDT and the file stripping info, we could easily restore the missing parts. > 4.2 How is a purge request recovered? For example, MDT says purge obj1 > from ost1, ost1 replies "ok", but then dies before it actually > does the purge. Reboots, doesn't know anything about purge request now, > but MDT has marked it as purged. The OST asynchronously acknowledges the purge when it is done. The MDT marks it purged only when it is really done. I will clarify this. > V2.1 How long does OST wait for completion? Is there a timeout? We > probably need a "no timeout if progress is being > made" kind of function - clients currently do this kind of thing with OSTs. I'm sure Lustre already has similar mechanisms for optimized timeout in this kind of situation we could reused here. What you describe is a good approach I think. > V2.2 No need to copy-in purged data on full-object-size writes. True. We could had such optimization. But this is only useful for small files or very widely stripped ones, doesn't it? Thanks for your comments. -- Aurelien Degremont CEA From adilger at sun.com Mon Feb 11 18:18:50 2008 From: adilger at sun.com (Andreas Dilger) Date: Mon, 11 Feb 2008 11:18:50 -0700 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47AC7B75.30603@cea.fr> References: <47AB2FA7.10205@Sun.COM> <47AC7B75.30603@cea.fr> Message-ID: <20080211181850.GA6552@webber.adilger.int> On Feb 08, 2008 16:55 +0100, Aurelien Degremont wrote: > But the touch could be problematic. Lustre gurus, is there another time > field we could use instead ? Should we add a > "last-modification-field-which-ignore-touch" ? Is this really a problem > is we use display a "touched" time ? In this case, we display what the > user set on the file, we suppose he did it in purpose. There was work done in ext4/ldiskfs to add a 64-bit "version" field to the on-disk inode, for use by lustre and NFSv4. In the ldiskfs case Lustre was free to store any information in this field it wanted. The planned use for this field is for "version based recovery" and it has the semantic that it is an increasing (though not necessarily sequential) version number that tracks any change to the file. This is stored in each inode on the MDT and each object on the OSTs. In ZFS I believe there is also a "last modified transaction group" (txg) number stored with each dnode that could be used in a similar manner. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Peter.Braam at Sun.COM Mon Feb 11 19:38:33 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Mon, 11 Feb 2008 12:38:33 -0700 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <20080211181850.GA6552@webber.adilger.int> References: <47AB2FA7.10205@Sun.COM> <47AC7B75.30603@cea.fr> <20080211181850.GA6552@webber.adilger.int> Message-ID: <47B0A439.5@sun.com> Versions are critical - we need them for multiple things, let's make sure we get exactly the right thing in ZFS also. - Peter - Andreas Dilger wrote: > On Feb 08, 2008 16:55 +0100, Aurelien Degremont wrote: > >> But the touch could be problematic. Lustre gurus, is there another time >> field we could use instead ? Should we add a >> "last-modification-field-which-ignore-touch" ? Is this really a problem >> is we use display a "touched" time ? In this case, we display what the >> user set on the file, we suppose he did it in purpose. >> > > There was work done in ext4/ldiskfs to add a 64-bit "version" field to > the on-disk inode, for use by lustre and NFSv4. In the ldiskfs case > Lustre was free to store any information in this field it wanted. The > planned use for this field is for "version based recovery" and it has > the semantic that it is an increasing (though not necessarily sequential) > version number that tracks any change to the file. This is stored in > each inode on the MDT and each object on the OSTs. > > In ZFS I believe there is also a "last modified transaction group" (txg) > number stored with each dnode that could be used in a similar manner. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nathan.Rutman at Sun.COM Mon Feb 11 20:33:03 2008 From: Nathan.Rutman at Sun.COM (Nathaniel Rutman) Date: Mon, 11 Feb 2008 12:33:03 -0800 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47B062BA.1070809@cea.fr> References: <47AAE307.9040305@cea.fr> <47ACC71C.7050808@sun.com> <47B062BA.1070809@cea.fr> Message-ID: <47B0B0FF.7060805@sun.com> Aurelien Degremont wrote: > Nathaniel Rutman a écrit : > >> 5.1 external storage list - is this to be stored on the MGS device or a >> separate device? If the coordinator lives on the MGS, why not it's >> storage as well? In any case, it should be possible to co-locate the >> coordinator on the MGS and used the MGS's storage device, in the same >> way that the MGS can currently co-locate with the MDT. >> How does the coordinator request activity from an agent? If the >> coordinator is the RPC server, then it's up to the agents to make >> requests; agents aren't listening for RPC requests themselves. >> > > Presently, it is never said that the coordinator will live on the MGS. > The Coordinator constrains are: > 1 - Must receive various migration requests from OST/MDT. > 2 - Should be able to communicate with Agents and asks them migrations. > 3 - Should store configuration and migration logs. > > I think #1 and #2 are two differents API. The coordinator is clearly a > RPC server for the first one. How #2 should be implemented is not so > clear. What would be be the "Lustre-way" here? > With userspace servers, presumably we have some way of passing LNET messages from kernel to userspace. We should probably still go through LNET for #2 in order to use the broadest range of network fabrics. So it could be the same or similar RPC. There is no "Lustre-way" for this area - we've never done this kind of thing before. > For #3, the few logs that will be backed up here are not huge, and it > surely could be colocated with another Target, but I'm not sure this > should be mandatory. This device should be available to several servers, > for failover like the other Targets. We could imagine having more than 1 > coordinator at long term. I'm not sure it is a good idea to stick it to > another target. > Not mandatory, but possible is nice. Minimize the number of required partitions. > >> 6.3 object ref should include version number. Also include checksum? >> > > For data coherency? Should we add a explicit checksum for those values > (stored in an EA) or used a possible backend feature (Can ZFS and > ldiskfs detect EA value corruption by themselves?) ? > ZFS can, ldiskfs cannot. Anyhow, it was just a thought. Doesn't hurt to allow space for it. > >> 2.1Archiving one Lustre file >> There should not be a cache miss when archiving a lustre file; perhaps >> open-by-fid is intended to bypass atime updates >> so that the file isn't marked as "recently accessed"? >> > > Transparent access - should this avoid modification of atime/mtime? > > I would say yes. > > >> 2.2Restoring a file >> "External ID" presumably contains all information required to retrieve >> the file - tape #, path name, etc? >> Once file is copied back, we should probably restore original ctime, >> mtime, atime - coordinator is storing this, correct? >> > > External ID is an opaque value manage by the archiving tool. If the HSM > can store a lot of metadata, only a ref is needed, if not, the tool is > responsible for storing all the data it needs. Anyway, this is totally > opaque for Lustre. > I hope the HSMs will not need so many data in this field. HPSS does not > need so many data, it uses its internal DB to store them. I suppose SAM > also. > What about restore of original ctime, mtime, atime? I think we must store it in the coordinator because we must work with all HSMs, and I think it is important to restore it. > >> IV2 - why not multiple purged windows? Seems like if you're going to >> purge 1 object out of a file, you might want to purge more. >> Specifically, it will probably be a common case to purge every object of >> a file from a particular OST. This is not contiguous in a >> striped file. >> I don't see any reason to purge anything smaller than an entire object >> on an OST - is there good reason for this? >> > > Multiple purged window is subtle. If you permit this feature, you could > technically have, in the worst case, one purged window per byte, and > this could be very huge to store. Do you think you will do several holes > in the same file? In which cases? > Like I said, I don't see any reason to purge anything smaller than a full object; I would in fact disallow purging of an arbitrary byte range, and only allow purging on full-object boundaries. > In fact, the more common case is to totally purge a file which have been > migrated on HSM, and it is only an optimisation to keep the start and > the end of the file on disk, to avoid triggering tons of cache misses > with commands like "file foo/*" or a tool like Nautilus or Windows > Explorer browsing the directory. > Again, since Lustre is optimized to work with 1MB chunks anyhow, I don't think it helps much to keep less than that in the beginning / end objects, so I would say just keep the first and last blocks instead. > The purged window is stored by per object, OST object and MDT object. > So, if several objects are purged, each object will store its own purged > window. But the MDT object describing this file will store a special > purged window which starts at the smallest unavailable bytes and ends at > the first available one. The MDT purged window indicates "if you do I/O > in this range, you're not sure the date are there." or "Outside of this > area, I guarantee data are present." > Maintain multiple purged windows will be an headache, with no real need > I think. > Moreover, people have asked for an OST-object based migration, even if I > think whole file migration will be the most common case. > > > > If that's the case, then it > >> the OST must keep track of purged objects, not ranges within an existing >> object. >> > > Objects are not removed, only their datas. All metadata are kept. > > >> If the MDT is tracking purged areas also, then there's a good potential >> synergy here with a missing OST -- >> If the missing OST's objects are marked as purged, then we can >> potentially recover them automatically from >> HSM... >> > > What do you call a "missing OST" ? A corrupt one ? A offline one? > Unavailable? > Yes. All of the above. Obviously we need to distinguish between "permanently gone" and "temporarily gone". > Where will you copy back the object data ? On another OST object ? > Yes. Some kind of recovery will take place to generate a new object on a different OST and we can restore the data there. > With the purged window on each OST object and MDT and the file stripping > info, we could easily restore the missing parts. > Exactly. This is why I say we should think about this now, to allow for this possibility. > >> 4.2 How is a purge request recovered? For example, MDT says purge obj1 >> from ost1, ost1 replies "ok", but then dies before it actually >> does the purge. Reboots, doesn't know anything about purge request now, >> but MDT has marked it as purged. >> > > The OST asynchronously acknowledges the purge when it is done. The MDT > marks it purged only when it is really done. I will clarify this. > > >> V2.1 How long does OST wait for completion? Is there a timeout? We >> probably need a "no timeout if progress is being >> made" kind of function - clients currently do this kind of thing with OSTs. >> > > I'm sure Lustre already has similar mechanisms for optimized timeout in > this kind of situation we could reused here. > What you describe is a good approach I think. > > >> V2.2 No need to copy-in purged data on full-object-size writes. >> > > True. We could had such optimization. But this is only useful for small > files or very widely stripped ones, doesn't it? > No, we very frequently write entire stripes (objects). Lustre clients can optimize for this. > > Thanks for your comments. > > From Ricardo.M.Correia at Sun.COM Mon Feb 11 21:11:43 2008 From: Ricardo.M.Correia at Sun.COM (Ricardo M. Correia) Date: Mon, 11 Feb 2008 21:11:43 +0000 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <20080211181850.GA6552@webber.adilger.int> References: <47AB2FA7.10205@Sun.COM> <47AC7B75.30603@cea.fr> <20080211181850.GA6552@webber.adilger.int> Message-ID: <1202764303.6391.42.camel@localhost> Hi, On Seg, 2008-02-11 at 11:18 -0700, Andreas Dilger wrote: > On Feb 08, 2008 16:55 +0100, Aurelien Degremont wrote: > > But the touch could be problematic. Lustre gurus, is there another time > > field we could use instead ? Should we add a > > "last-modification-field-which-ignore-touch" ? Is this really a problem > > is we use display a "touched" time ? In this case, we display what the > > user set on the file, we suppose he did it in purpose. > > (snip) > > In ZFS I believe there is also a "last modified transaction group" (txg) > number stored with each dnode that could be used in a similar manner. Hmm.. I think ZFS only has zp_gen in the dnode/znode, which is the txg of the file creation. We also cannot use the txg birth time of the block where the dnode is stored because a metadnode block holds several dnodes. I may be missing something here, but isn't the "ctime" the appropriate value to use here? Regards, Ricardo -- Ricardo Manuel Correia Lustre Engineering Sun Microsystems, Inc. Portugal Phone +351.214134023 / x58723 Mobile +351.912590825 Email Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From adilger at sun.com Mon Feb 11 21:39:56 2008 From: adilger at sun.com (Andreas Dilger) Date: Mon, 11 Feb 2008 14:39:56 -0700 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <1202764303.6391.42.camel@localhost> References: <47AB2FA7.10205@Sun.COM> <47AC7B75.30603@cea.fr> <20080211181850.GA6552@webber.adilger.int> <1202764303.6391.42.camel@localhost> Message-ID: <20080211213956.GK3029@webber.adilger.int> On Feb 11, 2008 21:11 +0000, Ricardo Correia wrote: > On Seg, 2008-02-11 at 11:18 -0700, Andreas Dilger wrote: > > > On Feb 08, 2008 16:55 +0100, Aurelien Degremont wrote: > > > But the touch could be problematic. Lustre gurus, is there another time > > > field we could use instead ? Should we add a > > > "last-modification-field-which-ignore-touch" ? Is this really a problem > > > is we use display a "touched" time ? In this case, we display what the > > > user set on the file, we suppose he did it in purpose. > > > > (snip) > > > > In ZFS I believe there is also a "last modified transaction group" (txg) > > number stored with each dnode that could be used in a similar manner. > > > Hmm.. I think ZFS only has zp_gen in the dnode/znode, which is the txg > of the file creation. We also cannot use the txg birth time of the block > where the dnode is stored because a metadnode block holds several > dnodes. > > I may be missing something here, but isn't the "ctime" the appropriate > value to use here? The problem with ctime (on Linux as well) is that it is possible for the system clock to go backward, whether due to ntp, or because the hardware clock is incorrect/reset, so it cannot be depended upon to be monotonically increasing for the life of the lustre filesystem. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Ricardo.M.Correia at Sun.COM Mon Feb 11 22:07:17 2008 From: Ricardo.M.Correia at Sun.COM (Ricardo M. Correia) Date: Mon, 11 Feb 2008 22:07:17 +0000 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <20080211213956.GK3029@webber.adilger.int> References: <47AB2FA7.10205@Sun.COM> <47AC7B75.30603@cea.fr> <20080211181850.GA6552@webber.adilger.int> <1202764303.6391.42.camel@localhost> <20080211213956.GK3029@webber.adilger.int> Message-ID: <1202767637.6391.60.camel@localhost> On Seg, 2008-02-11 at 14:39 -0700, Andreas Dilger wrote: > The problem with ctime (on Linux as well) is that it is possible for the > system clock to go backward, whether due to ntp, or because the hardware > clock is incorrect/reset, so it cannot be depended upon to be monotonically > increasing for the life of the lustre filesystem. Ok. In that case, we could either add a new 64-bit version field to the dnode (or znode) similar to the one in ldiskfs, or we could look at the birth time (txg nr) of all the block pointers in the dnode. Using txg numbers might not be very useful if an object is migrated from one storage device to another, but I have not read the HSM HLD so I'm not sure if this is a problem or not. Cheers, Ricardo -- Ricardo Manuel Correia Lustre Engineering Sun Microsystems, Inc. Portugal Phone +351.214134023 / x58723 Mobile +351.912590825 Email Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From Nathan.Rutman at Sun.COM Mon Feb 11 22:32:28 2008 From: Nathan.Rutman at Sun.COM (Nathaniel Rutman) Date: Mon, 11 Feb 2008 14:32:28 -0800 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <1202767637.6391.60.camel@localhost> References: <47AB2FA7.10205@Sun.COM> <47AC7B75.30603@cea.fr> <20080211181850.GA6552@webber.adilger.int> <1202764303.6391.42.camel@localhost> <20080211213956.GK3029@webber.adilger.int> <1202767637.6391.60.camel@localhost> Message-ID: <47B0CCFC.80102@sun.com> Ricardo M. Correia wrote: > > On Seg, 2008-02-11 at 14:39 -0700, Andreas Dilger wrote: >> The problem with ctime (on Linux as well) is that it is possible for the >> system clock to go backward, whether due to ntp, or because the hardware >> clock is incorrect/reset, so it cannot be depended upon to be monotonically >> increasing for the life of the lustre filesystem. >> > > Ok. In that case, we could either add a new 64-bit version field to > the dnode (or znode) similar to the one in ldiskfs, or we could look > at the birth time (txg nr) of all the block pointers in the dnode. > Using txg numbers might not be very useful if an object is migrated > from one storage device to another, but I have not read the HSM HLD so > I'm not sure if this is a problem or not. I'm missing the point of this discussion. Clearly we shouldn't/can't use ctime/mtime for anything internal to Lustre; that is what object versions are all about. Why are we talking about adding new fields or anything else? From Richard.Matthews at Sun.COM Mon Feb 11 22:46:07 2008 From: Richard.Matthews at Sun.COM (Rick Matthews) Date: Mon, 11 Feb 2008 16:46:07 -0600 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47B0CCFC.80102@sun.com> References: <47AB2FA7.10205@Sun.COM> <47AC7B75.30603@cea.fr> <20080211181850.GA6552@webber.adilger.int> <1202764303.6391.42.camel@localhost> <20080211213956.GK3029@webber.adilger.int> <1202767637.6391.60.camel@localhost> <47B0CCFC.80102@sun.com> Message-ID: <47B0D02F.6030803@Sun.COM> I'm probably responsible for opening this can of worms. I inferred from the HSM HLD that mtime was proposed to be used for state change, or version of the file/object. As the discussion bears out, mtime for this purpose would be a bad idea. A reliable way of detecting change is needed, and if it already exists withing Lustre, great!. What I think is far more significant is the involvement of the community on issues such as this. More folks examining (and critiquing) the details, the better. Nice to see such an active community. -- Nathaniel Rutman wrote: > Ricardo M. Correia wrote: >> >> On Seg, 2008-02-11 at 14:39 -0700, Andreas Dilger wrote: >>> The problem with ctime (on Linux as well) is that it is possible for >>> the >>> system clock to go backward, whether due to ntp, or because the >>> hardware >>> clock is incorrect/reset, so it cannot be depended upon to be >>> monotonically >>> increasing for the life of the lustre filesystem. >>> >> >> Ok. In that case, we could either add a new 64-bit version field to >> the dnode (or znode) similar to the one in ldiskfs, or we could look >> at the birth time (txg nr) of all the block pointers in the dnode. >> Using txg numbers might not be very useful if an object is migrated >> from one storage device to another, but I have not read the HSM HLD >> so I'm not sure if this is a problem or not. > I'm missing the point of this discussion. Clearly we shouldn't/can't > use ctime/mtime for anything internal to Lustre; that is what object > versions are all about. Why are we talking about adding new fields or > anything else? > > -- --------------------------------------------------------------------- Rick Matthews email: Rick.Matthews at sun.com Sun Microsystems, Inc. phone:+1(651) 554-1518 1270 Eagan Industrial Road phone(internal): 54418 Suite 160 fax: +1(651) 554-1540 Eagan, MN 55121-1231 USA main: +1(651) 554-1500 --------------------------------------------------------------------- From eeb at sun.com Mon Feb 11 23:07:42 2008 From: eeb at sun.com (Eric Barton) Date: Mon, 11 Feb 2008 23:07:42 +0000 Subject: [Lustre-devel] [RFC] parallel enqueue In-Reply-To: <47AC75E3.7010901@sun.com> References: <47AC75E3.7010901@sun.com> Message-ID: <01b801c86d02$e7a520a0$0281a8c0@ebpc> This sounds great! But are there any livelock issues? > -----Original Message----- > From: lustre-devel-bounces at lists.lustre.org > [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of > Alex Zhuravlev > Sent: 08 February 2008 3:32 PM > To: lustre-devel at lists.lustre.org > Subject: [Lustre-devel] [RFC] parallel enqueue > > Hi, > > in some cases (truncate, append) we still use serialized enqueue > when all locks have to be enqueued synchronously one by one. > > what if we could mark all locks issued by single client with some > unique tag (timestamp + nid?), then enqueue them all and then, in > case of conflict, in blocking ast handler compare tag of conflicting > lock with own tag, cancel our granted locks if our tag is greater > than tag of conflicting lock and enqueue them again? > > thanks, Alex > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Ricardo.M.Correia at Sun.COM Tue Feb 12 00:25:14 2008 From: Ricardo.M.Correia at Sun.COM (Ricardo M. Correia) Date: Tue, 12 Feb 2008 00:25:14 +0000 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47B0CCFC.80102@sun.com> References: <47AB2FA7.10205@Sun.COM> <47AC7B75.30603@cea.fr> <20080211181850.GA6552@webber.adilger.int> <1202764303.6391.42.camel@localhost> <20080211213956.GK3029@webber.adilger.int> <1202767637.6391.60.camel@localhost> <47B0CCFC.80102@sun.com> Message-ID: <1202775914.6391.77.camel@localhost> On Seg, 2008-02-11 at 14:32 -0800, Nathaniel Rutman wrote: > I'm missing the point of this discussion. Clearly we shouldn't/can't > use ctime/mtime for anything internal to Lustre; that is what object > versions are all about. Why are we talking about adding new fields or > anything else? If by object versions you are referring to the version field in the ldiskfs inodes that Andreas mentioned, then we need to add a similar field/attribute in ZFS. It seems that Andreas has already filed bug 14865 for this. Cheers, Ricardo -- Ricardo Manuel Correia Lustre Engineering Sun Microsystems, Inc. Portugal Phone +351.214134023 / x58723 Mobile +351.912590825 Email Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From adilger at sun.com Tue Feb 12 03:55:57 2008 From: adilger at sun.com (Andreas Dilger) Date: Mon, 11 Feb 2008 20:55:57 -0700 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47B0B0FF.7060805@sun.com> References: <47AAE307.9040305@cea.fr> <47ACC71C.7050808@sun.com> <47B062BA.1070809@cea.fr> <47B0B0FF.7060805@sun.com> Message-ID: <20080212035557.GP3029@webber.adilger.int> On Feb 11, 2008 12:33 -0800, Nathaniel Rutman wrote: > Aurelien Degremont wrote: > > Nathaniel Rutman a écrit : > >> IV2 - why not multiple purged windows? Seems like if you're going to > >> purge 1 object out of a file, you might want to purge more. > >> Specifically, it will probably be a common case to purge every object of > >> a file from a particular OST. This is not contiguous in a > >> striped file. > >> I don't see any reason to purge anything smaller than an entire object > >> on an OST - is there good reason for this? > > > > Multiple purged window is subtle. If you permit this feature, you could > > technically have, in the worst case, one purged window per byte, and > > this could be very huge to store. Do you think you will do several holes > > in the same file? In which cases? One issue is that if you are purging individual objects from a file your windows will be quite disjoint at the file level. That may not be a serious problem for applications that only look at the first and last chunks of a file. I can imagine use cases for extremely large files and limited-sized caches where there is a need to access only subsets of the file (i.e. the entire file cannot be resident at one time). That said, it may be this is too complex for the initial implementation. > Like I said, I don't see any reason to purge anything smaller than a > full object; I would in fact disallow purging of an arbitrary byte range, > and only allow purging on full-object boundaries. That is impractical, for the reasons that Aurelien mentioned - we want to avoid file re-staging for tools like "file" and GUIs that read the start/end of files to determine file type and icons. > > In fact, the more common case is to totally purge a file which have been > > migrated on HSM, and it is only an optimisation to keep the start and > > the end of the file on disk, to avoid triggering tons of cache misses > > with commands like "file foo/*" or a tool like Nautilus or Windows > > Explorer browsing the directory. > > Again, since Lustre is optimized to work with 1MB chunks anyhow, I don't > think it helps much to keep less than that in the beginning / end objects, > so I would say just keep the first and last blocks instead. What if file is N*1MB + 1 byte? We need to be able to keep something like 64kB for a windows icon, so having some arbitrary byte range seems reasonable. > > The purged window is stored by per object, OST object and MDT object. > > So, if several objects are purged, each object will store its own purged > > window. But the MDT object describing this file will store a special > > purged window which starts at the smallest unavailable bytes and ends at > > the first available one. I think this should read "ends at the highest range contiguous to the end of the file" or similar, or it will be misleading in the multi-object case. > >> the OST must keep track of purged objects, not ranges within an existing > >> object. > > > > Objects are not removed, only their datas. All metadata are kept. The one drawback with this approach is that it is not possible to HSM copy-in objects to a different OST than where they were originally stored. BUT... in conjunction with the migration tool it should be able to migrate an (empty) object from one OST to another before the copy-in from HSM, so long as there is no OST-specific data in the HSM identifier (i.e. the HSM label is truely opaque). > >> If the MDT is tracking purged areas also, then there's a good potential > >> synergy here with a missing OST -- > >> If the missing OST's objects are marked as purged, then we can > >> potentially recover them automatically from > >> HSM... > > > > What do you call a "missing OST" ? A corrupt one ? A offline one? > > Unavailable? > > Yes. All of the above. Obviously we need to distinguish between > "permanently gone" and "temporarily gone". I suppose this leads to a requirement to store the object in HSM so that it can be accessed just by the object FID+version. That would allow the OST to be restored from HSM even if the entire OST filesystem is lost, potentially modifying the FLDB to relocate the FID to a different OST. > > Where will you copy back the object data ? On another OST object ? > > Yes. Some kind of recovery will take place to generate a new object on > a different OST and we can restore the data there. > > With the purged window on each OST object and MDT and the file stripping > > info, we could easily restore the missing parts. > > Exactly. This is why I say we should think about this now, to allow for > this possibility. Right. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From eeb at sun.com Tue Feb 12 11:04:15 2008 From: eeb at sun.com (Eric Barton) Date: Tue, 12 Feb 2008 11:04:15 +0000 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <20080212035557.GP3029@webber.adilger.int> References: <47AAE307.9040305@cea.fr> <47ACC71C.7050808@sun.com> <47B062BA.1070809@cea.fr> <47B0B0FF.7060805@sun.com> <20080212035557.GP3029@webber.adilger.int> Message-ID: <026801c86d67$01a0beb0$0281a8c0@ebpc> Hi, Sorry if these questions duplicates previous debate. Have I understood correctly that the design allows individual objects within a lustre file (i.e. stripes?) to be purged independently? If so why is this needed? I would have thought that when you purge a file, you need only record the purged extent as an attribute of the whole lustre file and punch its stripes to free the space. Am I missing a use case? -- Cheers, Eric From Alex.Zhuravlev at Sun.COM Tue Feb 12 11:09:40 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Tue, 12 Feb 2008 14:09:40 +0300 Subject: [Lustre-devel] [RFC] parallel enqueue In-Reply-To: <01b801c86d02$e7a520a0$0281a8c0@ebpc> References: <47AC75E3.7010901@sun.com> <01b801c86d02$e7a520a0$0281a8c0@ebpc> Message-ID: <47B17E74.7030300@sun.com> well, the idea is to fallback to serialized enqueue once such collision is detected: this allow to avoid livelock problem and rpc storms. thanks, Alex Eric Barton wrote: > This sounds great! > > But are there any livelock issues? > >> -----Original Message----- >> From: lustre-devel-bounces at lists.lustre.org >> [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of >> Alex Zhuravlev >> Sent: 08 February 2008 3:32 PM >> To: lustre-devel at lists.lustre.org >> Subject: [Lustre-devel] [RFC] parallel enqueue >> >> Hi, >> >> in some cases (truncate, append) we still use serialized enqueue >> when all locks have to be enqueued synchronously one by one. >> >> what if we could mark all locks issued by single client with some >> unique tag (timestamp + nid?), then enqueue them all and then, in >> case of conflict, in blocking ast handler compare tag of conflicting >> lock with own tag, cancel our granted locks if our tag is greater >> than tag of conflicting lock and enqueue them again? >> >> thanks, Alex >> >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From aurelien.degremont at cea.fr Tue Feb 12 15:25:06 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Tue, 12 Feb 2008 16:25:06 +0100 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <026801c86d67$01a0beb0$0281a8c0@ebpc> References: <47AAE307.9040305@cea.fr> <47ACC71C.7050808@sun.com> <47B062BA.1070809@cea.fr> <47B0B0FF.7060805@sun.com> <20080212035557.GP3029@webber.adilger.int> <026801c86d67$01a0beb0$0281a8c0@ebpc> Message-ID: <47B1BA52.8040304@cea.fr> Eric Barton a écrit : > Hi, > > Sorry if these questions duplicates previous debate. > > Have I understood correctly that the design allows individual objects > within a lustre file (i.e. stripes?) to be purged independently? > > If so why is this needed? > > I would have thought that when you purge a file, you need only record > the purged extent as an attribute of the whole lustre file and punch > its stripes to free the space. Am I missing a use case? Since the beginning CFS required this feature. It seems a lab ask for it. I do not know who. Unfortunately we have no use case for what they want to do with this. I'm wondering if their need could not be met with other features like the internal Lustre migration... -- Aurelien Degremont CEA From aurelien.degremont at cea.fr Tue Feb 12 15:41:41 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Tue, 12 Feb 2008 16:41:41 +0100 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47B0D02F.6030803@Sun.COM> References: <47AB2FA7.10205@Sun.COM> <47AC7B75.30603@cea.fr> <20080211181850.GA6552@webber.adilger.int> <1202764303.6391.42.camel@localhost> <20080211213956.GK3029@webber.adilger.int> <1202767637.6391.60.camel@localhost> <47B0CCFC.80102@sun.com> <47B0D02F.6030803@Sun.COM> Message-ID: <47B1BE35.7060506@cea.fr> It is important to note that all comparisons and modifications are done at Lustre-object level: OST stripe object or MDT file object, each of those objects already has a version field, in the FID. This is the version inside the FID that we will use for all treatments. All purges are always requested for a specific FID. The mtime is stored only for information, for the users. It is simpler to display to the user: user$ list_hsm_copies ./foo Date ============ Feb 2 2006 Jun 18 2007 Jun 19 2007 than: user$ list_hsm_copies ./foo Version ============ 0x0012356 0x001a250 0x001a011 If the user "touched" the file sometime, he knew what he has done. Just the output will be different, but internaly, we manipulate Lustre FID and so we don't care of mtime. So the "version" in the backend is not a problem. We do not rely on the ldiskfs/zfs inode versioning. Aurelien Degremont Rick Matthews a écrit : > I'm probably responsible for opening this can of worms. I inferred from > the HSM HLD that > mtime was proposed to be used for state change, or version of the > file/object. As the discussion > bears out, mtime for this purpose would be a bad idea. A reliable way of > detecting change is > needed, and if it already exists withing Lustre, great!. > > What I think is far more significant is the involvement of the community > on issues > such as this. More folks examining (and critiquing) the details, the > better. > Nice to see such an active community. > -- > > Nathaniel Rutman wrote: >> Ricardo M. Correia wrote: >>> On Seg, 2008-02-11 at 14:39 -0700, Andreas Dilger wrote: >>>> The problem with ctime (on Linux as well) is that it is possible for >>>> the >>>> system clock to go backward, whether due to ntp, or because the >>>> hardware >>>> clock is incorrect/reset, so it cannot be depended upon to be >>>> monotonically >>>> increasing for the life of the lustre filesystem. >>>> >>> Ok. In that case, we could either add a new 64-bit version field to >>> the dnode (or znode) similar to the one in ldiskfs, or we could look >>> at the birth time (txg nr) of all the block pointers in the dnode. >>> Using txg numbers might not be very useful if an object is migrated >>> from one storage device to another, but I have not read the HSM HLD >>> so I'm not sure if this is a problem or not. >> I'm missing the point of this discussion. Clearly we shouldn't/can't >> use ctime/mtime for anything internal to Lustre; that is what object >> versions are all about. Why are we talking about adding new fields or >> anything else? >> >> > > -- Aurelien Degremont CEA From adilger at sun.com Tue Feb 12 17:23:21 2008 From: adilger at sun.com (Andreas Dilger) Date: Tue, 12 Feb 2008 10:23:21 -0700 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47B1BA52.8040304@cea.fr> References: <47AAE307.9040305@cea.fr> <47ACC71C.7050808@sun.com> <47B062BA.1070809@cea.fr> <47B0B0FF.7060805@sun.com> <20080212035557.GP3029@webber.adilger.int> <026801c86d67$01a0beb0$0281a8c0@ebpc> <47B1BA52.8040304@cea.fr> Message-ID: <20080212172321.GQ3029@webber.adilger.int> On Feb 12, 2008 16:25 +0100, Aurelien Degremont wrote: > Eric Barton a écrit : > > Sorry if these questions duplicates previous debate. > > > > Have I understood correctly that the design allows individual objects > > within a lustre file (i.e. stripes?) to be purged independently? > > > > If so why is this needed? > > > > I would have thought that when you purge a file, you need only record > > the purged extent as an attribute of the whole lustre file and punch > > its stripes to free the space. Am I missing a use case? > > Since the beginning CFS required this feature. It seems a lab ask for > it. I do not know who. Unfortunately we have no use case for what they > want to do with this. > I'm wondering if their need could not be met with other features like > the internal Lustre migration... That is my understanding also - I believe one of the Labs wanted this (to be able to do HSM on a per-stripe basis instead of a per-file basis). Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From eeb at sun.com Tue Feb 12 19:43:44 2008 From: eeb at sun.com (Eric Barton) Date: Tue, 12 Feb 2008 19:43:44 +0000 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <20080212172321.GQ3029@webber.adilger.int> References: <47AAE307.9040305@cea.fr> <47ACC71C.7050808@sun.com> <47B062BA.1070809@cea.fr> <47B0B0FF.7060805@sun.com> <20080212035557.GP3029@webber.adilger.int> <026801c86d67$01a0beb0$0281a8c0@ebpc> <47B1BA52.8040304@cea.fr> <20080212172321.GQ3029@webber.adilger.int> Message-ID: <032701c86daf$94333f30$0281a8c0@ebpc> Andreas, Is this requirement documented? I'd appreciate any pointers... > -----Original Message----- > From: Andreas.Dilger at Sun.COM [mailto:Andreas.Dilger at Sun.COM] > On Behalf Of Andreas Dilger > Sent: 12 February 2008 5:23 PM > To: Aurelien Degremont > Cc: Eric Barton; lustre-devel at lists.lustre.org > Subject: Re: [Lustre-devel] Lustre HSM HLD draft > > On Feb 12, 2008 16:25 +0100, Aurelien Degremont wrote: > > Eric Barton a écrit : > > > Sorry if these questions duplicates previous debate. > > > > > > Have I understood correctly that the design allows > individual objects > > > within a lustre file (i.e. stripes?) to be purged independently? > > > > > > If so why is this needed? > > > > > > I would have thought that when you purge a file, you need > only record > > > the purged extent as an attribute of the whole lustre > file and punch > > > its stripes to free the space. Am I missing a use case? > > > > Since the beginning CFS required this feature. It seems a > lab ask for > > it. I do not know who. Unfortunately we have no use case > for what they > > want to do with this. > > I'm wondering if their need could not be met with other > features like > > the internal Lustre migration... > > That is my understanding also - I believe one of the Labs wanted this > (to be able to do HSM on a per-stripe basis instead of a > per-file basis). > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > From Nathan.Rutman at Sun.COM Tue Feb 12 23:24:34 2008 From: Nathan.Rutman at Sun.COM (Nathaniel Rutman) Date: Tue, 12 Feb 2008 15:24:34 -0800 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <20080212172321.GQ3029@webber.adilger.int> References: <47AAE307.9040305@cea.fr> <47ACC71C.7050808@sun.com> <47B062BA.1070809@cea.fr> <47B0B0FF.7060805@sun.com> <20080212035557.GP3029@webber.adilger.int> <026801c86d67$01a0beb0$0281a8c0@ebpc> <47B1BA52.8040304@cea.fr> <20080212172321.GQ3029@webber.adilger.int> Message-ID: <47B22AB2.9010109@sun.com> Andreas Dilger wrote: > On Feb 12, 2008 16:25 +0100, Aurelien Degremont wrote: > >> Eric Barton a écrit : >> >>> Sorry if these questions duplicates previous debate. >>> >>> Have I understood correctly that the design allows individual objects >>> within a lustre file (i.e. stripes?) to be purged independently? >>> >>> If so why is this needed? >>> >>> I would have thought that when you purge a file, you need only record >>> the purged extent as an attribute of the whole lustre file and punch >>> its stripes to free the space. Am I missing a use case? >>> >> Since the beginning CFS required this feature. It seems a lab ask for >> it. I do not know who. Unfortunately we have no use case for what they >> want to do with this. >> I'm wondering if their need could not be met with other features like >> the internal Lustre migration... >> > > That is my understanding also - I believe one of the Labs wanted this > (to be able to do HSM on a per-stripe basis instead of a per-file basis). > This doesn't make any sense to me. Layouts may change; a stripe on one filesystem may not correspond to a stripe on a replica of the filesystem; exposing stripes to user apps is a bad idea. I'm going to propose what I think we need: 1. Punch a single, arbitrary byte range from the middle of a file (thus leaving beginning and end for file type, icons, filesize. 2. No other arbitrary punch patterns. 3. The punched range is stored on the MDT alone. 4. Once punched, the OST may forget about any fully-punched stripes it used to hold. 5. Clients must take a layout lock (CR) when they retrieve the layout from the MDT. If the MDT punches from the middle, it revokes the layout lock, and clients must re-enqueue it for further read/write on the file. The MDT is the sole keeper of the layout, and it must be protected by a lock. 6. Client access within a punched range results in an RPC to the MDT. The MDT decides where to put the restored data, organizes the restoration (via the coordinator), and rewrites the layout (under lock, of course). Client gets the new layout, and can contact the appropriate OST. From ron.rex at gmail.com Wed Feb 13 00:50:43 2008 From: ron.rex at gmail.com (Ron) Date: Tue, 12 Feb 2008 16:50:43 -0800 (PST) Subject: [Lustre-devel] lustre client goes wacky? Message-ID: Hi, I don't know if this is a bug or it's it's a misconfig or something else. What I have is: server = 1.6.4.1+vanilla 2.6.18.8 (mgs+2*ost+mdt all on a single server) clients = cvs.20080116+2.6.23.12 I mounted the server from several clients and several hours later noticed the top display below. dmesg show some lustre errors (also below).Can someone comment on what could be going on? Thanks, Ron top - 18:28:09 up 5 days, 3:36, 1 user, load average: 12.00, 12.00, 11.94 Tasks: 168 total, 13 running, 136 sleeping, 0 stopped, 19 zombie Cpu(s): 0.0% us, 37.5% sy, 0.0% ni, 62.5% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 16468196k total, 526828k used, 15941368k free, 42996k buffers Swap: 4192924k total, 0k used, 4192924k free, 294916k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1533 root 20 0 0 0 0 R 100 0.0 308:54.05 ll_cfg_requeue 32071 root 20 0 0 0 0 R 100 0.0 308:15.95 socknal_reaper 32073 root 20 0 0 0 0 R 100 0.0 308:48.90 ptlrpcd 1 root 20 0 4832 588 492 R 0 0.0 0:02.48 init 2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd Lustre: OBD class driver, info at clusterfs.com Lustre Version: 1.6.4.50 Build Version: b1_6-20080210103536- CHANGED-.usr.src.linux-2.6.23.12-2.6.23.12 Lustre: Added LNI 192.168.241.42 at tcp [8/256] Lustre: Accept secure, port 988 Lustre: Lustre Client File System; info at clusterfs.com Lustre: Binding irq 17 to CPU 0 with cmd: echo 1 > /proc/irq/17/ smp_affinity Lustre: MGC192.168.241.247 at tcp: Reactivating import Lustre: setting import datafs-OST0002_UUID INACTIVE by administrator request Lustre: datafs-OST0002-osc-ffff810241ad7800.osc: set parameter active=0 LustreError: 32181:0:(lov_obd.c:230:lov_connect_obd()) not connecting OSC datafs-OST0002_UUID; administratively disabled Lustre: Client datafs-client has started Lustre: Request x7684 sent from MGC192.168.241.247 at tcp to NID 192.168.241.247 at tcp 15s ago has timed out (limit 15s). LustreError: 166-1: MGC192.168.241.247 at tcp: Connection to service MGS via nid 192.168.241.247 at tcp was lost; in progress operations using this service will fail. LustreError: 32073:0:(import.c:212:ptlrpc_invalidate_import()) MGS: rc = -110 waiting for callback (1 != 0) LustreError: 32073:0:(import.c:216:ptlrpc_invalidate_import()) @@@ still on sending list req at ffff81040fa14600 x7684/t0 o400- >MGS at 192.168.241.247@tcp:26/25 lens 128/256 e 0 to 11 dl 1202843837 ref 1 fl Rpc:EXN/0/0 rc -4/0 Lustre: Request x7685 sent from datafs-MDT0000-mdc-ffff810241ad7800 to NID 192.168.241.247 at tcp 115s ago has timed out (limit 15s). Lustre: datafs-MDT0000-mdc-ffff810241ad7800: Connection to service datafs-MDT0000 via nid 192.168.241.247 at tcp was lost; in progress operations using this service will wait for recovery to complete. Lustre: MGC192.168.241.247 at tcp: Reactivating import Lustre: MGC192.168.241.247 at tcp: Connection restored to service MGS using nid 192.168.241.247 at tcp. LustreError: 32059:0:(events.c:116:reply_in_callback()) ASSERTION(ev- >mlength == lustre_msg_early_size()) failed LustreError: 32059:0:(tracefile.c:432:libcfs_assertion_failed()) LBUG Call Trace: [] :libcfs:lbug_with_loc+0x73/0xc0 [] :libcfs:libcfs_assertion_failed+0x54/0x60 [] :ptlrpc:reply_in_callback+0x426/0x430 [] :lnet:lnet_enq_event_locked+0xc5/0xf0 [] :lnet:lnet_finalize+0x1e5/0x270 [] :ksocklnd:ksocknal_process_receive+0x469/0xab0 [] :ksocklnd:ksocknal_tx_done+0x80/0x1e0 [] :ksocklnd:ksocknal_scheduler+0x12c/0x7e0 [] autoremove_wake_function+0x0/0x30 [] autoremove_wake_function+0x0/0x30 [] child_rip+0xa/0x12 [] :ksocklnd:ksocknal_scheduler+0x0/0x7e0 [] child_rip+0x0/0x12 LustreError: dumping log to /tmp/lustre-log.1202843942.32059 Lustre: Request x7707 sent from MGC192.168.241.247 at tcp to NID 192.168.241.247 at tcp 15s ago has timed out (limit 15s). Lustre: Skipped 2 previous similar messages From ron.rex at gmail.com Wed Feb 13 03:32:22 2008 From: ron.rex at gmail.com (Ron) Date: Tue, 12 Feb 2008 19:32:22 -0800 (PST) Subject: [Lustre-devel] lustrecvs pindate Message-ID: <2fadefbd-d9fe-422c-90ee-64803e193033@s37g2000prg.googlegroups.com> When I execute "./lustrecvs b1_6 20080116" I get: lustrecvs: updating lustrecvs cvs update: Updating . lustrecvs: Checking out build from HEAD cvs [checkout aborted]: Can't parse date/time: @1200441600 lustrecvs: build/buildcvs does not exist; not updating other modules. I've tried this with cvs versions 1.11.22 and 1.11.7. To get the "pindate" functionality to work, I had to comment out the conversion to the "@" format and the cvs -l update at the beginning. --Ron From Nathan.Rutman at Sun.COM Wed Feb 13 16:41:09 2008 From: Nathan.Rutman at Sun.COM (Nathaniel Rutman) Date: Wed, 13 Feb 2008 08:41:09 -0800 Subject: [Lustre-devel] lustre client goes wacky? In-Reply-To: References: Message-ID: <47B31DA5.6050003@sun.com> The clients you pulled from CVS have a feature called adaptive timeouts which apparently are having an issue with your 1.6.4.1 servers. Eric, can you make sure our interoperability is working? Moving this thread to lustre-discuss; devel is more for architecture/coding stuff. Ron wrote: > Hi, > I don't know if this is a bug or it's it's a misconfig or something > else. > > What I have is: > server = 1.6.4.1+vanilla 2.6.18.8 (mgs+2*ost+mdt all on a single > server) > clients = cvs.20080116+2.6.23.12 > > I mounted the server from several clients and several hours later > noticed the top display below. dmesg show some lustre errors (also > below).Can someone comment on what could be going on? > > Thanks, > Ron > > top - 18:28:09 up 5 days, 3:36, 1 user, load average: 12.00, 12.00, > 11.94 > Tasks: 168 total, 13 running, 136 sleeping, 0 stopped, 19 zombie > Cpu(s): 0.0% us, 37.5% sy, 0.0% ni, 62.5% id, 0.0% wa, 0.0% hi, > 0.0% si > Mem: 16468196k total, 526828k used, 15941368k free, 42996k > buffers > Swap: 4192924k total, 0k used, 4192924k free, 294916k > cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 1533 root 20 0 0 0 0 R 100 0.0 308:54.05 > ll_cfg_requeue > 32071 root 20 0 0 0 0 R 100 0.0 308:15.95 > socknal_reaper > 32073 root 20 0 0 0 0 R 100 0.0 308:48.90 > ptlrpcd > 1 root 20 0 4832 588 492 R 0 0.0 0:02.48 > init > 2 root 15 -5 0 0 0 S 0 0.0 0:00.00 > kthreadd > > > Lustre: OBD class driver, info at clusterfs.com > Lustre Version: 1.6.4.50 > Build Version: b1_6-20080210103536- > CHANGED-.usr.src.linux-2.6.23.12-2.6.23.12 > Lustre: Added LNI 192.168.241.42 at tcp [8/256] > Lustre: Accept secure, port 988 > Lustre: Lustre Client File System; info at clusterfs.com > Lustre: Binding irq 17 to CPU 0 with cmd: echo 1 > /proc/irq/17/ > smp_affinity > Lustre: MGC192.168.241.247 at tcp: Reactivating import > Lustre: setting import datafs-OST0002_UUID INACTIVE by administrator > request > Lustre: datafs-OST0002-osc-ffff810241ad7800.osc: set parameter > active=0 > LustreError: 32181:0:(lov_obd.c:230:lov_connect_obd()) not connecting > OSC datafs-OST0002_UUID; administratively disabled > Lustre: Client datafs-client has started > Lustre: Request x7684 sent from MGC192.168.241.247 at tcp to NID > 192.168.241.247 at tcp 15s ago has timed out (limit 15s). > LustreError: 166-1: MGC192.168.241.247 at tcp: Connection to service MGS > via nid 192.168.241.247 at tcp was lost; in progress operations using > this service will fail. > LustreError: 32073:0:(import.c:212:ptlrpc_invalidate_import()) MGS: rc > = -110 waiting for callback (1 != 0) > LustreError: 32073:0:(import.c:216:ptlrpc_invalidate_import()) @@@ > still on sending list req at ffff81040fa14600 x7684/t0 o400- > >> MGS at 192.168.241.247@tcp:26/25 lens 128/256 e 0 to 11 dl 1202843837 >> > ref 1 fl Rpc:EXN/0/0 rc -4/0 > Lustre: Request x7685 sent from datafs-MDT0000-mdc-ffff810241ad7800 to > NID 192.168.241.247 at tcp 115s ago has timed out (limit 15s). > Lustre: datafs-MDT0000-mdc-ffff810241ad7800: Connection to service > datafs-MDT0000 via nid 192.168.241.247 at tcp was lost; in progress > operations using this service will wait for recovery to complete. > Lustre: MGC192.168.241.247 at tcp: Reactivating import > Lustre: MGC192.168.241.247 at tcp: Connection restored to service MGS > using nid 192.168.241.247 at tcp. > LustreError: 32059:0:(events.c:116:reply_in_callback()) ASSERTION(ev- > >> mlength == lustre_msg_early_size()) failed >> > LustreError: 32059:0:(tracefile.c:432:libcfs_assertion_failed()) LBUG > > Call Trace: > [] :libcfs:lbug_with_loc+0x73/0xc0 > [] :libcfs:libcfs_assertion_failed+0x54/0x60 > [] :ptlrpc:reply_in_callback+0x426/0x430 > [] :lnet:lnet_enq_event_locked+0xc5/0xf0 > [] :lnet:lnet_finalize+0x1e5/0x270 > [] :ksocklnd:ksocknal_process_receive+0x469/0xab0 > [] :ksocklnd:ksocknal_tx_done+0x80/0x1e0 > [] :ksocklnd:ksocknal_scheduler+0x12c/0x7e0 > [] autoremove_wake_function+0x0/0x30 > [] autoremove_wake_function+0x0/0x30 > [] child_rip+0xa/0x12 > [] :ksocklnd:ksocknal_scheduler+0x0/0x7e0 > [] child_rip+0x0/0x12 > > LustreError: dumping log to /tmp/lustre-log.1202843942.32059 > Lustre: Request x7707 sent from MGC192.168.241.247 at tcp to NID > 192.168.241.247 at tcp 15s ago has timed out (limit 15s). > Lustre: Skipped 2 previous similar messages > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Nathan.Rutman at Sun.COM Wed Feb 13 18:23:50 2008 From: Nathan.Rutman at Sun.COM (Nathaniel Rutman) Date: Wed, 13 Feb 2008 10:23:50 -0800 Subject: [Lustre-devel] Global generic database Message-ID: <47B335B6.5010703@sun.com> The design of various new features in Lustre call for global (filesystem wide) databases, accessible from clients or other servers: A. pools - pool descriptions (pool #1 = OSTs 1-10,30-60), pool policies (all .jpg files to pool #1) B. filesets - fileset policies (log creates on fileset #1 to feed "foo") C. HSM - (aureleien - what was the use case here?) We've already implemented at least 2 of these: D. Fid Location Database - (is this done?) E. configuration parameters - stored in MGS llogs Rather than continue 1-off implementations, I think it's time we came up with a consistent, global, generic database mechanism for A-C as well as other future uses. Needs to be: 1. Fast. We need to cache database entries locally, which also means having them under locks. a. local caching b. locks 2. Generic. Store any kind of data, not limited to 8k page boundaries, etc. 3. Transactional. Power loss doesn't lead to inconsistent state. 4. Recoverable. Client changes are replayed if need be. 5. Remotely accessible, from a client or other servers. From canonrs at ornl.gov Wed Feb 13 20:35:02 2008 From: canonrs at ornl.gov (Canon, Richard Shane) Date: Wed, 13 Feb 2008 15:35:02 -0500 Subject: [Lustre-devel] Global generic database In-Reply-To: <47B335B6.5010703@sun.com> References: <47B335B6.5010703@sun.com> Message-ID: <537C6C0940C6C143AA46A88946B854170C1C9BB7@ORNLEXCHANGE.ornl.gov> JC mentioned this when we were talking about the Space Manager component in the HSM design. The Space Manager would have some type of policy language and the policies need to be stored some where. Another use case would be QoS policies when that starts to appear. --Shane -----Original Message----- From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Nathaniel Rutman Sent: Wednesday, February 13, 2008 1:24 PM To: lustre-devel at lists.lustre.org; aurelien.degremont at cea.fr; Eric.Barton at Sun.COM Subject: [Lustre-devel] Global generic database The design of various new features in Lustre call for global (filesystem wide) databases, accessible from clients or other servers: A. pools - pool descriptions (pool #1 = OSTs 1-10,30-60), pool policies (all .jpg files to pool #1) B. filesets - fileset policies (log creates on fileset #1 to feed "foo") C. HSM - (aureleien - what was the use case here?) We've already implemented at least 2 of these: D. Fid Location Database - (is this done?) E. configuration parameters - stored in MGS llogs Rather than continue 1-off implementations, I think it's time we came up with a consistent, global, generic database mechanism for A-C as well as other future uses. Needs to be: 1. Fast. We need to cache database entries locally, which also means having them under locks. a. local caching b. locks 2. Generic. Store any kind of data, not limited to 8k page boundaries, etc. 3. Transactional. Power loss doesn't lead to inconsistent state. 4. Recoverable. Client changes are replayed if need be. 5. Remotely accessible, from a client or other servers. _______________________________________________ Lustre-devel mailing list Lustre-devel at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-devel From aurelien.degremont at cea.fr Thu Feb 14 12:58:36 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Thu, 14 Feb 2008 13:58:36 +0100 Subject: [Lustre-devel] Global generic database In-Reply-To: <47B335B6.5010703@sun.com> References: <47B335B6.5010703@sun.com> Message-ID: <47B43AFC.5080709@cea.fr> Nathaniel Rutman a écrit : > The design of various new features in Lustre call for global (filesystem > wide) databases, accessible from > clients or other servers: > A. pools - pool descriptions (pool #1 = OSTs 1-10,30-60), pool policies > (all .jpg files to pool #1) > B. filesets - fileset policies (log creates on fileset #1 to feed "foo") > C. HSM - (aureleien - what was the use case here?) -pre-stage files when they are unused for 2 weeks -purge files when fs occupation reaches 95%. -do not purge file whose name matches ".local/*" and so on... There is a lot of possibilities with: -events. When doing a check? ie: fs occupations reaches N%, ... -filters. Which object is concerned? ie: file attributes (path, name, size, age, user, ...) -actions. What to do with them? ie: copy in, copy out, purge, ... -- Aurelien Degremont CEA From Peter.Braam at Sun.COM Thu Feb 14 14:57:16 2008 From: Peter.Braam at Sun.COM (Peter J Braam) Date: Thu, 14 Feb 2008 07:57:16 -0700 Subject: [Lustre-devel] Global generic database In-Reply-To: <47B335B6.5010703@sun.com> References: <47B335B6.5010703@sun.com> Message-ID: <47B456CC.6090400@sun.com> Hmm ... here are my thoughts. 1. The word scalable is missing below. 2. Any database that relates to file system policies and file system objects (HSM?) should be a separate mechanism coupled to the file system, so that you can pick up the server disks and the policies. 3. I think all updates to the database should be made on the server, and the use cases should be restricted (e.g. this is for relatively small databases). 4. Imho pools belong in the configuration log. 5. Fileset attributes belong with the file system (see 2) - either these are implemented as special directory files and/or EA's (does the design specify the purpose and items that need to be stored in databases?). Hmm, so can we revisit why we need a new database mechanism? - Peter - Nathaniel Rutman wrote: > The design of various new features in Lustre call for global (filesystem > wide) databases, accessible from > clients or other servers: > A. pools - pool descriptions (pool #1 = OSTs 1-10,30-60), pool policies > (all .jpg files to pool #1) > B. filesets - fileset policies (log creates on fileset #1 to feed "foo") > C. HSM - (aureleien - what was the use case here?) > > We've already implemented at least 2 of these: > D. Fid Location Database - (is this done?) > E. configuration parameters - stored in MGS llogs > > Rather than continue 1-off implementations, I think it's time we came up > with a consistent, > global, generic database mechanism for A-C as well as other future uses. > Needs to be: > 1. Fast. We need to cache database entries locally, which also means > having them under locks. > a. local caching > b. locks > 2. Generic. Store any kind of data, not limited to 8k page boundaries, etc. > 3. Transactional. Power loss doesn't lead to inconsistent state. > 4. Recoverable. Client changes are replayed if need be. > 5. Remotely accessible, from a client or other servers. > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Nathan.Rutman at Sun.COM Thu Feb 14 19:56:21 2008 From: Nathan.Rutman at Sun.COM (Nathaniel Rutman) Date: Thu, 14 Feb 2008 11:56:21 -0800 Subject: [Lustre-devel] Global generic database In-Reply-To: <47B456CC.6090400@sun.com> References: <47B335B6.5010703@sun.com> <47B456CC.6090400@sun.com> Message-ID: <47B49CE5.50608@sun.com> Peter J Braam wrote: > Hmm ... here are my thoughts. > > 1. The word scalable is missing below. That is implicit in any Lustre design :) > > 2. Any database that relates to file system policies and file system > objects (HSM?) should be a separate mechanism coupled to the file > system, so that you can pick up the server disks and the policies. What I am trying to avoid is multiple mechanisms to reduce the number of database implementations we have to write/maintain. > > 3. I think all updates to the database should be made on the server, > and the use cases should be restricted (e.g. this is for relatively > small databases). Maybe updates can only be made on the server, but the data needs to be readable from anywhere. > > 4. Imho pools belong in the configuration log. Pool definitions can easily be put in the configuration logs - but pool policies can be complex ("all .mov files greater than 10GB go to pool 7") and malleable - configuration logs are not easily accessible, not random access (config log records are arbitrary size, so we must walk the file from the beginning to find a record). If they grow too big performance will suffer. > 5. Fileset attributes belong with the file system (see 2) - either > these are implemented as special directory files and/or EA's (does the > design specify the purpose and items that need to be stored in > databases?). Fileset membership is stored with the filesystem (EAs), but fileset policies may again be larger, complex entities that should probably be stored once in a central database, and looked up as needed. For the 10,000 fileset case, clearly we don't want to read in 10,000 fileset policies from the config log at startup; they should be loaded on-demand as needed. > > Hmm, so can we revisit why we need a new database mechanism? > > - Peter - > > > > Nathaniel Rutman wrote: >> The design of various new features in Lustre call for global >> (filesystem wide) databases, accessible from >> clients or other servers: >> A. pools - pool descriptions (pool #1 = OSTs 1-10,30-60), pool >> policies (all .jpg files to pool #1) >> B. filesets - fileset policies (log creates on fileset #1 to feed "foo") >> C. HSM - (aureleien - what was the use case here?) Space manager policies >> >> We've already implemented at least 2 of these: >> D. Fid Location Database - (is this done?) >> E. configuration parameters - stored in MGS llogs >> >> Rather than continue 1-off implementations, I think it's time we came >> up with a consistent, >> global, generic database mechanism for A-C as well as other future uses. >> Needs to be: >> 1. Fast. We need to cache database entries locally, which also means >> having them under locks. >> a. local caching >> b. locks >> 2. Generic. Store any kind of data, not limited to 8k page >> boundaries, etc. >> 3. Transactional. Power loss doesn't lead to inconsistent state. >> 4. Recoverable. Client changes are replayed if need be. >> 5. Remotely accessible, from a client or other servers. >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> From Peter.Braam at Sun.COM Fri Feb 15 03:32:16 2008 From: Peter.Braam at Sun.COM (Peter J Braam) Date: Thu, 14 Feb 2008 20:32:16 -0700 Subject: [Lustre-devel] Global generic database In-Reply-To: <47B49CE5.50608@sun.com> References: <47B335B6.5010703@sun.com> <47B456CC.6090400@sun.com> <47B49CE5.50608@sun.com> Message-ID: <47B507C0.5010203@sun.com> Nathaniel Rutman wrote: > Peter J Braam wrote: >> Hmm ... here are my thoughts. >> >> 1. The word scalable is missing below. > That is implicit in any Lustre design :) >> >> 2. Any database that relates to file system policies and file system >> objects (HSM?) should be a separate mechanism coupled to the file >> system, so that you can pick up the server disks and the policies. > What I am trying to avoid is multiple mechanisms to reduce the number > of database implementations we have to write/maintain. >> >> 3. I think all updates to the database should be made on the server, >> and the use cases should be restricted (e.g. this is for relatively >> small databases). > Maybe updates can only be made on the server, but the data needs to be > readable from anywhere. >> >> 4. Imho pools belong in the configuration log. > Pool definitions can easily be put in the configuration logs - but > pool policies can be complex ("all .mov files greater than 10GB go > to pool 7") and malleable - configuration logs are not easily > accessible, not random access (config log records are arbitrary size, > so we must walk the file from the beginning to find a record). If > they grow too big performance will suffer. >> 5. Fileset attributes belong with the file system (see 2) - either >> these are implemented as special directory files and/or EA's (does >> the design specify the purpose and items that need to be stored in >> databases?). > Fileset membership is stored with the filesystem (EAs), but fileset > policies may again be larger, complex entities that should probably be > stored once in a central database, and looked up as needed. For the > 10,000 fileset case, clearly we don't want to read in 10,000 fileset > policies fro > the config log at startup; they should be loaded on-demand as needed. They need to be in the filesystem, not on the management server. - Peter - >> >> Hmm, so can we revisit why we need a new database mechanism? >> >> - Peter - >> >> >> >> Nathaniel Rutman wrote: >>> The design of various new features in Lustre call for global >>> (filesystem wide) databases, accessible from >>> clients or other servers: >>> A. pools - pool descriptions (pool #1 = OSTs 1-10,30-60), pool >>> policies (all .jpg files to pool #1) >>> B. filesets - fileset policies (log creates on fileset #1 to feed >>> "foo") >>> C. HSM - (aureleien - what was the use case here?) > Space manager policies >>> >>> We've already implemented at least 2 of these: >>> D. Fid Location Database - (is this done?) >>> E. configuration parameters - stored in MGS llogs >>> >>> Rather than continue 1-off implementations, I think it's time we >>> came up with a consistent, >>> global, generic database mechanism for A-C as well as other future >>> uses. >>> Needs to be: >>> 1. Fast. We need to cache database entries locally, which also means >>> having them under locks. >>> a. local caching >>> b. locks >>> 2. Generic. Store any kind of data, not limited to 8k page >>> boundaries, etc. >>> 3. Transactional. Power loss doesn't lead to inconsistent state. >>> 4. Recoverable. Client changes are replayed if need be. >>> 5. Remotely accessible, from a client or other servers. >>> _______________________________________________ >>> Lustre-devel mailing list >>> Lustre-devel at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>> > From Vladimir.Saveliev at Sun.COM Fri Feb 15 12:54:22 2008 From: Vladimir.Saveliev at Sun.COM (Vladimir V. Saveliev) Date: Fri, 15 Feb 2008 14:54:22 +0200 Subject: [Lustre-devel] subtree locks and path re-validation avoidance Message-ID: <1203080062.19279.91.camel@linux.site> Hello On last rabbit meeting in Moscow we agreed, that with subtree locks (http://arch.lustre.org/index.php?title=Sub_Tree_Locks) any use of ".." on client requires path re-validation. The example shows the details: 1. A client C1 holds ordinary lock on an object O1 (it did chdir(/a/b/c/d/e), O1 is inode of /a/b/c/d/e). C1 is idle now. 2. Another client C2 does ls -ld /a/b/c/d/e, MD server sends a BAST to C1 and C1 cancels the lock of O1. 3. C2 is not interested anymore in O1, so it drops the lock. 4. Yet another client C3 acquires subtree lock on /a/b and caches and possibly changes (if under WBC) objects under /a/b including /a/b/c/d/e (the object O1). The key issue is that MDS neither remembers about O1 on C1 nor keeps information about objects cached by a client under a subtree lock. 5. Now C1 continues with stat(``.''). It sees that the lock on O1 is canceled, so it goes to MD server and acquires the lock on O1. Now we have: uptodate O1 is on C3; MDS has a request for O1 from C1 and MDS can not easily deterimine whether O1 is under any subtree lock. In order to find whether the lock conflict exists we need to have a special procedure. It is referred to as path re-validation. The main thing to be done on path re-validation is to look for above subtree lock. While it is probably doable, the path re-validation is not going to be very efficient (especially in case of CMD). I can provide more details if necessary. However, it looks like it is possible to avoid having to do path re-validation completely. The problem appears when clients request locks on objects directly, without doing downward lookup through a directory structure. This happens, for example, when clients access directly components of current working directories (CWDs). If a client cancels locks on such objects (either due to a BAST or voluntary) - it has to go through the path re-validation later. Objects to which a client may access directly appear in result of normal downward lookup. Therefore, they were locked, and their locks can be canceled. That is the point where we can take care about future accesses without re-validation. On canceling a lock of directly accessible object we have to inform DLM that the ordinary locking has to be used for that object. That will prevent the object from getting cached under a subtree lock. The problem with this schema is to determine which objects are directly accessible. But wouldn't solving it be worth doing given that it may help to avoid path re-validation deal. Any comments are welcome. Best regards, Vladimir From Alexander.Zarochentsev at Sun.COM Fri Feb 15 17:50:27 2008 From: Alexander.Zarochentsev at Sun.COM (Alexander Zarochentsev) Date: Fri, 15 Feb 2008 20:50:27 +0300 Subject: [Lustre-devel] Global generic database In-Reply-To: <47B335B6.5010703@sun.com> References: <47B335B6.5010703@sun.com> Message-ID: <200802152050.28398.alexander.zarochentsev@sun.com> Hello, On 13 February 2008 21:23:50 Nathaniel Rutman wrote: > The design of various new features in Lustre call for global > (filesystem wide) databases, accessible from > clients or other servers: > A. pools - pool descriptions (pool #1 = OSTs 1-10,30-60), pool > policies (all .jpg files to pool #1) > B. filesets - fileset policies (log creates on fileset #1 to feed > "foo") C. HSM - (aureleien - what was the use case here?) > > We've already implemented at least 2 of these: > D. Fid Location Database - (is this done?) > E. configuration parameters - stored in MGS llogs Сould be the same (file?) interface used for anything Lustre-specific under /proc? Anyway we need a /proc replacement for use-level lustre servers. > Rather than continue 1-off implementations, I think it's time we came > up with a consistent, > global, generic database mechanism for A-C as well as other future > uses. Needs to be: > 1. Fast. We need to cache database entries locally, which also means > having them under locks. > a. local caching > b. locks > 2. Generic. Store any kind of data, not limited to 8k page > boundaries, etc. 3. Transactional. Power loss doesn't lead to > inconsistent state. 4. Recoverable. Client changes are replayed if > need be. > 5. Remotely accessible, from a client or other servers. > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel Thanks, Zam. From Nikita.Danilov at Sun.COM Fri Feb 15 20:40:24 2008 From: Nikita.Danilov at Sun.COM (Nikita Danilov) Date: Fri, 15 Feb 2008 23:40:24 +0300 Subject: [Lustre-devel] Global generic database In-Reply-To: <47B456CC.6090400@sun.com> References: <47B335B6.5010703@sun.com> <47B456CC.6090400@sun.com> Message-ID: <18357.63672.838319.62694@gargle.gargle.HOWL> Peter J Braam writes: > Hmm ... here are my thoughts. Can we use our existing directory/lookup/read/write mechanism to implement this database? That is, imagine, that clients somehow get special fid (DB_FID), representing directory not visible through the normal namespace (this can be implemented as a /DB directory on the MDS local file-system, alongside the /ROOT directory). Typical use of that would be something along the lines of int db_value_get(const char *key, void *buf, size_t count) { static struct dt_object *topdir = object_by_fid(DB_FID); fd = lookup(topdir, key); read(fd, buf, count); close(fd); } db_value_get("filesets.FOO.policy", buf, BUFSIZE); db_value_get("pools.BAR.width", &pool_width, sizeof pool_width); etc. Main advantage of this approach is of course that all code is already here, moreover... > > 1. The word scalable is missing below. fixed through the standards means: CMD, placement policies, split directories, pdirops-locking, > > 2. Any database that relates to file system policies and file system > objects (HSM?) should be a separate mechanism coupled to the file > system, so that you can pick up the server disks and the policies. achieved automatically (if I understand the issue correctly), > > 3. I think all updates to the database should be made on the server, and > the use cases should be restricted (e.g. this is for relatively small > databases). > > 4. Imho pools belong in the configuration log. > > 5. Fileset attributes belong with the file system (see 2) - either these > are implemented as special directory files and/or EA's (does the design > specify the purpose and items that need to be stored in databases?). > [...] > > Needs to be: > > 1. Fast. We need to cache database entries locally, which also means hopefully fast. :-) Caching is already here, > > having them under locks. > > a. local caching already here, > > b. locks already here, > > 2. Generic. Store any kind of data, not limited to 8k page boundaries, etc. already here, > > 3. Transactional. Power loss doesn't lead to inconsistent state. already here, > > 4. Recoverable. Client changes are replayed if need be. already here, > > 5. Remotely accessible, from a client or other servers. already here. Plus, we can allow clients to mount DB_FID as a separate file system, so that usual tools can be used to maintain the database. Nikita. From adilger at sun.com Sat Feb 16 07:40:46 2008 From: adilger at sun.com (Andreas Dilger) Date: Sat, 16 Feb 2008 00:40:46 -0700 Subject: [Lustre-devel] Global generic database In-Reply-To: <200802152050.28398.alexander.zarochentsev@sun.com> References: <47B335B6.5010703@sun.com> <200802152050.28398.alexander.zarochentsev@sun.com> Message-ID: <20080216074046.GD3029@webber.adilger.int> On Feb 15, 2008 20:50 +0300, Alexander Zarochentsev wrote: > On 13 February 2008 21:23:50 Nathaniel Rutman wrote: > > The design of various new features in Lustre call for global > > (filesystem wide) databases, accessible from > > clients or other servers: > > A. pools - pool descriptions (pool #1 = OSTs 1-10,30-60), pool > > policies (all .jpg files to pool #1) > > B. filesets - fileset policies (log creates on fileset #1 to feed > > "foo") C. HSM - (aureleien - what was the use case here?) > > > > We've already implemented at least 2 of these: > > D. Fid Location Database - (is this done?) > > E. configuration parameters - stored in MGS llogs > > Сould be the same (file?) interface used for anything Lustre-specific > under /proc? Anyway we need a /proc replacement for use-level lustre > servers. There won't immediately be a /proc replacement for uOSS. Instead, there will be a new "lctl {get,set}_param" command that reads/writes the same proc entries. At some later time we will make a .lustre/proc directory which will allow access to the files currently in /proc, possibly also allowing access to /proc values on all lustre nodes. That hasn't been designed yet, but would definitely be convenient. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Alexey.Lyashkov at Sun.COM Sun Feb 17 11:27:05 2008 From: Alexey.Lyashkov at Sun.COM (Alex Lyashkov) Date: Sun, 17 Feb 2008 13:27:05 +0200 Subject: [Lustre-devel] Global generic database In-Reply-To: <20080216074046.GD3029@webber.adilger.int> References: <47B335B6.5010703@sun.com> <200802152050.28398.alexander.zarochentsev@sun.com> <20080216074046.GD3029@webber.adilger.int> Message-ID: <1203247625.2203.23.camel@bear.shadowland> On Sat, 2008-02-16 at 00:40 -0700, Andreas Dilger wrote: > > There won't immediately be a /proc replacement for uOSS. Instead, > there > will be a new "lctl {get,set}_param" command that reads/writes the > same proc entries. At some later time we will make a .lustre/proc > directory which will allow access to the files currently in /proc, > possibly also allowing access to /proc values on all lustre nodes. > That hasn't been designed yet, but would definitely be convenient. I think we can put all parameters into sysctl tree and write simple wraper for userland tools. Which add functions for register/list/access to sysctl node. -- Alex Lyashkov Lustre Group, Sun Microsystems From Nikita.Danilov at Sun.COM Mon Feb 18 18:49:13 2008 From: Nikita.Danilov at Sun.COM (Nikita Danilov) Date: Mon, 18 Feb 2008 21:49:13 +0300 Subject: [Lustre-devel] WBC HLD: next round Message-ID: <18361.54057.703866.863905@gargle.gargle.HOWL> Hello, please take a look at the next version of the High Level Design specification for the Write-Back Cache subsystem, that has Use Cases and Logical Specification sections fleshed out. I'd appreciate any comments, especially on the topics marked with "Q!" on a margin. Peter, can you please take a look at the description of a security hole on page 10? Nikita. -------------- next part -------------- A non-text attachment was scrubbed... Name: wbc-hld.pdf Type: application/pdf Size: 128230 bytes Desc: not available URL: From canonrs at ornl.gov Mon Feb 18 21:51:35 2008 From: canonrs at ornl.gov (Canon, Richard Shane) Date: Mon, 18 Feb 2008 16:51:35 -0500 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47AAE307.9040305@cea.fr> References: <47AAE307.9040305@cea.fr> Message-ID: <537C6C0940C6C143AA46A88946B854170C2EFE91@ORNLEXCHANGE.ornl.gov> Aurelien and JC, Sorry that my feedback is late. Here are my questions/remarks. General * Any thought on how quotas will be handled? Coordinator * 3.4 - I was curious what the precise use case was that was driving this? I don't disagree with it, but I was curious for more background * 3.7.1 - The coordinator could become a scaling bottleneck. We should think about how this will be scaled in the future * 4.1 - Does the coordinator store the ext obj id or does the agent * 4.3 item 2 - This looks like the coordinator could become a bottle neck for unlinks and slow down performance. Could this be put in some type of async queue to be processed later (or some type of attic space)? Use Cases * 2.3 (Use cases) - I'm really keen on this feature. I think it is very important in order to make small file performance work well. Unfortunately, it isn't clear how the file list gets communicated to the archive tool. The coordinator and agent seem to only take one file at a time. So how would this work exactly? * 2.4 - The copy tool should be allowed to preemptively restage files. I think this will work with the design, but we should make sure of this. This would be useful for restaging a whole tar file versus doing things piece-meal. Part IV 2 EAs - I'm worried that the EA list could get huge for holes. 3.2 -item 3 - Who insures a file is archived before punches are made? 3.3 - Another use case... The user checks to see if a file has been archived. Also, someone earlier made the point about the archive tool being able to reorder request. This is really important since an archival system wants to know all the files being restaged in order to order tape mounts and reads. Thanks for taking the lead on this. It looks like there is a lot of interest in it. --Shane -----Original Message----- From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of DEGREMONT Aurelien Sent: Thursday, February 07, 2008 5:53 AM To: lustre-devel at lists.lustre.org Subject: [Lustre-devel] Lustre HSM HLD draft Hello Here is a first draft for comments of the Lustre HSM HLD. It is intended to be a support for further analyzes and comments from CFS/Sun. The document covers the main parts of the HSM features but some elements are still lacking. The policy management and the space manager will be describe later. Let us know your comments and ideas about it. Regards, Aurelien Degremont CEA From Yury.Umanets at Sun.COM Mon Feb 18 21:57:04 2008 From: Yury.Umanets at Sun.COM (Yuriy Umanets) Date: Mon, 18 Feb 2008 23:57:04 +0200 Subject: [Lustre-devel] Global generic database In-Reply-To: <47B335B6.5010703@sun.com> References: <47B335B6.5010703@sun.com> Message-ID: <47B9FF30.6040409@sun.com> Nathaniel Rutman wrote: > The design of various new features in Lustre call for global (filesystem > wide) databases, accessible from > clients or other servers: > A. pools - pool descriptions (pool #1 = OSTs 1-10,30-60), pool policies > (all .jpg files to pool #1) > B. filesets - fileset policies (log creates on fileset #1 to feed "foo") > C. HSM - (aureleien - what was the use case here?) > > We've already implemented at least 2 of these: > D. Fid Location Database - (is this done?) > This is basic service in new MDS stack. Though its implementation lacks DHT support as it was designed originally. Currently it uses round-robin as a policy for spreading parts of FLD over MDS nodes in a cluster. Another issue is that, originally intention was to make whole Lustre FLD aware but now only MDS nodes store parts of FLD as only MDS uses new fids. Thanks. -- umka From yok at shibatsu.co.jp Tue Feb 19 03:08:21 2008 From: yok at shibatsu.co.jp (yokoyama) Date: Tue, 19 Feb 2008 12:08:21 +0900 Subject: [Lustre-devel] Thank you Message-ID: <47BA4825.3090506@shibatsu.co.jp> Thank you very much for your mailng many report for Lustre to me long time. But I will retire Shinatsu coprporation next month. And please stop these mails. From Alex.Zhuravlev at Sun.COM Tue Feb 19 09:48:55 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Tue, 19 Feb 2008 12:48:55 +0300 Subject: [Lustre-devel] storing SOM epoch in EA Message-ID: <47BAA607.1000600@sun.com> Good day, some time ago we discussed that it would be very helpful to store epoch in inode on mds. the perfect solution could be to store epoch in old inode body, but there is no much space for this in the body and with DMU we'll have this problem again. given the minimal inode size we use on MDS is 512 bytes, we can store upto 13 stripes in the body. larger EAs go to a dedicated block. if we add 8 byte epoch, then we can store upto 12 stripes in the body. so, epoch stored in EA affects only files with exactly 13 stripes. files with different stripes are unaffected at all. couple lesser concerns are: 1) cpu usage 2) epoch on old filesystem with insufficient inode space any objections to use EA to store SOM epoch? thanks, Alex From Yury.Umanets at Sun.COM Tue Feb 19 10:28:15 2008 From: Yury.Umanets at Sun.COM (Yuriy Umanets) Date: Tue, 19 Feb 2008 12:28:15 +0200 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAA607.1000600@sun.com> References: <47BAA607.1000600@sun.com> Message-ID: <47BAAF3F.6030301@sun.com> Alex Zhuravlev wrote: > Good day, > > some time ago we discussed that it would be very helpful to > store epoch in inode on mds. the perfect solution could be > to store epoch in old inode body, but there is no much space > for this in the body and with DMU we'll have this problem > again. > > given the minimal inode size we use on MDS is 512 bytes, we > can store upto 13 stripes in the body. larger EAs go to a > dedicated block. if we add 8 byte epoch, then we can store > upto 12 stripes in the body. so, epoch stored in EA affects > only files with exactly 13 stripes. files with different > stripes are unaffected at all. > > couple lesser concerns are: > 1) cpu usage > 2) epoch on old filesystem with insufficient inode space > > any objections to use EA to store SOM epoch? > hi! Can we use IAM for storing epoch? It is fast and does not have such strong size limitations. We could make "epoch" index in mkfs time (like it is done for existing indexes now) and use object's fid as a key and epoch as value. Thanks. -- umka From Alex.Zhuravlev at Sun.COM Tue Feb 19 10:30:27 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Tue, 19 Feb 2008 13:30:27 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAAF3F.6030301@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> Message-ID: <47BAAFC3.90608@sun.com> hmm. not sure I got it. epoch is per-inode. and we don't need >1 epoch for any inode. thanks, Alex Yuriy Umanets wrote: > Can we use IAM for storing epoch? It is fast and does not have such > strong size limitations. We could make "epoch" index in mkfs time (like > it is done for existing indexes now) and use object's fid as a key and > epoch as value. > > Thanks. > From Yury.Umanets at Sun.COM Tue Feb 19 10:38:39 2008 From: Yury.Umanets at Sun.COM (Yuriy Umanets) Date: Tue, 19 Feb 2008 12:38:39 +0200 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAAFC3.90608@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <47BAAFC3.90608@sun.com> Message-ID: <47BAB1AF.2050405@sun.com> Alex Zhuravlev wrote: > hmm. not sure I got it. epoch is per-inode. and we don't need >1 epoch > for > any inode. > Yes, right. We will not have few epochs for the inode. I think we need Nikita here as he is author of IAM and may help us. In HEAD we have got OI (Object Index) which purpose is to map object fids into object store cookies (inode + generation). Fid here is the key and inode store info is value. We have only one such mapping entry for any inode. I proposed to have similar mapping, but store SOM epoch for the inode same way. Use fid as key and epoch as value. Nikita, is this correct using of IAM? Thanks. > thanks, Alex > > Yuriy Umanets wrote: >> Can we use IAM for storing epoch? It is fast and does not have such >> strong size limitations. We could make "epoch" index in mkfs time >> (like it is done for existing indexes now) and use object's fid as a >> key and epoch as value. >> >> Thanks. >> > -- umka From vitaly at sun.com Tue Feb 19 10:59:47 2008 From: vitaly at sun.com (Vitaly Fertman) Date: Tue, 19 Feb 2008 13:59:47 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAAF3F.6030301@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> Message-ID: <200802191359.47379.vitaly@sun.com> > hi! > > Can we use IAM for storing epoch? It is fast and does not have such > strong size limitations. there are no size limitations, EA can be stored in a separate block, we just want to minimize IO. > We could make "epoch" index in mkfs time (like > it is done for existing indexes now) and use object's fid as a key and > epoch as value. this looks like it will double IO/seeks for each inode. -- Vitaly From Yury.Umanets at Sun.COM Tue Feb 19 11:11:30 2008 From: Yury.Umanets at Sun.COM (Yuriy Umanets) Date: Tue, 19 Feb 2008 13:11:30 +0200 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <200802191359.47379.vitaly@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> Message-ID: <47BAB962.8010901@sun.com> Vitaly Fertman wrote: >> hi! >> >> Can we use IAM for storing epoch? It is fast and does not have such >> strong size limitations. >> > there are no size limitations, EA can be stored in a separate > block, we just want to minimize IO. > > EA is separate block is evil. It makes things slow. >> We could make "epoch" index in mkfs time (like >> it is done for existing indexes now) and use object's fid as a key and >> epoch as value. >> > this looks like it will double IO/seeks for each inode. > > Well, it did not in cmd3 :) -- umka From Alex.Zhuravlev at Sun.COM Tue Feb 19 11:18:25 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Tue, 19 Feb 2008 14:18:25 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAB962.8010901@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> Message-ID: <47BABB01.8060402@sun.com> Yuriy Umanets wrote: > EA is separate block is evil. It makes things slow. we have fast EAs (stored in inode, this is why we make them large) for years. > Well, it did not in cmd3 :) if it isn't stored in inode, it's a seek. thanks, Alex From Yury.Umanets at Sun.COM Tue Feb 19 12:02:02 2008 From: Yury.Umanets at Sun.COM (Yuriy Umanets) Date: Tue, 19 Feb 2008 14:02:02 +0200 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BABB01.8060402@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> Message-ID: <47BAC53A.2030106@sun.com> Alex Zhuravlev wrote: > Yuriy Umanets wrote: > >> EA is separate block is evil. It makes things slow. >> > > we have fast EAs (stored in inode, this is why we make them large) for years. > Well, people used horses for ages but this did not stop them from building cars :) Guys, I gave you idea, not worse than using EAs. I will not insist it is great. If you can't estimate its value yourself, well, let it be. We have such a nice thing as IAM and you keep talking about EAs... Seriously, IMHO what is bad about EAs: 1. You need to control their size, you need to bother; 2. Large-fast inodes make create/lookup slow. You need to load this thing to memory after all. I think this is complement to additional seeks caused by IAM; 3. Storing epoch in EA makes you use this chain to access epoch: fid->inode->epoch (in EA), IAM makes it shorter: fid->epoch (in IAM); 4. Large inodes consume more RAM; 5. There others... but they are less related to technical downsides/advantages so I will omit them. Thanks. -- umka From Alex.Zhuravlev at Sun.COM Tue Feb 19 12:09:32 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Tue, 19 Feb 2008 15:09:32 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAC53A.2030106@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> Message-ID: <47BAC6FC.7020404@sun.com> I guess there is some sort of misunderstanding here. we don't need fid->epoch mapping. we only need epoch along with other inode attributes. epoch is fixed size (8 bytes, probably few more for flags in future) thanks, Alex Yuriy Umanets wrote: > Alex Zhuravlev wrote: >> Yuriy Umanets wrote: >> >>> EA is separate block is evil. It makes things slow. >>> >> we have fast EAs (stored in inode, this is why we make them large) for years. >> > Well, people used horses for ages but this did not stop them from > building cars :) Guys, I gave you idea, not worse than using EAs. I will > not insist it is great. If you can't estimate its value yourself, well, > let it be. We have such a nice thing as IAM and you keep talking about > EAs... > > Seriously, IMHO what is bad about EAs: > > 1. You need to control their size, you need to bother; > 2. Large-fast inodes make create/lookup slow. You need to load this > thing to memory after all. I think this is complement to additional > seeks caused by IAM; > 3. Storing epoch in EA makes you use this chain to access epoch: > fid->inode->epoch (in EA), IAM makes it shorter: fid->epoch (in IAM); > 4. Large inodes consume more RAM; > 5. There others... but they are less related to technical > downsides/advantages so I will omit them. > > Thanks. > From Alex.Zhuravlev at Sun.COM Tue Feb 19 12:13:08 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Tue, 19 Feb 2008 15:13:08 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAC53A.2030106@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> Message-ID: <47BAC7D4.3010007@sun.com> btw, are you proposing to store LOV in global IAM? thanks, Alex Yuriy Umanets wrote: > Seriously, IMHO what is bad about EAs: > > 1. You need to control their size, you need to bother; > 2. Large-fast inodes make create/lookup slow. You need to load this > thing to memory after all. I think this is complement to additional > seeks caused by IAM; > 3. Storing epoch in EA makes you use this chain to access epoch: > fid->inode->epoch (in EA), IAM makes it shorter: fid->epoch (in IAM); > 4. Large inodes consume more RAM; > 5. There others... but they are less related to technical > downsides/advantages so I will omit them. > > Thanks. > From Yury.Umanets at Sun.COM Tue Feb 19 14:28:38 2008 From: Yury.Umanets at Sun.COM (Yuriy Umanets) Date: Tue, 19 Feb 2008 16:28:38 +0200 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAC6FC.7020404@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> <47BAC6FC.7020404@sun.com> Message-ID: <47BAE796.1040403@sun.com> Alex Zhuravlev wrote: > I guess there is some sort of misunderstanding here. > > we don't need fid->epoch mapping. we only need epoch along with other > inode attributes. epoch is fixed size (8 bytes, probably few more for > flags in future) > > Alex, Yes, this is what I understand as well. And we were discussing that EA approach has some downsides. In fact what you propose, that is, store it in EA is logical taking into account that epoch is kind of extension to inode fields. It is property of inode-object. It is logical to store it with inode, I see your point. But as we saw, this has/may have some downsides which may be solved with IAM. Just take this in mind when you think/work on it. I do not see why IAM is such a bad here. Thanks. > thanks, Alex > > > Yuriy Umanets wrote: > >> Alex Zhuravlev wrote: >> >>> Yuriy Umanets wrote: >>> >>> >>>> EA is separate block is evil. It makes things slow. >>>> >>>> >>> we have fast EAs (stored in inode, this is why we make them large) for years. >>> >>> >> Well, people used horses for ages but this did not stop them from >> building cars :) Guys, I gave you idea, not worse than using EAs. I will >> not insist it is great. If you can't estimate its value yourself, well, >> let it be. We have such a nice thing as IAM and you keep talking about >> EAs... >> >> Seriously, IMHO what is bad about EAs: >> >> 1. You need to control their size, you need to bother; >> 2. Large-fast inodes make create/lookup slow. You need to load this >> thing to memory after all. I think this is complement to additional >> seeks caused by IAM; >> 3. Storing epoch in EA makes you use this chain to access epoch: >> fid->inode->epoch (in EA), IAM makes it shorter: fid->epoch (in IAM); >> 4. Large inodes consume more RAM; >> 5. There others... but they are less related to technical >> downsides/advantages so I will omit them. >> >> Thanks. >> >> > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > -- umka From Yury.Umanets at Sun.COM Tue Feb 19 14:30:34 2008 From: Yury.Umanets at Sun.COM (Yuriy Umanets) Date: Tue, 19 Feb 2008 16:30:34 +0200 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAC7D4.3010007@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> <47BAC7D4.3010007@sun.com> Message-ID: <47BAE80A.8050702@sun.com> Alex Zhuravlev wrote: > btw, are you proposing to store LOV in global IAM? by "LOV" you mean LOV EA? If yes, well, this is too radical idea seems, but it may be worse to think on. Finally using IAM with it will cost almost nothing in meaning of additional development. IAM should be ready for that. Nikita, is there any limitations for value size in IAM? Thanks. > > thanks, Alex > > Yuriy Umanets wrote: >> Seriously, IMHO what is bad about EAs: >> >> 1. You need to control their size, you need to bother; >> 2. Large-fast inodes make create/lookup slow. You need to load this >> thing to memory after all. I think this is complement to additional >> seeks caused by IAM; >> 3. Storing epoch in EA makes you use this chain to access epoch: >> fid->inode->epoch (in EA), IAM makes it shorter: fid->epoch (in IAM); >> 4. Large inodes consume more RAM; >> 5. There others... but they are less related to technical >> downsides/advantages so I will omit them. >> >> Thanks. >> > -- umka From Nikita.Danilov at Sun.COM Tue Feb 19 14:36:25 2008 From: Nikita.Danilov at Sun.COM (Nikita Danilov) Date: Tue, 19 Feb 2008 17:36:25 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAE80A.8050702@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> <47BAC7D4.3010007@sun.com> <47BAE80A.8050702@sun.com> Message-ID: <18362.59753.121957.454081@gargle.gargle.HOWL> Yuriy Umanets writes: > Alex Zhuravlev wrote: > > btw, are you proposing to store LOV in global IAM? > by "LOV" you mean LOV EA? If yes, well, this is too radical idea seems, > but it may be worse to think on. Finally using IAM with it will cost > almost nothing in meaning of additional development. IAM should be ready > for that. > > Nikita, is there any limitations for value size in IAM? Htree shift code will be upset if key+value are larger than one fourth of a block, but that's easy to fix. > > Thanks. Nikita. From Yury.Umanets at Sun.COM Tue Feb 19 14:39:21 2008 From: Yury.Umanets at Sun.COM (Yuriy Umanets) Date: Tue, 19 Feb 2008 16:39:21 +0200 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAE80A.8050702@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> <47BAC7D4.3010007@sun.com> <47BAE80A.8050702@sun.com> Message-ID: <47BAEA19.7000201@sun.com> Yuriy Umanets wrote: > Alex Zhuravlev wrote: > >> btw, are you proposing to store LOV in global IAM? >> > by "LOV" you mean LOV EA? If yes, well, this is too radical idea seems, > but it may be worse to think on. Finally using IAM with it will cost > s/worse/valuable/ -- umka From Alex.Zhuravlev at Sun.COM Tue Feb 19 14:42:13 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Tue, 19 Feb 2008 17:42:13 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAE80A.8050702@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> <47BAC7D4.3010007@sun.com> <47BAE80A.8050702@sun.com> Message-ID: <47BAEAC5.3020805@sun.com> Yuriy Umanets wrote: > Alex Zhuravlev wrote: >> btw, are you proposing to store LOV in global IAM? > by "LOV" you mean LOV EA? If yes, well, this is too radical idea seems, > but it may be worse to think on. Finally using IAM with it will cost > almost nothing in meaning of additional development. IAM should be ready > for that. it will cost additional seek to access something through IAM. same applies to LOV and to epoch. thanks, Alex From Yury.Umanets at Sun.COM Tue Feb 19 14:44:14 2008 From: Yury.Umanets at Sun.COM (Yuriy Umanets) Date: Tue, 19 Feb 2008 16:44:14 +0200 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <18362.59753.121957.454081@gargle.gargle.HOWL> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> <47BAC7D4.3010007@sun.com> <47BAE80A.8050702@sun.com> <18362.59753.121957.454081@gargle.gargle.HOWL> Message-ID: <47BAEB3E.40309@sun.com> Nikita Danilov wrote: > Yuriy Umanets writes: > > Alex Zhuravlev wrote: > > > btw, are you proposing to store LOV in global IAM? > > by "LOV" you mean LOV EA? If yes, well, this is too radical idea seems, > > but it may be worse to think on. Finally using IAM with it will cost > > almost nothing in meaning of additional development. IAM should be ready > > for that. > > > > Nikita, is there any limitations for value size in IAM? > > Htree shift code will be upset if key+value are larger than one fourth > of a block, but that's easy to fix. > This is in fact interesting idea. An object (inode + EA, etc) always gets more and more info while adding new features and one day we will face the need to get rid of EA seems because it's too big. Thanks. > > > > > Thanks. > > Nikita. > -- umka From Alex.Zhuravlev at Sun.COM Tue Feb 19 14:47:29 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Tue, 19 Feb 2008 17:47:29 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAE796.1040403@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> <47BAC6FC.7020404@sun.com> <47BAE796.1040403@sun.com> Message-ID: <47BAEC01.5050102@sun.com> Yuriy Umanets wrote: > Yes, this is what I understand as well. And we were discussing that EA > approach has some downsides. In fact what you propose, that is, store it > in EA is logical taking into account that epoch is kind of extension to > inode fields. It is property of inode-object. It is logical to store it > with inode, I see your point. But as we saw, this has/may have some > downsides which may be solved with IAM. Just take this in mind when you > think/work on it. I do not see why IAM is such a bad here. 1) additional seek(s) 2) shared structure (additional cost on concurrent access) 3) inode is already 512 bytes thanks, Alex From Mikhail.Pershin at Sun.COM Tue Feb 19 14:59:05 2008 From: Mikhail.Pershin at Sun.COM (Mikhail Pershin) Date: Tue, 19 Feb 2008 17:59:05 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAC53A.2030106@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> Message-ID: On Tue, 19 Feb 2008 15:02:02 +0300, Yuriy Umanets wrote: > Alex Zhuravlev wrote: >> Yuriy Umanets wrote: >> >>> EA is separate block is evil. It makes things slow. >>> >> >> we have fast EAs (stored in inode, this is why we make them large) for >> years. >> > Well, people used horses for ages but this did not stop them from > building cars :) Guys, I gave you idea, not worse than using EAs. I will > not insist it is great. If you can't estimate its value yourself, well, > let it be. We have such a nice thing as IAM and you keep talking about > EAs... > > Seriously, IMHO what is bad about EAs: > > 1. You need to control their size, you need to bother; > 2. Large-fast inodes make create/lookup slow. You need to load this > thing to memory after all. I think this is complement to additional > seeks caused by IAM; but this is still better than extra block for EA or IAM. Btw IAM data is also in memory and takes it no less than extra inode size possibly > 3. Storing epoch in EA makes you use this chain to access epoch: > fid->inode->epoch (in EA), IAM makes it shorter: fid->epoch (in IAM); not true actually. inode will be read anyway until you are proposing to put whole inode body in IAM, so there is no benefits. Moreover inode->ea is direct mapping while fid->epoch will need index lookup and may invoke several blocks to read if IAM is large and it will be large in this case, so IO will be not better than even EA in extra block. > 4. Large inodes consume more RAM; this is the same as 2. Guys, don't forget about DMU as well. -- Mikhail Pershin Staff Engineer Lustre Group Sun Microsystems, Inc From Yury.Umanets at Sun.COM Tue Feb 19 15:10:08 2008 From: Yury.Umanets at Sun.COM (Yuriy Umanets) Date: Tue, 19 Feb 2008 17:10:08 +0200 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAEC01.5050102@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> <47BAC6FC.7020404@sun.com> <47BAE796.1040403@sun.com> <47BAEC01.5050102@sun.com> Message-ID: <47BAF150.70804@sun.com> Alex Zhuravlev wrote: > Yuriy Umanets wrote: > >> Yes, this is what I understand as well. And we were discussing that EA >> approach has some downsides. In fact what you propose, that is, store it >> in EA is logical taking into account that epoch is kind of extension to >> inode fields. It is property of inode-object. It is logical to store it >> with inode, I see your point. But as we saw, this has/may have some >> downsides which may be solved with IAM. Just take this in mind when you >> think/work on it. I do not see why IAM is such a bad here. >> > > 1) additional seek(s) > 2) shared structure (additional cost on concurrent access) > 3) inode is already 512 bytes > > Agreed, but this is all not measured and may happen that IAM is not worse but more handy in many respects. Thanks. > thanks, Alex > > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > -- umka From Kalpak.Shah at Sun.COM Tue Feb 19 15:11:31 2008 From: Kalpak.Shah at Sun.COM (Kalpak Shah) Date: Tue, 19 Feb 2008 20:41:31 +0530 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> Message-ID: <1203433891.3999.8.camel@localhost> On Tue, 2008-02-19 at 17:59 +0300, Mikhail Pershin wrote: > On Tue, 19 Feb 2008 15:02:02 +0300, Yuriy Umanets > wrote: > > > Alex Zhuravlev wrote: > >> Yuriy Umanets wrote: > >> > >>> EA is separate block is evil. It makes things slow. > >>> > >> > >> we have fast EAs (stored in inode, this is why we make them large) for > >> years. > >> > > Well, people used horses for ages but this did not stop them from > > building cars :) Guys, I gave you idea, not worse than using EAs. I will > > not insist it is great. If you can't estimate its value yourself, well, > > let it be. We have such a nice thing as IAM and you keep talking about > > EAs... > > > > Seriously, IMHO what is bad about EAs: > > > > 1. You need to control their size, you need to bother; > > 2. Large-fast inodes make create/lookup slow. You need to load this > > thing to memory after all. I think this is complement to additional > > seeks caused by IAM; > > but this is still better than extra block for EA or IAM. Btw IAM data is > also in memory and takes it no less than extra inode size possibly > > > 3. Storing epoch in EA makes you use this chain to access epoch: > > fid->inode->epoch (in EA), IAM makes it shorter: fid->epoch (in IAM); > > not true actually. inode will be read anyway until you are proposing to > put whole inode body in IAM, so there is no benefits. Moreover inode->ea > is direct mapping while fid->epoch will need index lookup and may invoke > several blocks to read if IAM is large and it will be large in this case, > so IO will be not better than even EA in extra block. > > > 4. Large inodes consume more RAM; > > this is the same as 2. > > Guys, don't forget about DMU as well. For the DMU, we will be using 1024-byte dnodes by default to store the striping information. So the epoch can be stored in the in-dnode system attributes. The epoch will need to be stored in an external block or FatZap (depending on implementation of in-dnode EAs) only in-case the file is striped across more than 10-15 OSTs. (The exact number of striped will again depend on the design of in-dnode EAs) Thanks, Kalpak. > From Yury.Umanets at Sun.COM Tue Feb 19 15:14:26 2008 From: Yury.Umanets at Sun.COM (Yuriy Umanets) Date: Tue, 19 Feb 2008 17:14:26 +0200 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> Message-ID: <47BAF252.9050409@sun.com> Mikhail Pershin wrote: > On Tue, 19 Feb 2008 15:02:02 +0300, Yuriy Umanets > wrote: > > >> Alex Zhuravlev wrote: >> >>> Yuriy Umanets wrote: >>> >>> >>>> EA is separate block is evil. It makes things slow. >>>> >>>> >>> we have fast EAs (stored in inode, this is why we make them large) for >>> years. >>> >>> >> Well, people used horses for ages but this did not stop them from >> building cars :) Guys, I gave you idea, not worse than using EAs. I will >> not insist it is great. If you can't estimate its value yourself, well, >> let it be. We have such a nice thing as IAM and you keep talking about >> EAs... >> >> Seriously, IMHO what is bad about EAs: >> >> 1. You need to control their size, you need to bother; >> 2. Large-fast inodes make create/lookup slow. You need to load this >> thing to memory after all. I think this is complement to additional >> seeks caused by IAM; >> > > but this is still better than extra block for EA or IAM. Btw IAM data is > also in memory and takes it no less than extra inode size possibly > If it is in memory it will generate less seeks :-) > >> 3. Storing epoch in EA makes you use this chain to access epoch: >> fid->inode->epoch (in EA), IAM makes it shorter: fid->epoch (in IAM); >> > > not true actually. inode will be read anyway until you are proposing to > put whole inode body in IAM, so there is no benefits. Moreover inode->ea > is direct mapping while fid->epoch will need index lookup and may invoke > several blocks to read if IAM is large and it will be large in this case, > so IO will be not better than even EA in extra block. > > I did not mean to put whole inode in IAM. I meant only put there fid as key and epoch as value. So way to access epoch is shorter with IAM as no need to load inode. But these all need to be well thought as all your mention more seeks, new reads, etc. >> 4. Large inodes consume more RAM; >> > > this is the same as 2. > > Guys, don't forget about DMU as well. > > -- umka From Ricardo.M.Correia at Sun.COM Tue Feb 19 15:18:10 2008 From: Ricardo.M.Correia at Sun.COM (Ricardo M. Correia) Date: Tue, 19 Feb 2008 15:18:10 +0000 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> Message-ID: <1203434290.10570.0.camel@localhost> On Ter, 2008-02-19 at 17:59 +0300, Mikhail Pershin wrote: > Guys, don't forget about DMU as well. For the DMU, we haven't reached a consensus on a final design for EAs in dnode with the ZFS team yet. The ZFS team proposed having variably-sized system attributes (with integer indexes) instead of having name-value attributes like ext3. I guess this is another good point to discuss in today's ZFS team meeting. Thanks, Ricardo -- Ricardo Manuel Correia Lustre Engineering Sun Microsystems, Inc. Portugal Phone +351.214134023 / x58723 Mobile +351.912590825 Email Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From Alex.Zhuravlev at Sun.COM Tue Feb 19 15:19:12 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Tue, 19 Feb 2008 18:19:12 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAF252.9050409@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> <47BAF252.9050409@sun.com> Message-ID: <47BAF370.7030804@sun.com> Yuriy Umanets wrote: > I did not mean to put whole inode in IAM. I meant only put there fid as > key and epoch as value. So way to access epoch is shorter with IAM as no > need to load inode. But these all need to be well thought as all your > mention more seeks, new reads, etc. I don't understand benefits of this approach. the idea is to pack frequently accessed data together so that we don't need additional seeks and load/store these data with a single contiguous IO. thanks, Alex From Ricardo.M.Correia at Sun.COM Tue Feb 19 15:23:11 2008 From: Ricardo.M.Correia at Sun.COM (Ricardo M. Correia) Date: Tue, 19 Feb 2008 15:23:11 +0000 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <1203433891.3999.8.camel@localhost> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> <1203433891.3999.8.camel@localhost> Message-ID: <1203434591.10570.3.camel@localhost> On Ter, 2008-02-19 at 20:41 +0530, Kalpak Shah wrote: > The epoch will need to be stored in an external block or > FatZap (depending on implementation of in-dnode EAs) only in-case the > file is striped across more than 10-15 OSTs. That may not be true. For example, with Matthew's proposed design, we could put the epoch as a system attribute with "higher priority" (lower index) than LOV data, which means it would always fit in the dnode even if we have lots of LOVs. Regards, Ricardo -- Ricardo Manuel Correia Lustre Engineering Sun Microsystems, Inc. Portugal Phone +351.214134023 / x58723 Mobile +351.912590825 Email Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From vitaly at sun.com Tue Feb 19 15:28:05 2008 From: vitaly at sun.com (Vitaly Fertman) Date: Tue, 19 Feb 2008 18:28:05 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAF252.9050409@sun.com> References: <47BAA607.1000600@sun.com> <47BAF252.9050409@sun.com> Message-ID: <200802191828.05848.vitaly@sun.com> > I did not mean to put whole inode in IAM. I meant only put there fid as > key and epoch as value. So way to access epoch is shorter with IAM as no > need to load inode. But these all need to be well thought as all your > mention more seeks, new reads, etc. as Alex already mentioned, we do not need fid->ioepoch mapping. ioepoch is a tag for inode attributes and all them need to be loaded together, i.e. there is a need to load inode anyway. -- Vitaly From eeb at sun.com Tue Feb 19 15:31:43 2008 From: eeb at sun.com (Eric Barton) Date: Tue, 19 Feb 2008 15:31:43 +0000 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAA607.1000600@sun.com> References: <47BAA607.1000600@sun.com> Message-ID: <05e801c8730c$87cbc7a0$0281a8c0@ebpc> Rather than discussing this one EA at a time, should we not consider any other EAs (e.g. being considered in current architecture work) that might contend for space in the inode? > -----Original Message----- > From: lustre-devel-bounces at lists.lustre.org > [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of > Alex Zhuravlev > Sent: 19 February 2008 9:49 AM > To: lustre-devel at lists.lustre.org > Subject: [Lustre-devel] storing SOM epoch in EA > > Good day, > > some time ago we discussed that it would be very helpful to > store epoch in inode on mds. the perfect solution could be > to store epoch in old inode body, but there is no much space > for this in the body and with DMU we'll have this problem > again. > > given the minimal inode size we use on MDS is 512 bytes, we > can store upto 13 stripes in the body. larger EAs go to a > dedicated block. if we add 8 byte epoch, then we can store > upto 12 stripes in the body. so, epoch stored in EA affects > only files with exactly 13 stripes. files with different > stripes are unaffected at all. > > couple lesser concerns are: > 1) cpu usage > 2) epoch on old filesystem with insufficient inode space > > any objections to use EA to store SOM epoch? > > thanks, Alex > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Alex.Zhuravlev at Sun.COM Tue Feb 19 15:42:03 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Tue, 19 Feb 2008 18:42:03 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <05e801c8730c$87cbc7a0$0281a8c0@ebpc> References: <47BAA607.1000600@sun.com> <05e801c8730c$87cbc7a0$0281a8c0@ebpc> Message-ID: <47BAF8CB.7090903@sun.com> well, this was one of the reasons I asked lustre-devel@ for inputs. what else we do consider important to store in inode. probably we should list all existing and planned EAs and rank them. also, I've got to think that for some cases we don't need to load LOV EA. for example, for getattr in case of SOM (size/blocks are cached on MDS). or for revalidation when client already has LOV EA. thanks, Alex Eric Barton wrote: > Rather than discussing this one EA at a time, should we not > consider any other EAs (e.g. being considered in current > architecture work) that might contend for space in the inode? > >> -----Original Message----- >> From: lustre-devel-bounces at lists.lustre.org >> [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of >> Alex Zhuravlev >> Sent: 19 February 2008 9:49 AM >> To: lustre-devel at lists.lustre.org >> Subject: [Lustre-devel] storing SOM epoch in EA >> >> Good day, >> >> some time ago we discussed that it would be very helpful to >> store epoch in inode on mds. the perfect solution could be >> to store epoch in old inode body, but there is no much space >> for this in the body and with DMU we'll have this problem >> again. >> >> given the minimal inode size we use on MDS is 512 bytes, we >> can store upto 13 stripes in the body. larger EAs go to a >> dedicated block. if we add 8 byte epoch, then we can store >> upto 12 stripes in the body. so, epoch stored in EA affects >> only files with exactly 13 stripes. files with different >> stripes are unaffected at all. >> >> couple lesser concerns are: >> 1) cpu usage >> 2) epoch on old filesystem with insufficient inode space >> >> any objections to use EA to store SOM epoch? >> >> thanks, Alex >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From Nikita.Danilov at Sun.COM Tue Feb 19 16:21:19 2008 From: Nikita.Danilov at Sun.COM (Nikita Danilov) Date: Tue, 19 Feb 2008 19:21:19 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BABB01.8060402@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> Message-ID: <18363.511.252694.497169@gargle.gargle.HOWL> Alex Zhuravlev writes: > > Yuriy Umanets wrote: > > EA is separate block is evil. It makes things slow. > > we have fast EAs (stored in inode, this is why we make them large) for years. > > > Well, it did not in cmd3 :) > > if it isn't stored in inode, it's a seek. One possible point here is that OSD has to do fid->ino translation anyway, and it makes sense to use the same index to store other information besides inode number. That is, we can use "object index" iam file to map fid into (ino, gen, ioepoch, LOV, ...) records, and this would not cause any additional seeks. The downside here is that object index is so heavily used, that making its records larger is going to increase the amount of IO significantly, so it only worth to place there things that we are absolutely sure will be needed for every inode. > > thanks, Alex Nikita. From Alex.Zhuravlev at Sun.COM Tue Feb 19 16:27:51 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Tue, 19 Feb 2008 19:27:51 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <18363.511.252694.497169@gargle.gargle.HOWL> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <18363.511.252694.497169@gargle.gargle.HOWL> Message-ID: <47BB0387.2010802@sun.com> Nikita Danilov wrote: > One possible point here is that OSD has to do fid->ino translation > anyway, and it makes sense to use the same index to store other > information besides inode number. That is, we can use "object index" iam > file to map fid into (ino, gen, ioepoch, LOV, ...) records, and this > would not cause any additional seeks. The downside here is that object > index is so heavily used, that making its records larger is going to > increase the amount of IO significantly, so it only worth to place there > things that we are absolutely sure will be needed for every inode. well, we also need to update ioepoch when we update size/blocks. thanks, Alex From aurelien.degremont at cea.fr Tue Feb 19 17:13:09 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Tue, 19 Feb 2008 18:13:09 +0100 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <537C6C0940C6C143AA46A88946B854170C2EFE91@ORNLEXCHANGE.ornl.gov> References: <47AAE307.9040305@cea.fr> <537C6C0940C6C143AA46A88946B854170C2EFE91@ORNLEXCHANGE.ornl.gov> Message-ID: <47BB0E25.3090607@cea.fr> Canon, Richard Shane a écrit : > General > * Any thought on how quotas will be handled? That's a very good question. I think this point should be discussed. The purge possibility introduces two values which could be under quota. 1 - File size (current case) 2 - The disk occupation are used (migrated files free quota) The first point are the simplest to implement and will need fewer modifications, but users could not free quota even if all their files are migrated. The second point could help users but this will be problematic when they will copy back some of their file, because this will trigger space issues and purge requests on theirs other files, and so on. IMO, the best way is to take choice #1 and possibly add a 'real disk use' quota value that could be tuned by admins. I'm not a Lustre quota specialist and AFAIK this code is a bit touchy. > Coordinator > * 3.4 - I was curious what the precise use case was that was driving > this? I don't disagree with it, but I was curious for more background Coordinator is designed to also manage internal Lustre migrations. > * 3.7.1 - The coordinator could become a scaling bottleneck. We should > think about how this will be scaled in the future I think we should be able to have several coordinators in the future. Each of them dealing with different external storages. > * 4.1 - Does the coordinator store the ext obj id or does the agent The agent does not have a storage device. It stores nothing. The external IDs are in MDT device. > * 4.3 item 2 - This looks like the coordinator could become a bottle > neck for unlinks and slow down performance. Could this be put in some > type of async queue to be processed later (or some type of attic space)? Yes, I think unlinks should be handled asynchronously. > Use Cases > * 2.3 (Use cases) - I'm really keen on this feature. I think it is very > important in order to make small file performance work well. > Unfortunately, it isn't clear how the file list gets communicated to the > archive tool. The coordinator and agent seem to only take one file at a > time. So how would this work exactly? In fact, we have presently designed the archiving tool to support this feature and only it because the archiving tool could be developped by anyone and we want this API being as stable as possible. The current Lustre component design does not handle it. But it will be added later, in a second step, and the copy tool developped since will be already compatible with it. > * 2.4 - The copy tool should be allowed to preemptively restage files. > I think this will work with the design, but we should make sure of this. > This would be useful for restaging a whole tar file versus doing things > piece-meal. That's an interesting point. I think we could avoid it but it is an interesting feature. I must think how we should modify the design to permit it. (The tool should be able to warn the coordinator: oh, i'm staging this file also! please note it) > 2 EAs - I'm worried that the EA list could get huge for holes. This part has been redesigned. The data that were stored in EA have been moved. It will be explained in the new document version. > 3.2 -item 3 - Who insures a file is archived before punches are made? The space manager did it. It is the only one which will make punch request. May be MDT could ensure it before dealing with it. > 3.3 - Another use case... The user checks to see if a file has been > archived. Ok > Also, someone earlier made the point about the archive tool being able > to reorder request. This is really important since an archival system > wants to know all the files being restaged in order to order tape mounts > and reads. I do not see any problem with this. I will add this point in the doc. > Thanks for taking the lead on this. It looks like there is a lot of > interest in it. Thanks you for your very interesting comments. -- Aurelien Degremont CEA From adilger at sun.com Tue Feb 19 20:13:52 2008 From: adilger at sun.com (Andreas Dilger) Date: Tue, 19 Feb 2008 13:13:52 -0700 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BAE80A.8050702@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <47BAC53A.2030106@sun.com> <47BAC7D4.3010007@sun.com> <47BAE80A.8050702@sun.com> Message-ID: <20080219201352.GP3029@webber.adilger.int> On Feb 19, 2008 16:30 +0200, Yuriy Umanets wrote: > Alex Zhuravlev wrote: > by "LOV" you mean LOV EA? If yes, well, this is too radical idea seems, > but it may be worse to think on. Finally using IAM with it will cost > almost nothing in meaning of additional development. IAM should be ready > for that. > > Nikita, is there any limitations for value size in IAM? One of the major problems with IAM is that e2fsck doesn't work with it, it will only exist for ldiskfs (though ZAP works for DMU), and there is a consistency issue between items stored in IAM and in rest of filesystem. If e2fsck deletes an inode, it will not delete entry in IAM, so now we have to patch e2fsck to understand not only IAM, but also specific uses of IAM that link items there to inodes in another place. I don't think that introducing dependence on IAM is practical. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From adilger at sun.com Tue Feb 19 20:19:04 2008 From: adilger at sun.com (Andreas Dilger) Date: Tue, 19 Feb 2008 13:19:04 -0700 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <05e801c8730c$87cbc7a0$0281a8c0@ebpc> References: <47BAA607.1000600@sun.com> <05e801c8730c$87cbc7a0$0281a8c0@ebpc> Message-ID: <20080219201904.GQ3029@webber.adilger.int> On Feb 19, 2008 15:31 +0000, Eric Barton wrote: > Rather than discussing this one EA at a time, should we not > consider any other EAs (e.g. being considered in current > architecture work) that might contend for space in the inode? I wouldn't object to this. There were several other proposals to add EAs to the inode, but individually the overhead is high. If we have a single aggregated EA struct for Lustre that would be more reasonable. > > -----Original Message----- > > From: lustre-devel-bounces at lists.lustre.org > > [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of > > Alex Zhuravlev > > Sent: 19 February 2008 9:49 AM > > To: lustre-devel at lists.lustre.org > > Subject: [Lustre-devel] storing SOM epoch in EA > > > > Good day, > > > > some time ago we discussed that it would be very helpful to > > store epoch in inode on mds. the perfect solution could be > > to store epoch in old inode body, but there is no much space > > for this in the body and with DMU we'll have this problem > > again. > > > > given the minimal inode size we use on MDS is 512 bytes, we > > can store upto 13 stripes in the body. larger EAs go to a > > dedicated block. if we add 8 byte epoch, then we can store > > upto 12 stripes in the body. so, epoch stored in EA affects > > only files with exactly 13 stripes. files with different > > stripes are unaffected at all. > > > > couple lesser concerns are: > > 1) cpu usage > > 2) epoch on old filesystem with insufficient inode space > > > > any objections to use EA to store SOM epoch? > > > > thanks, Alex > > _______________________________________________ > > Lustre-devel mailing list > > Lustre-devel at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-devel > > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From vitaly at sun.com Tue Feb 19 20:33:52 2008 From: vitaly at sun.com (Vitaly Fertman) Date: Tue, 19 Feb 2008 23:33:52 +0300 Subject: [Lustre-devel] on-disk SOM attributes [former storing SOM epoch in EA] In-Reply-To: <47BAA607.1000600@sun.com> References: <47BAA607.1000600@sun.com> Message-ID: <200802192333.52663.vitaly@sun.com> Hi All, Besides the question Alex asked, there are some more issues I would like to discuss, so let me list all of them here. 1) where to store on-disk IOepoch on MDS -- this question was described in the Alex's initial "storing SOM epoch in EA" email, so I will not repeat it here. 2) where to store SOM-ENABLE flag in inode? currently it is stored in inode flags, but it may be not acceptable for DMU. If so, we will probably need to move it to the place we will store on-disk IOepoch in (EA?). I also want to mention that on-disk IOepoch is needed at the attribute update time only, to be sure we write newer attributes. Whereas SOM-ENABLE flag is needed more often, thus it is also checked when we tell a client size is valid at getattr. 3) how to avoid e2fsck zeroing i_blocks on MDS? we could patch e2fsck, or alternatively store i_blocks copy in inode that fsck does not know about, e.g. in the same EA. As i_blocks is needed on each getattr, it is worth to store it along with SOM-ENABLE flag. Please advise. -- Vitaly From Peter.Bojanic at Sun.COM Wed Feb 20 02:32:32 2008 From: Peter.Bojanic at Sun.COM (Peter Bojanic) Date: Tue, 19 Feb 2008 22:32:32 -0400 Subject: [Lustre-devel] NFSv4 ACLs Message-ID: <038AFFD1-A0D1-4861-8A47-5FF3281BD60F@sun.com> Bryon, Can you advise the general impact and effort of adding support to Lustre NFSv4 ACLs. This is not an urgent matter, but one that Braam had advised we look into. Thanks, Bojanic From Peter.Bojanic at Sun.COM Wed Feb 20 12:17:32 2008 From: Peter.Bojanic at Sun.COM (Peter Bojanic) Date: Wed, 20 Feb 2008 08:17:32 -0400 Subject: [Lustre-devel] WBC HLD: next round In-Reply-To: <18361.54057.703866.863905@gargle.gargle.HOWL> References: <18361.54057.703866.863905@gargle.gargle.HOWL> Message-ID: <3A1C3CF2-7AD7-4B51-9CFB-2FA6B230DE09@sun.com> Hi Nikita, LLNL highly encouraged us to consider all of the failure use cases for WBC. I haven't looked at the HLD in detail, so perhaps this already addressed sufficiently. I would advise asking LLNL explicitly for an inspection. Cheers, Bojanic On 2008-02-18, at 14:49 , Nikita Danilov wrote: > Hello, > > please take a look at the next version of the High Level Design > specification for the Write-Back Cache subsystem, that has Use Cases > and > Logical Specification sections fleshed out. > > I'd appreciate any comments, especially on the topics marked with "Q!" > on a margin. > > Peter, can you please take a look at the description of a security > hole > on page 10? > > Nikita. > From Nikita.Danilov at Sun.COM Wed Feb 20 16:08:21 2008 From: Nikita.Danilov at Sun.COM (Nikita Danilov) Date: Wed, 20 Feb 2008 19:08:21 +0300 Subject: [Lustre-devel] WBC HLD: next round In-Reply-To: <3A1C3CF2-7AD7-4B51-9CFB-2FA6B230DE09@sun.com> References: <18361.54057.703866.863905@gargle.gargle.HOWL> <3A1C3CF2-7AD7-4B51-9CFB-2FA6B230DE09@sun.com> Message-ID: <18364.20597.834440.653880@gargle.gargle.HOWL> Peter Bojanic writes: > Hi Nikita, Hello Peter, > > LLNL highly encouraged us to consider all of the failure use cases for > WBC. I haven't looked at the HLD in detail, so perhaps this already > addressed sufficiently. I would advise asking LLNL explicitly for an > inspection. OK, I shall. > > Cheers, > Bojanic Nikita. > > On 2008-02-18, at 14:49 , Nikita Danilov wrote: > > > Hello, > > > > please take a look at the next version of the High Level Design > > specification for the Write-Back Cache subsystem, that has Use Cases > > and > > Logical Specification sections fleshed out. > > > > I'd appreciate any comments, especially on the topics marked with "Q!" > > on a margin. > > > > Peter, can you please take a look at the description of a security > > hole > > on page 10? From Nikita.Danilov at Sun.COM Wed Feb 20 16:10:08 2008 From: Nikita.Danilov at Sun.COM (Nikita Danilov) Date: Wed, 20 Feb 2008 19:10:08 +0300 Subject: [Lustre-devel] storing SOM epoch in EA In-Reply-To: <47BB0387.2010802@sun.com> References: <47BAA607.1000600@sun.com> <47BAAF3F.6030301@sun.com> <200802191359.47379.vitaly@sun.com> <47BAB962.8010901@sun.com> <47BABB01.8060402@sun.com> <18363.511.252694.497169@gargle.gargle.HOWL> <47BB0387.2010802@sun.com> Message-ID: <18364.20704.788012.460498@gargle.gargle.HOWL> Alex Zhuravlev writes: > Nikita Danilov wrote: > > One possible point here is that OSD has to do fid->ino translation > > anyway, and it makes sense to use the same index to store other > > information besides inode number. That is, we can use "object index" iam > > file to map fid into (ino, gen, ioepoch, LOV, ...) records, and this > > would not cause any additional seeks. The downside here is that object > > index is so heavily used, that making its records larger is going to > > increase the amount of IO significantly, so it only worth to place there > > things that we are absolutely sure will be needed for every inode. > > well, we also need to update ioepoch when we update size/blocks. Indeed. We still can do this for read-only and read-mostly attributes, like LOV and avoid dirtying extra blocks in the common case. > > thanks, Alex Nikita. From Alex.Zhuravlev at Sun.COM Thu Feb 21 13:30:53 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Thu, 21 Feb 2008 16:30:53 +0300 Subject: [Lustre-devel] subtree locks and path re-validation avoidance In-Reply-To: <1203080062.19279.91.camel@linux.site> References: <1203080062.19279.91.camel@linux.site> Message-ID: <47BD7D0D.6090906@sun.com> Hi, couple comments inline ... Vladimir V. Saveliev wrote: > The example shows the details: > > 1. A client C1 holds ordinary lock on an object O1 (it did > chdir(/a/b/c/d/e), O1 is inode of /a/b/c/d/e). C1 is idle now. chdir doesn't return any lock. should it? > 2. Another client C2 does ls -ld /a/b/c/d/e, MD server sends a BAST to > C1 and C1 cancels the lock of O1. > > 3. C2 is not interested anymore in O1, so it drops the lock. > > 4. Yet another client C3 acquires subtree lock on /a/b and caches and > possibly changes (if under WBC) objects under /a/b including /a/b/c/d/e > (the object O1). The key issue is that MDS neither remembers about O1 on > C1 nor keeps information about objects cached by a client under a > subtree lock. > > 5. Now C1 continues with stat(``.''). It sees that the lock on O1 is > canceled, so it goes to MD server and acquires the lock on O1. > > Now we have: > uptodate O1 is on C3; > MDS has a request for O1 from C1 and MDS can not easily deterimine > whether O1 is under any subtree lock. In order to find whether the lock > conflict exists we need to have a special procedure. It is referred to > as path re-validation. > > The main thing to be done on path re-validation is to look for above > subtree lock. While it is probably doable, the path re-validation is not > going to be very efficient (especially in case of CMD). I can provide > more details if necessary. > > > However, it looks like it is possible to avoid having to do path > re-validation completely. > > > The problem appears when clients request locks on objects directly, > without doing downward lookup through a directory structure. > This happens, for example, when clients access directly components of > current working directories (CWDs). > If a client cancels locks on such objects (either due to a BAST or > voluntary) - it has to go through the path re-validation later. > > Objects to which a client may access directly appear in result of normal > downward lookup. Therefore, they were locked, and their locks can be > canceled. That is the point where we can take care about future accesses > without re-validation. > On canceling a lock of directly accessible object we have to inform DLM > that the ordinary locking has to be used for that object. That will > prevent the object from getting cached under a subtree lock. 1) there may be thousands of such objects (many processes on many nodes) 2) it's not clear when to enable this back > > The problem with this schema is to determine which objects are directly > accessible. But wouldn't solving it be worth doing given that it may > help to avoid path re-validation deal. > > Any comments are welcome. thanks, Alex From aurelien.degremont at cea.fr Thu Feb 21 15:26:07 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Thu, 21 Feb 2008 16:26:07 +0100 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47AAE307.9040305@cea.fr> References: <47AAE307.9040305@cea.fr> Message-ID: <47BD980F.50201@cea.fr> Hello I've got several wondering about some specific point in HSM implementation and I would like your opinion about them. Coordinator: This element will manage migration externally (HSM) and internally of Lustre (space balancing?). Is the current API acceptable (specific calls for external migration, and other ones for internal migration)? The best way could have been to have generic call for migration, but we must also have generic objects to describe the migration sources and destinations and those are not simples. We finally conclude with the API presented in the HLD document. Tell me if this is *really* a bad idea or if only adjustments are needed. We presented two modes of migration, explicit and implicit migrations. The first one result of an administrative request, the second one was triggered automatically (cache miss by example). Is that ok? (See the doc for all details). Agent: It seems, to support Lustre internal migration, you have planned to implement specific Agents which will reside on OST. HSM will need specific agent on clients. Do those two kinds of agent are acceptable ? The current API only describe HSM-based agent. Maybe we should think of a generic agent framework and add specialized implementations for ost,hsm,etc ? -- Aurelien Degremont CEA From Peter.Braam at Sun.COM Sat Feb 23 03:15:06 2008 From: Peter.Braam at Sun.COM (Peter J Braam) Date: Fri, 22 Feb 2008 20:15:06 -0700 Subject: [Lustre-devel] subtree locks and path re-validation avoidance In-Reply-To: <47BD7D0D.6090906@sun.com> References: <1203080062.19279.91.camel@linux.site> <47BD7D0D.6090906@sun.com> Message-ID: <47BF8FBA.3090705@sun.com> I'd like to make a suggestion to perhaps immediately find the right primitives for getcwd to return a reasonably correct pathname in Lustre. I believe this is the simplest case where I have seen pathname revalidation being important. In the context of that example the subtree lock discussion might gain more clarity. I would also like to note that I had a discussion with Linus at one of the kernel workshops in Ottawa maybe almost 4-5 years ago. First Linus attacked the idea of using file identifiers - he suggested that doing everything with pathnames was better (which is what InterMezzo did). When we explained to him that this requires locking all parents he began to see the problems we had with this and understood the locking at the fid/name level that we use in Lustre. I found little resistance when I mentioned to him that for this model the VFS does not have a correct implementation of getcwd, unless the dcache is kept current. UCSC has received funding from the National Labs and now been turned into a peta-scale I/O institute I believe did more results on file systems implemented with pathnames. Some things are beautiful and easy with pathnames, but others get really ugly, and so far I don't see this displacing fid ideas that govern NFS, AFS and Lustre. - Peter - Alex Zhuravlev wrote: > Hi, > > couple comments inline ... > > Vladimir V. Saveliev wrote: > >> The example shows the details: >> >> 1. A client C1 holds ordinary lock on an object O1 (it did >> chdir(/a/b/c/d/e), O1 is inode of /a/b/c/d/e). C1 is idle now. >> > > chdir doesn't return any lock. should it? > > >> 2. Another client C2 does ls -ld /a/b/c/d/e, MD server sends a BAST to >> C1 and C1 cancels the lock of O1. >> >> 3. C2 is not interested anymore in O1, so it drops the lock. >> >> 4. Yet another client C3 acquires subtree lock on /a/b and caches and >> possibly changes (if under WBC) objects under /a/b including /a/b/c/d/e >> (the object O1). The key issue is that MDS neither remembers about O1 on >> C1 nor keeps information about objects cached by a client under a >> subtree lock. >> >> 5. Now C1 continues with stat(``.''). It sees that the lock on O1 is >> canceled, so it goes to MD server and acquires the lock on O1. >> >> Now we have: >> uptodate O1 is on C3; >> MDS has a request for O1 from C1 and MDS can not easily deterimine >> whether O1 is under any subtree lock. In order to find whether the lock >> conflict exists we need to have a special procedure. It is referred to >> as path re-validation. >> >> The main thing to be done on path re-validation is to look for above >> subtree lock. While it is probably doable, the path re-validation is not >> going to be very efficient (especially in case of CMD). I can provide >> more details if necessary. >> >> >> However, it looks like it is possible to avoid having to do path >> re-validation completely. >> >> >> The problem appears when clients request locks on objects directly, >> without doing downward lookup through a directory structure. >> This happens, for example, when clients access directly components of >> current working directories (CWDs). >> If a client cancels locks on such objects (either due to a BAST or >> voluntary) - it has to go through the path re-validation later. >> >> Objects to which a client may access directly appear in result of normal >> downward lookup. Therefore, they were locked, and their locks can be >> canceled. That is the point where we can take care about future accesses >> without re-validation. >> On canceling a lock of directly accessible object we have to inform DLM >> that the ordinary locking has to be used for that object. That will >> prevent the object from getting cached under a subtree lock. >> > > 1) there may be thousands of such objects (many processes on many nodes) > 2) it's not clear when to enable this back > > >> The problem with this schema is to determine which objects are directly >> accessible. But wouldn't solving it be worth doing given that it may >> help to avoid path re-validation deal. >> >> Any comments are welcome. >> > > thanks, Alex > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Alexander.Zarochentsev at Sun.COM Sun Feb 24 21:01:40 2008 From: Alexander.Zarochentsev at Sun.COM (Alexander Zarochentsev) Date: Mon, 25 Feb 2008 00:01:40 +0300 Subject: [Lustre-devel] subtree locks and path re-validation avoidance In-Reply-To: <1203080062.19279.91.camel@linux.site> References: <1203080062.19279.91.camel@linux.site> Message-ID: <200802250001.40778.alexander.zarochentsev@sun.com> Hi, On 15 February 2008 15:54:22 Vladimir V. Saveliev wrote: > Hello [...] > > The problem appears when clients request locks on objects directly, > without doing downward lookup through a directory structure. > This happens, for example, when clients access directly components of > current working directories (CWDs). > If a client cancels locks on such objects (either due to a BAST or > voluntary) - it has to go through the path re-validation later. > > Objects to which a client may access directly appear in result of > normal downward lookup. Therefore, they were locked, and their locks > can be canceled. That is the point where we can take care about > future accesses without re-validation. what to do if a lookup looses all its locks due to a conflict with STL holder? Of course the parent lock can be correctly re-acquired but the lookup result may be incorrect -- the result lock may be done for STL-cached object. There is a more detailed example: Suppose parent lock in a lookup step has been lost before acquiring a lock on child. If we don't want to perform path re-validation we have to inform any STL-holder that the child is not-stl-lockable. I think the problem looks even worse, any ancestor of the revoked parent should be not-stl-lockable. When lookup (C1) holding at least one lock, the lock is a barrier for STL-holder, who can't go over the lock into the subtree. Once C1 looses all its locks, an STL-holder may leak into the subtree, cache those locks and cause locking correctness problems. I suggest any STL-holder to switch to ordinary locks mode when going over that parent dir. The parent dir would have a marker: "well, STL behave not well under this dir, please use ordinary locks instead". > On canceling a lock of directly accessible object we have to inform > DLM that the ordinary locking has to be used for that object. That > will prevent the object from getting cached under a subtree lock. > > The problem with this schema is to determine which objects are > directly accessible. But wouldn't solving it be worth doing given > that it may help to avoid path re-validation deal. > > Any comments are welcome. > > Best regards, > Vladimir > Thanks. From ckerner at ncsa.uiuc.edu Mon Feb 25 16:06:16 2008 From: ckerner at ncsa.uiuc.edu (Chad Kerner) Date: Mon, 25 Feb 2008 10:06:16 -0600 (CST) Subject: [Lustre-devel] problem starting an mds Message-ID: Hello, I have an mds with 16 ost's, when I try to start the MDS, I am getting the following errors in syslog: Feb 25 08:44:57 linc-io17 Lustre: 10510:0:(llog_obd.c:160:cat_cancel_cb()) processing log 0x13fd84c:c271bef6 at index 124 of catalog 0x13fd802 Feb 25 08:44:57 linc-io17 LustreError: 10510:0:(llog_lvfs.c:552:llog_lvfs_create()) error looking up logfile 0x13fd84c:0xc271bef6: rc -2 Feb 25 08:44:57 linc-io17 LustreError: 10510:0:(llog_cat.c:158:llog_cat_id2handle()) error opening log id 0x13fd84c:c271bef6: rc -2 Feb 25 08:44:57 linc-io17 LustreError: 10510:0:(llog_obd.c:165:cat_cancel_cb()) Cannot find handle for log 0x13fd84c Feb 25 08:44:57 linc-io17 LustreError: 10510:0:(llog_obd.c:232:llog_obd_origin_setup()) llog_process with cat_cancel_cb failed: -2 Feb 25 08:44:57 linc-io17 LustreError: 10510:0:(lov_log.c:204:lov_llog_init()) error osc_llog_init 0 Feb 25 08:44:57 linc-io17 LustreError: 10510:0:(mds_log.c:197:mds_llog_init()) error lov_llog_init Feb 25 08:44:57 linc-io17 LustreError: 10510:0:(llog_obd.c:322:llog_cat_initialize()) rc: -2 Feb 25 08:44:57 linc-io17 LustreError: 10510:0:(mds_lov.c:240:mds_lov_connect()) failed to initialize catalog -2 Was wondering if anyone else had seen a similar problem or had any places I should start looking. Thanks, Chad -- Chad Kerner - ckerner at ncsa.uiuc.edu Systems Engineer, Storage Enabling Technologies National Center for Supercomputing Applications http://www.ncsa.uiuc.edu/~ckerner From Brian.Murrell at Sun.COM Mon Feb 25 16:28:48 2008 From: Brian.Murrell at Sun.COM (Brian J. Murrell) Date: Mon, 25 Feb 2008 11:28:48 -0500 Subject: [Lustre-devel] problem starting an mds In-Reply-To: References: Message-ID: <1203956928.8507.16.camel@pc.ilinx> On Mon, 2008-02-25 at 10:06 -0600, Chad Kerner wrote: > Hello, > > I have an mds with 16 ost's, when I try to start the MDS, I am > getting the following errors in syslog: > > Feb 25 08:44:57 linc-io17 Lustre: > 10510:0:(llog_obd.c:160:cat_cancel_cb()) processing log > 0x13fd84c:c271bef6 at index 124 of catalog 0x13fd802 > Feb 25 08:44:57 linc-io17 LustreError: > 10510:0:(llog_lvfs.c:552:llog_lvfs_create()) error looking up logfile > 0x13fd84c:0xc271bef6: rc -2 > Feb 25 08:44:57 linc-io17 LustreError: > 10510:0:(llog_cat.c:158:llog_cat_id2handle()) error opening log id > 0x13fd84c:c271bef6: rc -2 > Feb 25 08:44:57 linc-io17 LustreError: > 10510:0:(llog_obd.c:165:cat_cancel_cb()) Cannot find handle for log > 0x13fd84c > Feb 25 08:44:57 linc-io17 LustreError: > 10510:0:(llog_obd.c:232:llog_obd_origin_setup()) llog_process with > cat_cancel_cb failed: -2 > Feb 25 08:44:57 linc-io17 LustreError: > 10510:0:(lov_log.c:204:lov_llog_init()) error osc_llog_init 0 > Feb 25 08:44:57 linc-io17 LustreError: > 10510:0:(mds_log.c:197:mds_llog_init()) error lov_llog_init > Feb 25 08:44:57 linc-io17 LustreError: > 10510:0:(llog_obd.c:322:llog_cat_initialize()) rc: -2 > Feb 25 08:44:57 linc-io17 LustreError: > 10510:0:(mds_lov.c:240:mds_lov_connect()) failed to initialize catalog > -2 This appears similar to bug 14651. Caveat that I have not read through the bug's whole history, but the error messages look the same. Perhaps you'd like to read through that bug and see if it applies to you. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From Peter.Bojanic at Sun.COM Mon Feb 25 19:35:04 2008 From: Peter.Bojanic at Sun.COM (Peter Bojanic) Date: Mon, 25 Feb 2008 15:35:04 -0400 Subject: [Lustre-devel] Lustre-devel Digest, Vol 23, Issue 26 In-Reply-To: References: Message-ID: <7E28A608-A10D-492F-A9E0-5A68914010B1@sun.com> Hi Chad, On 2008-02-25, at 13:00 , lustre-devel-request at lists.lustre.org wrote: > Date: Mon, 25 Feb 2008 10:06:16 -0600 (CST) > From: Chad Kerner > Subject: [Lustre-devel] problem starting an mds > To: lustre-devel at lists.lustre.org > > Hello, > > I have an mds with 16 ost's, when I try to start the MDS, I am > getting the following errors in syslog: > [...] You want the lustre-discuss at lists.lustre.org mailing list for Lustre usage discussions. This list is strictly for development and architectural discussion. Thanks, Bojanic From acuselton at lbl.gov Mon Feb 25 21:44:40 2008 From: acuselton at lbl.gov (Andrew C. Uselton) Date: Mon, 25 Feb 2008 13:44:40 -0800 Subject: [Lustre-devel] Thoughts on benchmarking Message-ID: <47C336C8.4090108@lbl.gov> Howdy all, I'd been in conversation with Cliff White over the last few weeks, and he'd expressed an interest in having me post a draft of a report I've been working on. If you've already heard of it here it is. For those who hadn't I'll try to describe it briefly. In December I assisted with some Lustre benchmark tests on the Franklin Cray XT here at NERSC. Since then I've tried to summarize our analysis and results. The attached pdf is a draft of that summary. The introduction is almost completely useless, so feel free to skip (unless you want to have a laugh at the author's expense). Section 3 has the main details about what we observed and what we thought about it. Section 2 may be amusing for those (like me) who care about methodology. Cheers, Andrew -------------- next part -------------- A non-text attachment was scrubbed... Name: uselton.pdf Type: application/pdf Size: 292842 bytes Desc: not available URL: From Peter.Braam at Sun.COM Mon Feb 25 22:38:16 2008 From: Peter.Braam at Sun.COM (Peter J Braam) Date: Mon, 25 Feb 2008 15:38:16 -0700 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47BD980F.50201@cea.fr> References: <47AAE307.9040305@cea.fr> <47BD980F.50201@cea.fr> Message-ID: <47C34358.6050806@sun.com> Aurelien Degremont wrote: > Hello > > I've got several wondering about some specific point in HSM > implementation and I would like your opinion about them. > > Coordinator: > > This element will manage migration externally (HSM) and internally of > Lustre (space balancing?). Is the current API acceptable (specific calls > for external migration, and other ones for internal migration)? I would like to see a parameter indicating what agent will be used and keep all other parameters the same. > The best > way could have been to have generic call for migration, but we must also > have generic objects to describe the migration sources and destinations > and those are not simples. For migration to and from external sources, Lustre must already manage this data in an extended attribute (e.g. to describe the file on tape to which a Lustre file was migrated). This data is opaque to Lustre and can be passed as a blob. > We finally conclude with the API presented in > the HLD document. Tell me if this is *really* a bad idea or if only > adjustments are needed. > > I have not yet looked at these. > We presented two modes of migration, explicit and implicit migrations. > The first one result of an administrative request, the second one was > triggered automatically (cache miss by example). Is that ok? (See the > doc for all details). > Yes, that seems ok. > Agent: > > It seems, to support Lustre internal migration, you have planned to > implement specific Agents which will reside on OST. To avoid many complications involving locks, we decided that even the agents used for internal migrations will layer on the file system. The Lustre file system will be mounted on the OST's and it will use the "LOLND" to transport the data efficiently between the OST process and the client file system cache. In the internal case source and destination lie in Lustre in the HSM case only one of them. As a result I believe these two cases are closer together than you may think, and should be one "type". The key aspect we/you need to design is what an agent has to make sure happens, for example in terms of locking file extents and in terms of avoiding triggering a recursive cache miss (open by fid with a flag?). - Peter - > HSM will need > specific agent on clients. Do those two kinds of agent are acceptable ? > The current API only describe HSM-based agent. Maybe we should think of > a generic agent framework and add specialized implementations for > ost,hsm,etc ? > > > > From Peter.Braam at Sun.COM Mon Feb 25 22:44:24 2008 From: Peter.Braam at Sun.COM (Peter J Braam) Date: Mon, 25 Feb 2008 15:44:24 -0700 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <537C6C0940C6C143AA46A88946B854170C2EFE91@ORNLEXCHANGE.ornl.gov> References: <47AAE307.9040305@cea.fr> <537C6C0940C6C143AA46A88946B854170C2EFE91@ORNLEXCHANGE.ornl.gov> Message-ID: <47C344C8.8090203@sun.com> Just a few initial responses from me, I haven't read things systematically yet. Canon, Richard Shane wrote: > Aurelien and JC, > > Sorry that my feedback is late. Here are my questions/remarks. > > General > * Any thought on how quotas will be handled? > > This is very very important and will require a lot of detail. Well spotted Shane!!! > Coordinator > * 3.4 - I was curious what the precise use case was that was driving > this? I don't disagree with it, but I was curious for more background > In internal migrations many objects will be restriped to another set of objects to move the data. The coordinator handles the completion and abortion of the agents accomplishing this. > * 3.7.1 - The coordinator could become a scaling bottleneck. We should > think about how this will be scaled in the future > In my writings I was always anticipating a family of load balancing coordinators. > * 4.1 - Does the coordinator store the ext obj id or does the agent > Coordinator, I suggest, in view of the fact that many agents may be required to move one file. > * 4.3 item 2 - This looks like the coordinator could become a bottle > neck for unlinks and slow down performance. Could this be put in some > type of async queue to be processed later (or some type of attic space)? > > I agree with this. > Use Cases > * 2.3 (Use cases) - I'm really keen on this feature. I think it is very > important in order to make small file performance work well. > Unfortunately, it isn't clear how the file list gets communicated to the > archive tool. The coordinator and agent seem to only take one file at a > time. So how would this work exactly? > * 2.4 - The copy tool should be allowed to preemptively restage files. > I think this will work with the design, but we should make sure of this. > This would be useful for restaging a whole tar file versus doing things > piece-meal. > > Part IV > 2 EAs - I'm worried that the EA list could get huge for holes. > The EA merely points to an extent tree (similar to the allocation extent tree). > 3.2 -item 3 - Who insures a file is archived before punches are made? > The coordinator. > 3.3 - Another use case... The user checks to see if a file has been > archived. > > > Also, someone earlier made the point about the archive tool being able > to reorder request. This is really important since an archival system > wants to know all the files being restaged in order to order tape mounts > and reads. > > Thanks for taking the lead on this. It looks like there is a lot of > interest in it. > > --Shane > > -----Original Message----- > From: lustre-devel-bounces at lists.lustre.org > [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of DEGREMONT > Aurelien > Sent: Thursday, February 07, 2008 5:53 AM > To: lustre-devel at lists.lustre.org > Subject: [Lustre-devel] Lustre HSM HLD draft > > Hello > > Here is a first draft for comments of the Lustre HSM HLD. > It is intended to be a support for further analyzes and comments from > CFS/Sun. > > The document covers the main parts of the HSM features but some elements > are still lacking. > The policy management and the space manager will be describe later. > > Let us know your comments and ideas about it. > > Regards, > > Aurelien Degremont > CEA > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From aurelien.degremont at cea.fr Wed Feb 27 16:51:04 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Wed, 27 Feb 2008 17:51:04 +0100 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47C34358.6050806@sun.com> References: <47AAE307.9040305@cea.fr> <47BD980F.50201@cea.fr> <47C34358.6050806@sun.com> Message-ID: <47C594F8.20409@cea.fr> Peter J Braam a écrit : >> Coordinator: >> >> This element will manage migration externally (HSM) and internally of >> Lustre (space balancing?). Is the current API acceptable (specific >> calls for external migration, and other ones for internal migration)? > I would like to see a parameter indicating what agent will be used and > keep all other parameters the same. Agreed. >> The best way could have been to have generic call for migration, but >> we must also have generic objects to describe the migration sources >> and destinations and those are not simples. > For migration to and from external sources, Lustre must already manage > this data in an extended attribute (e.g. to describe the file on tape to > which a Lustre file was migrated). This data is opaque to Lustre and > can be passed as a blob. >> It seems, to support Lustre internal migration, you have planned to >> implement specific Agents which will reside on OST. > To avoid many complications involving locks, we decided that even the > agents used for internal migrations will layer on the file system. The > Lustre file system will be mounted on the OST's and it will use the > "LOLND" to transport the data efficiently between the OST process and > the client file system cache. In the internal case source and > destination lie in Lustre in the HSM case only one of them. > > As a result I believe these two cases are closer together than you may > think, and should be one "type". If we unify the API, we must have a way to request some data movement like: copy elemA in placeP copy elemA,stored in placeP bak into Lustre copy elemA into placeC move elemB into elemB The elem could be unified using Lustre FID, but the places could be an external storage, or a precise OST. If we want a unify API, the API call should manipulate a generic object which could describe a Lustre storage element (ost) or a external storage (hsm,...) ie: struct storage_place { ... } copy(fid,storage_place*) move(fid,storage_place*) and their is some specific cases to handle. The other possibity: ext_copyout(fid, external storage) ext_copyin(fid, external object) int_copy(fid, fid, ost) int_move(fid, fid, ost) I think this one, even if the design is not the most beautiful one, if the easiest one. Instead you want to create some new generic objects to manipulate lustre object data and generic storage areas, the second case is the best one IMO. -- Aurelien Degremont CEA From Peter.Braam at Sun.COM Thu Feb 28 16:51:23 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Thu, 28 Feb 2008 09:51:23 -0700 Subject: [Lustre-devel] Thoughts on benchmarking In-Reply-To: <47C336C8.4090108@lbl.gov> Message-ID: Oh, and I forgot to say - WOW what a nice paper! - Peter - On 2/25/08 2:44 PM, "Andrew C. Uselton" wrote: > Howdy all, > I'd been in conversation with Cliff White over the last few weeks, and > he'd expressed an interest in having me post a draft of a report I've > been working on. If you've already heard of it here it is. For those > who hadn't I'll try to describe it briefly. > > In December I assisted with some Lustre benchmark tests on the > Franklin Cray XT here at NERSC. Since then I've tried to summarize our > analysis and results. The attached pdf is a draft of that summary. The > introduction is almost completely useless, so feel free to skip (unless > you want to have a laugh at the author's expense). Section 3 has the > main details about what we observed and what we thought about it. > Section 2 may be amusing for those (like me) who care about methodology. > Cheers, > Andrew > From Vladimir.Saveliev at Sun.COM Thu Feb 28 19:01:44 2008 From: Vladimir.Saveliev at Sun.COM (Vladimir V. Saveliev) Date: Thu, 28 Feb 2008 21:01:44 +0200 Subject: [Lustre-devel] subtree locks and path re-validation avoidance In-Reply-To: <47BF8FBA.3090705@sun.com> References: <1203080062.19279.91.camel@linux.site> <47BD7D0D.6090906@sun.com> <47BF8FBA.3090705@sun.com> Message-ID: <1204225304.4125.20.camel@linux.site> Hello On Fri, 2008-02-22 at 20:15 -0700, Peter J Braam wrote: > I'd like to make a suggestion to perhaps immediately find the right > primitives for getcwd to return a reasonably correct pathname in > Lustre. I believe this is the simplest case where I have seen pathname > revalidation being important. In the context of that example the > subtree lock discussion might gain more clarity. Can we have something like th When a client cancels a lock on an object on CWD, server sets NULL mode lock for the object. That NULL mode lock indicates that the object is > > I would also like to note that I had a discussion with Linus at one of > the kernel workshops in Ottawa maybe almost 4-5 years ago. First Linus > attacked the idea of using file identifiers - he suggested that doing > everything with pathnames was better (which is what InterMezzo did). > When we explained to him that this requires locking all parents he began > to see the problems we had with this and understood the locking at the > fid/name level that we use in Lustre. I found little resistance when I > mentioned to him that for this model the VFS does not have a correct > implementation of getcwd, unless the dcache is kept current. > > UCSC has received funding from the National Labs and now been turned > into a peta-scale I/O institute I believe did more results on file > systems implemented with pathnames. Some things are beautiful and easy > with pathnames, but others get really ugly, and so far I don't see this > displacing fid ideas that govern NFS, AFS and Lustre. > > - Peter - > > Alex Zhuravlev wrote: > > Hi, > > > > couple comments inline ... > > > > Vladimir V. Saveliev wrote: > > > >> The example shows the details: > >> > >> 1. A client C1 holds ordinary lock on an object O1 (it did > >> chdir(/a/b/c/d/e), O1 is inode of /a/b/c/d/e). C1 is idle now. > >> > > > > chdir doesn't return any lock. should it? > > > > > >> 2. Another client C2 does ls -ld /a/b/c/d/e, MD server sends a BAST to > >> C1 and C1 cancels the lock of O1. > >> > >> 3. C2 is not interested anymore in O1, so it drops the lock. > >> > >> 4. Yet another client C3 acquires subtree lock on /a/b and caches and > >> possibly changes (if under WBC) objects under /a/b including /a/b/c/d/e > >> (the object O1). The key issue is that MDS neither remembers about O1 on > >> C1 nor keeps information about objects cached by a client under a > >> subtree lock. > >> > >> 5. Now C1 continues with stat(``.''). It sees that the lock on O1 is > >> canceled, so it goes to MD server and acquires the lock on O1. > >> > >> Now we have: > >> uptodate O1 is on C3; > >> MDS has a request for O1 from C1 and MDS can not easily deterimine > >> whether O1 is under any subtree lock. In order to find whether the lock > >> conflict exists we need to have a special procedure. It is referred to > >> as path re-validation. > >> > >> The main thing to be done on path re-validation is to look for above > >> subtree lock. While it is probably doable, the path re-validation is not > >> going to be very efficient (especially in case of CMD). I can provide > >> more details if necessary. > >> > >> > >> However, it looks like it is possible to avoid having to do path > >> re-validation completely. > >> > >> > >> The problem appears when clients request locks on objects directly, > >> without doing downward lookup through a directory structure. > >> This happens, for example, when clients access directly components of > >> current working directories (CWDs). > >> If a client cancels locks on such objects (either due to a BAST or > >> voluntary) - it has to go through the path re-validation later. > >> > >> Objects to which a client may access directly appear in result of normal > >> downward lookup. Therefore, they were locked, and their locks can be > >> canceled. That is the point where we can take care about future accesses > >> without re-validation. > >> On canceling a lock of directly accessible object we have to inform DLM > >> that the ordinary locking has to be used for that object. That will > >> prevent the object from getting cached under a subtree lock. > >> > > > > 1) there may be thousands of such objects (many processes on many nodes) > > 2) it's not clear when to enable this back > > > > > >> The problem with this schema is to determine which objects are directly > >> accessible. But wouldn't solving it be worth doing given that it may > >> help to avoid path re-validation deal. > >> > >> Any comments are welcome. > >> > > > > thanks, Alex > > _______________________________________________ > > Lustre-devel mailing list > > Lustre-devel at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-devel > > From Vladimir.Saveliev at Sun.COM Thu Feb 28 19:05:13 2008 From: Vladimir.Saveliev at Sun.COM (Vladimir V. Saveliev) Date: Thu, 28 Feb 2008 21:05:13 +0200 Subject: [Lustre-devel] please ignore previous mail, it was sent accidentially Re: subtree locks and path re-validation avoidance In-Reply-To: <1204225304.4125.20.camel@linux.site> References: <1203080062.19279.91.camel@linux.site> <47BD7D0D.6090906@sun.com> <47BF8FBA.3090705@sun.com> <1204225304.4125.20.camel@linux.site> Message-ID: <1204225513.4125.24.camel@linux.site> Oops. Sorry, I did not complete the mail On Thu, 2008-02-28 at 21:02 +0200, Vladimir V. Saveliev wrote: > Hello > > On Fri, 2008-02-22 at 20:15 -0700, Peter J Braam wrote: > > I'd like to make a suggestion to perhaps immediately find the right > > primitives for getcwd to return a reasonably correct pathname in > > Lustre. I believe this is the simplest case where I have seen pathname > > revalidation being important. In the context of that example the > > subtree lock discussion might gain more clarity. > > Can we have something like th > > When a client cancels a lock on an object on CWD, server sets NULL mode > lock for the object. That NULL mode lock indicates that the object is > > > > > I would also like to note that I had a discussion with Linus at one of > > the kernel workshops in Ottawa maybe almost 4-5 years ago. First Linus > > attacked the idea of using file identifiers - he suggested that doing > > everything with pathnames was better (which is what InterMezzo did). > > When we explained to him that this requires locking all parents he began > > to see the problems we had with this and understood the locking at the > > fid/name level that we use in Lustre. I found little resistance when I > > mentioned to him that for this model the VFS does not have a correct > > implementation of getcwd, unless the dcache is kept current. > > > > UCSC has received funding from the National Labs and now been turned > > into a peta-scale I/O institute I believe did more results on file > > systems implemented with pathnames. Some things are beautiful and easy > > with pathnames, but others get really ugly, and so far I don't see this > > displacing fid ideas that govern NFS, AFS and Lustre. > > > > - Peter - > > > > Alex Zhuravlev wrote: > > > Hi, > > > > > > couple comments inline ... > > > > > > Vladimir V. Saveliev wrote: > > > > > >> The example shows the details: > > >> > > >> 1. A client C1 holds ordinary lock on an object O1 (it did > > >> chdir(/a/b/c/d/e), O1 is inode of /a/b/c/d/e). C1 is idle now. > > >> > > > > > > chdir doesn't return any lock. should it? > > > > > > > > >> 2. Another client C2 does ls -ld /a/b/c/d/e, MD server sends a BAST to > > >> C1 and C1 cancels the lock of O1. > > >> > > >> 3. C2 is not interested anymore in O1, so it drops the lock. > > >> > > >> 4. Yet another client C3 acquires subtree lock on /a/b and caches and > > >> possibly changes (if under WBC) objects under /a/b including /a/b/c/d/e > > >> (the object O1). The key issue is that MDS neither remembers about O1 on > > >> C1 nor keeps information about objects cached by a client under a > > >> subtree lock. > > >> > > >> 5. Now C1 continues with stat(``.''). It sees that the lock on O1 is > > >> canceled, so it goes to MD server and acquires the lock on O1. > > >> > > >> Now we have: > > >> uptodate O1 is on C3; > > >> MDS has a request for O1 from C1 and MDS can not easily deterimine > > >> whether O1 is under any subtree lock. In order to find whether the lock > > >> conflict exists we need to have a special procedure. It is referred to > > >> as path re-validation. > > >> > > >> The main thing to be done on path re-validation is to look for above > > >> subtree lock. While it is probably doable, the path re-validation is not > > >> going to be very efficient (especially in case of CMD). I can provide > > >> more details if necessary. > > >> > > >> > > >> However, it looks like it is possible to avoid having to do path > > >> re-validation completely. > > >> > > >> > > >> The problem appears when clients request locks on objects directly, > > >> without doing downward lookup through a directory structure. > > >> This happens, for example, when clients access directly components of > > >> current working directories (CWDs). > > >> If a client cancels locks on such objects (either due to a BAST or > > >> voluntary) - it has to go through the path re-validation later. > > >> > > >> Objects to which a client may access directly appear in result of normal > > >> downward lookup. Therefore, they were locked, and their locks can be > > >> canceled. That is the point where we can take care about future accesses > > >> without re-validation. > > >> On canceling a lock of directly accessible object we have to inform DLM > > >> that the ordinary locking has to be used for that object. That will > > >> prevent the object from getting cached under a subtree lock. > > >> > > > > > > 1) there may be thousands of such objects (many processes on many nodes) > > > 2) it's not clear when to enable this back > > > > > > > > >> The problem with this schema is to determine which objects are directly > > >> accessible. But wouldn't solving it be worth doing given that it may > > >> help to avoid path re-validation deal. > > >> > > >> Any comments are welcome. > > >> > > > > > > thanks, Alex > > > _______________________________________________ > > > Lustre-devel mailing list > > > Lustre-devel at lists.lustre.org > > > http://lists.lustre.org/mailman/listinfo/lustre-devel > > > From Peter.Braam at Sun.COM Fri Feb 29 03:20:56 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Thu, 28 Feb 2008 20:20:56 -0700 Subject: [Lustre-devel] Andrew's I/O graphs Message-ID: I see some worrying dips in the graphs - can our I/O specialists comment on which ones are understood and which are not? - peter - On 2/25/08 2:44 PM, "Andrew C. Uselton" wrote: > Howdy all, > I'd been in conversation with Cliff White over the last few weeks, and > he'd expressed an interest in having me post a draft of a report I've > been working on. If you've already heard of it here it is. For those > who hadn't I'll try to describe it briefly. > > In December I assisted with some Lustre benchmark tests on the > Franklin Cray XT here at NERSC. Since then I've tried to summarize our > analysis and results. The attached pdf is a draft of that summary. The > introduction is almost completely useless, so feel free to skip (unless > you want to have a laugh at the author's expense). Section 3 has the > main details about what we observed and what we thought about it. > Section 2 may be amusing for those (like me) who care about methodology. > Cheers, > Andrew > ------ End of Forwarded Message -------------- next part -------------- An HTML attachment was scrubbed... URL: From Peter.Braam at Sun.COM Fri Feb 29 04:30:45 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Thu, 28 Feb 2008 21:30:45 -0700 Subject: [Lustre-devel] Lustre HSM HLD draft In-Reply-To: <47C594F8.20409@cea.fr> Message-ID: The discussion below about the API's is a standard element of data abstraction taught in advanced programming courses (see e.g. Abelson et. al. Structure and Interpretation of Computer Programs (SICP)). From this one concludes that the coordinator and agents will use abstract data types and call abstract methods that accommodate multiple: - source and destination descriptors for the data - data movers implementing the methods to move data If you proceed along the lines you outline you will get a big matrix of movers and data types to keep track of. If you follow my approach you will encapsulate things much more cleanly. Think in terms of virtual classes data movers acting on source and destination objects. - peter - On 2/27/08 9:51 AM, "Aurelien Degremont" wrote: > > Peter J Braam a écrit : >>> Coordinator: >>> >>> This element will manage migration externally (HSM) and internally of >>> Lustre (space balancing?). Is the current API acceptable (specific >>> calls for external migration, and other ones for internal migration)? >> I would like to see a parameter indicating what agent will be used and >> keep all other parameters the same. > > Agreed. > >>> The best way could have been to have generic call for migration, but >>> we must also have generic objects to describe the migration sources >>> and destinations and those are not simples. >> For migration to and from external sources, Lustre must already manage >> this data in an extended attribute (e.g. to describe the file on tape to >> which a Lustre file was migrated). This data is opaque to Lustre and >> can be passed as a blob. >>> It seems, to support Lustre internal migration, you have planned to >>> implement specific Agents which will reside on OST. >> To avoid many complications involving locks, we decided that even the >> agents used for internal migrations will layer on the file system. The >> Lustre file system will be mounted on the OST's and it will use the >> "LOLND" to transport the data efficiently between the OST process and >> the client file system cache. In the internal case source and >> destination lie in Lustre in the HSM case only one of them. >> >> As a result I believe these two cases are closer together than you may >> think, and should be one "type". > > > If we unify the API, we must have a way to request some data movement like: > > copy elemA in placeP > copy elemA,stored in placeP bak into Lustre > copy elemA into placeC > move elemB into elemB > > > The elem could be unified using Lustre FID, but the places could be an > external storage, or a precise OST. If we want a unify API, the API call > should manipulate a generic object which could describe a Lustre storage > element (ost) or a external storage (hsm,...) > > ie: > struct storage_place { > ... > } > copy(fid,storage_place*) > move(fid,storage_place*) > > and their is some specific cases to handle. The other possibity: > > ext_copyout(fid, external storage) > ext_copyin(fid, external object) > int_copy(fid, fid, ost) > int_move(fid, fid, ost) > > I think this one, even if the design is not the most beautiful one, if > the easiest one. > > Instead you want to create some new generic objects to manipulate lustre > object data and generic storage areas, the second case is the best one IMO. > From Vladimir.Saveliev at Sun.COM Fri Feb 29 15:05:12 2008 From: Vladimir.Saveliev at Sun.COM (Vladimir V. Saveliev) Date: Fri, 29 Feb 2008 17:05:12 +0200 Subject: [Lustre-devel] subtree locks and path re-validation avoidance In-Reply-To: <47BF8FBA.3090705@sun.com> References: <1203080062.19279.91.camel@linux.site> <47BD7D0D.6090906@sun.com> <47BF8FBA.3090705@sun.com> Message-ID: <1204297512.24711.20.camel@linux.site> Hello On Fri, 2008-02-22 at 20:15 -0700, Peter J Braam wrote: > I'd like to make a suggestion to perhaps immediately find the right > primitives for getcwd to return a reasonably correct pathname in > Lustre. Peter, would you say a bit more about that: currently, there is nothing a filesystem can do in linux's getcwd. It simply returns instant dcache path from "." to "/". > I believe this is the simplest case where I have seen pathname > revalidation being important. In the context of that example the > subtree lock discussion might gain more clarity. > > I would also like to note that I had a discussion with Linus at one of > the kernel workshops in Ottawa maybe almost 4-5 years ago. First Linus > attacked the idea of using file identifiers - he suggested that doing > everything with pathnames was better (which is what InterMezzo did). > When we explained to him that this requires locking all parents he began > to see the problems we had with this and understood the locking at the > fid/name level that we use in Lustre. I found little resistance when I > mentioned to him that for this model the VFS does not have a correct > implementation of getcwd, unless the dcache is kept current. > > UCSC has received funding from the National Labs and now been turned > into a peta-scale I/O institute I believe did more results on file > systems implemented with pathnames. Some things are beautiful and easy > with pathnames, but others get really ugly, and so far I don't see this > displacing fid ideas that govern NFS, AFS and Lustre. > Best regards, Vladimir From adilger at sun.com Fri Feb 29 21:10:14 2008 From: adilger at sun.com (Andreas Dilger) Date: Fri, 29 Feb 2008 13:10:14 -0800 Subject: [Lustre-devel] Thoughts on benchmarking In-Reply-To: References: <47C336C8.4090108@lbl.gov> Message-ID: <20080229211014.GQ2997@webber.adilger.int> On Feb 28, 2008 20:20 -0700, Peter J. Braam wrote: > I see some worrying dips in the graphs - can our I/O specialists comment on > which ones are understood and which are not? I think one of the major problems that Andrew discusses at the end of the test runs is described in bug 7365 "Poor performance when files share an OSC". I didn't see anywhere in the paper which version of Lustre was being tested, but I know we did some work to improve the round-robin allocator to make it more uniform in more recent releases, up to a certain extent. That said, getting completely uniform file distribution will still need some effort, because the MDS doesn't do any correlation between create requests (e.g. from a single job, from a single client, etc). > On 2/25/08 2:44 PM, "Andrew C. Uselton" wrote: > > I'd been in conversation with Cliff White over the last few weeks, and > > he'd expressed an interest in having me post a draft of a report I've > > been working on. If you've already heard of it here it is. For those > > who hadn't I'll try to describe it briefly. > > > > In December I assisted with some Lustre benchmark tests on the > > Franklin Cray XT here at NERSC. Since then I've tried to summarize our > > analysis and results. The attached pdf is a draft of that summary. The > > introduction is almost completely useless, so feel free to skip (unless > > you want to have a laugh at the author's expense). Section 3 has the > > main details about what we observed and what we thought about it. > > Section 2 may be amusing for those (like me) who care about methodology. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From acuselton at lbl.gov Fri Feb 29 21:38:21 2008 From: acuselton at lbl.gov (Andrew C. Uselton) Date: Fri, 29 Feb 2008 13:38:21 -0800 Subject: [Lustre-devel] Thoughts on benchmarking In-Reply-To: <20080229211014.GQ2997@webber.adilger.int> References: <47C336C8.4090108@lbl.gov> <20080229211014.GQ2997@webber.adilger.int> Message-ID: <47C87B4D.7070803@lbl.gov> Howdy Andreas, Long time no electron :) The work on Franklin (NERSC's shiny new XT4) uses the Lustre delivered and supported by Cray. I believe it's 1.6.x, but I'd have to ask around to get the details. Is there a way to dig the Lustre version out of a client? I'm at a workshop now. I'll try to address this next week. Note that they updated things on Franklin earlier in February. After that we saw a substantial performance increase. The details of what changed have not been communicated to be. Sometime in the near future I'll be interested to follow up on the work I've written about. Feel free to contribute suggestions of tests you'd be interested in. Cheers, Andrew Andreas Dilger wrote: > On Feb 28, 2008 20:20 -0700, Peter J. Braam wrote: >> I see some worrying dips in the graphs - can our I/O specialists comment on >> which ones are understood and which are not? > > I think one of the major problems that Andrew discusses at the end of > the test runs is described in bug 7365 "Poor performance when files share > an OSC". I didn't see anywhere in the paper which version of Lustre was > being tested, but I know we did some work to improve the round-robin > allocator to make it more uniform in more recent releases, up to a > certain extent. > > That said, getting completely uniform file distribution will still need > some effort, because the MDS doesn't do any correlation between create > requests (e.g. from a single job, from a single client, etc). > >> On 2/25/08 2:44 PM, "Andrew C. Uselton" wrote: >>> I'd been in conversation with Cliff White over the last few weeks, and >>> he'd expressed an interest in having me post a draft of a report I've >>> been working on. If you've already heard of it here it is. For those >>> who hadn't I'll try to describe it briefly. >>> >>> In December I assisted with some Lustre benchmark tests on the >>> Franklin Cray XT here at NERSC. Since then I've tried to summarize our >>> analysis and results. The attached pdf is a draft of that summary. The >>> introduction is almost completely useless, so feel free to skip (unless >>> you want to have a laugh at the author's expense). Section 3 has the >>> main details about what we observed and what we thought about it. >>> Section 2 may be amusing for those (like me) who care about methodology. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > From sss at cray.com Fri Feb 29 22:56:20 2008 From: sss at cray.com (Stephen Sugiyama) Date: Fri, 29 Feb 2008 16:56:20 -0600 Subject: [Lustre-devel] Thoughts on benchmarking In-Reply-To: <47C87B4D.7070803@lbl.gov> References: <47C336C8.4090108@lbl.gov><20080229211014.GQ2997@webber.adilger.int> <47C87B4D.7070803@lbl.gov> Message-ID: <925346A443D4E340BEB20248BAFCDBDF044CCFB6@CFEVS1-IP.americas.cray.com> Hi Andrew and Andreas, In December franklin was running Lustre 1.4.9 (plus various patches). Today franklin is running Lustre 1.4.11. Stephen Sugiyama -----Original Message----- From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Andrew C. Uselton Sent: Friday, February 29, 2008 1:38 PM To: lustre-devel at lists.lustre.org Subject: Re: [Lustre-devel] Thoughts on benchmarking Howdy Andreas, Long time no electron :) The work on Franklin (NERSC's shiny new XT4) uses the Lustre delivered and supported by Cray. I believe it's 1.6.x, but I'd have to ask around to get the details. Is there a way to dig the Lustre version out of a client? I'm at a workshop now. I'll try to address this next week. Note that they updated things on Franklin earlier in February. After that we saw a substantial performance increase. The details of what changed have not been communicated to be. Sometime in the near future I'll be interested to follow up on the work I've written about. Feel free to contribute suggestions of tests you'd be interested in. Cheers, Andrew Andreas Dilger wrote: > On Feb 28, 2008 20:20 -0700, Peter J. Braam wrote: >> I see some worrying dips in the graphs - can our I/O specialists comment on >> which ones are understood and which are not? > > I think one of the major problems that Andrew discusses at the end of > the test runs is described in bug 7365 "Poor performance when files share > an OSC". I didn't see anywhere in the paper which version of Lustre was > being tested, but I know we did some work to improve the round-robin > allocator to make it more uniform in more recent releases, up to a > certain extent. > > That said, getting completely uniform file distribution will still need > some effort, because the MDS doesn't do any correlation between create > requests (e.g. from a single job, from a single client, etc). > >> On 2/25/08 2:44 PM, "Andrew C. Uselton" wrote: >>> I'd been in conversation with Cliff White over the last few weeks, and >>> he'd expressed an interest in having me post a draft of a report I've >>> been working on. If you've already heard of it here it is. For those >>> who hadn't I'll try to describe it briefly. >>> >>> In December I assisted with some Lustre benchmark tests on the >>> Franklin Cray XT here at NERSC. Since then I've tried to summarize our >>> analysis and results. The attached pdf is a draft of that summary. The >>> introduction is almost completely useless, so feel free to skip (unless >>> you want to have a laugh at the author's expense). Section 3 has the >>> main details about what we observed and what we thought about it. >>> Section 2 may be amusing for those (like me) who care about methodology. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > _______________________________________________ Lustre-devel mailing list Lustre-devel at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-devel