From Daire.Byrne at framestore.com Tue Jul 1 11:03:41 2008 From: Daire.Byrne at framestore.com (Daire Byrne) Date: Tue, 1 Jul 2008 12:03:41 +0100 (BST) Subject: [Lustre-devel] Wide area use of Lustre and client caches In-Reply-To: <372618435.977041214909597153.JavaMail.root@dahlback.prod.local> Message-ID: <2027440260.979211214910221062.JavaMail.root@dahlback.prod.local> Peter, I assume the same rational holds for NFS exporting too? I'm toying with the idea of putting lots of RAM in a server and exporting our LustreFS over NFS. We have some workloads which do a lot of seeking through a reasonably small set (~32Gigs worth) of files which may perform better if an NFS server caches the dataset and consequently doesn't have to do any disk seeks. Obviously this is not particularly scalable (cheaply) but in small scale tests it seems to perform better than seeking directly from Lustre. The "open lock" stuff you mention is the work going on in #14975 right? Using Lustre 1.6.5 server/client it seems like I can already get line speed (GigE) reads over NFS for a single file once the Lustre client on the NFS server has cached it. But I have not tested this at scale with many clients and files simultaneously. While we wait for Lustre caching (I assume the work done in #12182 is dead in the water?) this may be the best way for us to deal with heavy seek+read workloads. Our use of SATA based hardware RAID arrays doesn't help our seek performance either. Daire ----- "Peter Braam" wrote: > Wide area use of Lustre and client caches During the LUG I was > approached by a customer who wants to use a Lustre file system at the > far end of a WAN link. Since the situation may be of general interest, > I thought I would post a short report of the discussion here. > > His use pattern was interesting – a number of Windows clients must be > browsing files stored in Lustre in this remote location. It was > expected that the files would be fairly large, would be viewed by > multiple clients, and that few or no modifications would be made. > > After some discussion we proposed a solution that involved a > deployment as follows: > > > > 1. A single Lustre client with lots of RAM. The settings on the > client would be (1) that the memory available for caching by lustre is > large (2) that the number of locks that can be held by this client is > fairly large (3) that this client uses the “open cache”. > 2. A samba server on this Lustre client. > > With the settings above, we can expect that many of the files can be > cached in the Lustre client, hence after the initial read, I/O would > be local in the remote site. With the open file cache enabled, even > the open and close traffic will not go to the servers, but can be > handled by the client. We think that this will lead to a very good > solution, that can work today. > > A refinement is possible, that requires some development. There is a > feature in the Linux kernel to use a disk partition as a cache for a > file system – it is called cachefs. This requires a few hooks in > Lustre to store chunks of files that are transferred to the client > into this cache, and cache invalidation calls to remove them. It > allows us to achieve the same performance as with the solution above, > except that the disk will be a bit slower than memory, but it can also > be much larger. > > We are eagerly awaiting the results of testing this configuration! > > - peter - > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From Peter.Braam at Sun.COM Tue Jul 1 14:56:28 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Tue, 01 Jul 2008 08:56:28 -0600 Subject: [Lustre-devel] Wide area use of Lustre and client caches In-Reply-To: <2027440260.979211214910221062.JavaMail.root@dahlback.prod.local> Message-ID: Yes, it should help a lot - and it should apply in the same way. The open cache is automatically used by the NFS server (which runs in kernel space). Any volunteers to write the cachefs interfaces for us? Peter On 7/1/08 5:03 AM, "Daire Byrne" wrote: > Peter, > > I assume the same rational holds for NFS exporting too? I'm toying with the > idea of putting lots of RAM in a server and exporting our LustreFS over NFS. > We have some workloads which do a lot of seeking through a reasonably small > set (~32Gigs worth) of files which may perform better if an NFS server caches > the dataset and consequently doesn't have to do any disk seeks. Obviously this > is not particularly scalable (cheaply) but in small scale tests it seems to > perform better than seeking directly from Lustre. > > The "open lock" stuff you mention is the work going on in #14975 right? Using > Lustre 1.6.5 server/client it seems like I can already get line speed (GigE) > reads over NFS for a single file once the Lustre client on the NFS server has > cached it. But I have not tested this at scale with many clients and files > simultaneously. > > While we wait for Lustre caching (I assume the work done in #12182 is dead in > the water?) this may be the best way for us to deal with heavy seek+read > workloads. Our use of SATA based hardware RAID arrays doesn't help our seek > performance either. > > Daire > > > ----- "Peter Braam" wrote: > >> Wide area use of Lustre and client caches During the LUG I was >> approached by a customer who wants to use a Lustre file system at the >> far end of a WAN link. Since the situation may be of general interest, >> I thought I would post a short report of the discussion here. >> >> His use pattern was interesting ­ a number of Windows clients must be >> browsing files stored in Lustre in this remote location. It was >> expected that the files would be fairly large, would be viewed by >> multiple clients, and that few or no modifications would be made. >> >> After some discussion we proposed a solution that involved a >> deployment as follows: >> >> >> >> 1. A single Lustre client with lots of RAM. The settings on the >> client would be (1) that the memory available for caching by lustre is >> large (2) that the number of locks that can be held by this client is >> fairly large (3) that this client uses the ³open cache². >> 2. A samba server on this Lustre client. >> >> With the settings above, we can expect that many of the files can be >> cached in the Lustre client, hence after the initial read, I/O would >> be local in the remote site. With the open file cache enabled, even >> the open and close traffic will not go to the servers, but can be >> handled by the client. We think that this will lead to a very good >> solution, that can work today. >> >> A refinement is possible, that requires some development. There is a >> feature in the Linux kernel to use a disk partition as a cache for a >> file system ­ it is called cachefs. This requires a few hooks in >> Lustre to store chunks of files that are transferred to the client >> into this cache, and cache invalidation calls to remove them. It >> allows us to achieve the same performance as with the solution above, >> except that the disk will be a bit slower than memory, but it can also >> be much larger. >> >> We are eagerly awaiting the results of testing this configuration! >> >> - peter - >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel From aurelien.degremont at cea.fr Thu Jul 3 11:43:21 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Thu, 03 Jul 2008 13:43:21 +0200 Subject: [Lustre-devel] How store HSM metadata in MDT ? Message-ID: <486CBB59.4080206@cea.fr> HSM MetaData For Lustre HSM project, it will be needed to store, for each file, a list of information describing how many copies the file has in the HSM, what is their HSM ID, the copy date, and so on. This data could easily reach 500 bytes (I think we will need between 40 and 50 bytes per HSM copy, and we should be able to save at least 10 copies, surely more). The question is: where could we store this data on MDT, in which place (EA?) and how manage this. We had a discussion about this with Andreas and Nathan and it is not very clear what is the best solution here regarding to: - We must keep in mind that there is 2 available backends for MDT: ldiskfs and ZFS and both must be supported here. - EA space is not very wide on ldiskfs and quite used by several other features (stripping, ACL, ...) - Clients will need to read this data and so the RPC mechanism should be available and large enough to handle it. Moreover, we will store a purged data range on OST and MDT. This could easily fit in a EA. What is the possible solutions we have here ? -- Aurelien Degremont CEA From aurelien.degremont at cea.fr Thu Jul 3 16:03:20 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Thu, 03 Jul 2008 18:03:20 +0200 Subject: [Lustre-devel] Quota and Lustre HSM Message-ID: <486CF848.7090904@cea.fr> Hello, We got discussion with Andreas and Nathan concerning the best way to support quota with Lustre HSM. I'd like to discuss and clarify what should be the correct behavior in this case. 1 - Current status Here how the Lustre filesystem will behave with current quota system enabled and an active HSM binding. 1.1 Read on a purged file * User A reads a purged file with enough free quota o The file is copied back from the HSM. o User quota increases without new data created by user. * User A reads a purged file with no enough quota. o The file cannot be copied back from the HSM. o I/O failed (Same issue that when the FS is full with HSM support enabled). 1.2 Automatic space management * One of User A files is purged by the Space Manager, without actions from user A. o User quota decreases without reason from user point of view. 2 - Questions Is that the expected behavior? If not what should be changed? What should be the correct behavior? How this could be implemented? 3 - Possible solutions * Not enough quota When not enough quota is available for filling back a purged file when the user try to read it, a solution could be to trigger purges on the other user files. But it could impossible to find purge candidates. * Improve quota management Another approach could be to not removed purged blocks from user quota. A purged file is still considering using as much space as its current size. This solves all the issues presented before. HSM feature will track the purged window for each file anyway, so we can compute the purged window size. When the user quota is classically compute on a filesystem, we will just need to add it to the window size. (Could be a problem if the file was sparse). -- Aurelien Degremont CEA From Peter.Braam at Sun.COM Thu Jul 3 21:10:19 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Thu, 03 Jul 2008 15:10:19 -0600 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: <486CBB59.4080206@cea.fr> Message-ID: I do understand that we need HSM related metadata, but I learned more from Rick Matthews (cc'd) who is the architect of Sun's ADM project. Now I am not sure I am in agreement with what has been discussed so far. If there is more than one copy in the archive, it would be preferable if the archive could maintain a mapping from the Lustre fid of the file to the archived copies. Associated with the FID of the data would then be a list of archived copies, timestamps etc. Can that be done in HPSS? If not, policy related operations like purging older files etc will become very complex and not scalable. For example, a search to find older files in the archive would require an e2scan operation to find the inodes and then the objects in the archive. If the file system was not available anymore (for whatever reason), it is not even clear that such a purge could still happen. With an archive based database this can be an indexed search in the archive, which is faster and more appropriate. Clearly this has a major impact on how much attribute space we need. Thoughts? Peter On 7/3/08 5:43 AM, "Aurelien Degremont" wrote: > HSM MetaData > > For Lustre HSM project, it will be needed to store, for each file, a > list of information describing how many copies the file has in the HSM, > what is their HSM ID, the copy date, and so on. This data could easily > reach 500 bytes (I think we will need between 40 and 50 bytes per HSM > copy, and we should be able to save at least 10 copies, surely more). > The question is: where could we store this data on MDT, in which place > (EA?) and how manage this. > > We had a discussion about this with Andreas and Nathan and it is not > very clear what is the best solution here regarding to: > - We must keep in mind that there is 2 available backends for MDT: > ldiskfs and ZFS and both must be supported here. > - EA space is not very wide on ldiskfs and quite used by several other > features (stripping, ACL, ...) > - Clients will need to read this data and so the RPC mechanism should be > available and large enough to handle it. > > Moreover, we will store a purged data range on OST and MDT. This could > easily fit in a EA. > > What is the possible solutions we have here ? From aurelien.degremont at cea.fr Fri Jul 4 14:37:31 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Fri, 04 Jul 2008 16:37:31 +0200 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: References: Message-ID: <486E35AB.7010405@cea.fr> Peter Braam a écrit : > If there is more than one copy in the archive, it would be preferable if the > archive could maintain a mapping from the Lustre fid of the file to the > archived copies. Associated with the FID of the data would then be a list > of archived copies, timestamps etc. Do you mean that the HSM will be aware of various versions of one same file, identified in Lustre by a FID ? Or this will be masked by the archiving tool , doing some tricks to simulate it ? > Can that be done in HPSS? HPSS alone cannot do versioning on its files presently. > If not, policy related operations like purging older files etc will become > very complex and not scalable. For example, a search to find older files in > the archive would require an e2scan operation to find the inodes and then > the objects in the archive. If the file system was not available anymore > (for whatever reason), it is not even clear that such a purge could still > happen. > > With an archive based database this can be an indexed search in the archive, > which is faster and more appropriate. By purgin do mean purging in Lustre or in the HSM? There's no issue with purging in Lustre because this do not imply the HSM. And removal of oldest copies in the HSM could be done asynchronously, slowly. I'm not sure I see what you mean here -- Aurelien Degremont CEA From adilger at sun.com Sat Jul 5 16:50:25 2008 From: adilger at sun.com (Andreas Dilger) Date: Sat, 05 Jul 2008 10:50:25 -0600 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: <486E35AB.7010405@cea.fr> References: <486E35AB.7010405@cea.fr> Message-ID: <20080705165025.GI6239@webber.adilger.int> On Jul 04, 2008 16:37 +0200, Aurelien Degremont wrote: > Peter Braam a écrit : > > If there is more than one copy in the archive, it would be preferable if the > > archive could maintain a mapping from the Lustre fid of the file to the > > archived copies. Associated with the FID of the data would then be a list > > of archived copies, timestamps etc. > > Do you mean that the HSM will be aware of various versions of one same > file, identified in Lustre by a FID ? > Or this will be masked by the archiving tool , doing some tricks to > simulate it ? > > > Can that be done in HPSS? > > HPSS alone cannot do versioning on its files presently. When HPSS acts as both backup and HSM, is it still dependent on an external space/backup manager to track all of the files for the filesystem, or does it have a space manager built into it? > > If not, policy related operations like purging older files etc will become > > very complex and not scalable. For example, a search to find older files in > > the archive would require an e2scan operation to find the inodes and then > > the objects in the archive. If the file system was not available anymore > > (for whatever reason), it is not even clear that such a purge could still > > happen. > > > > With an archive based database this can be an indexed search in the archive, > > which is faster and more appropriate. > > By purgin do mean purging in Lustre or in the HSM? Purging old backups of the file in the offline storage (it isn't quite right to call this the HSM at this point, because there are multiple backup copies of the file, not strictly a heirarchy). > There's no issue with purging in Lustre because this do not imply the HSM. > And removal of oldest copies in the HSM could be done asynchronously, > slowly. What manages removal of the older copies in HPSS? If HPSS can purge older files based on policy (leaving at least the most recent copy always), then it would be possible to defer the backup policy to HPSS and Lustre would only ever need to reference a single offline file. Any queries for listing older versions of the file would be passed on from Lustre to HPSS in that case. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Peter.Braam at Sun.COM Sun Jul 6 03:20:53 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Sat, 05 Jul 2008 21:20:53 -0600 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: <20080705165025.GI6239@webber.adilger.int> Message-ID: On 7/5/08 10:50 AM, "Andreas Dilger" wrote: > What manages removal of the older copies in HPSS? If HPSS can purge older > files based on policy (leaving at least the most recent copy always), then > it would be possible to defer the backup policy to HPSS and Lustre would > only ever need to reference a single offline file. Any queries for listing > older versions of the file would be passed on from Lustre to HPSS in that > case. > The point is that there is a proposal here to have multiple pointers PER INODE - that is not a good idea. Regardless of what policies are in place and where they are managed, they should not affect every inode with a new pointer for every backup object of that inode. Peter > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From Peter.Braam at Sun.COM Sun Jul 6 03:24:55 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Sat, 05 Jul 2008 21:24:55 -0600 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: <486E35AB.7010405@cea.fr> Message-ID: On 7/4/08 8:37 AM, "Aurelien Degremont" wrote: > Peter Braam a écrit : >> If there is more than one copy in the archive, it would be preferable if the >> archive could maintain a mapping from the Lustre fid of the file to the >> archived copies. Associated with the FID of the data would then be a list >> of archived copies, timestamps etc. > > Do you mean that the HSM will be aware of various versions of one same > file, identified in Lustre by a FID ? > Or this will be masked by the archiving tool , doing some tricks to > simulate it ? > >> Can that be done in HPSS? > > HPSS alone cannot do versioning on its files presently. But your archiving utility that copies from Lustre to HPSS can maintain database of these objects - no need to store anything in Lustre. > > >> If not, policy related operations like purging older files etc will become >> very complex and not scalable. For example, a search to find older files in >> the archive would require an e2scan operation to find the inodes and then >> the objects in the archive. If the file system was not available anymore >> (for whatever reason), it is not even clear that such a purge could still >> happen. >> >> With an archive based database this can be an indexed search in the archive, >> which is faster and more appropriate. > > By purgin do mean purging in Lustre or in the HSM? The HSM. > There's no issue with purging in Lustre because this do not imply the HSM. > And removal of oldest copies in the HSM could be done asynchronously, > slowly. There is a rule in Lustre - no scanning, ever. This rule will not be broken by HSM. So, you have to move your management of ID's of the archvied copies outside of Lustre, in some database. This will actually save you time - doing this in the MDS will be no fun. The MDS should only get attributes to indicate if and what version of a file is in the archive and a cursor (maybe other information) in relation with ongoing restores. Peter > > I'm not sure I see what you mean here > From lee at sandia.gov Sun Jul 6 19:24:47 2008 From: lee at sandia.gov (Lee Ward) Date: Sun, 06 Jul 2008 13:24:47 -0600 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: References: Message-ID: <1215372287.17129.66.camel@wheel> Are you all talking about HSM, really, or simply backup? If backup, read no further. If HSM, then, do you intend that the user be allowed to specify *which* version of the file content is desired? If yes and you also want the standard API and utilities to function, seamlessly, then the version must be exposed in the name space, no? I.e. For any file named "foo" with 3 versions, for instance, there would be foo;1, foo;2, foo;3, and "foo" which is an alias for "foo;1". If no, then, you'll have to craft a special API that will motivate special tools. However, HPSS already has this API and set of tools so what's the point? Wouldn't it be better to just modify HPSS to understand versions? If HSM, then, do you intend that two users might be allowed to work with two, or more, versions of the file content simultaneously? If yes then same problem as above since those two versions might need to be in the same directory, at the same time, right? No matter what you do, you have problems that can't be resolved when mixing a POSIX name space with file versions, I believe. Since POSIX reserves no characters you can't pick a scheme that includes version information in the name without at least being confusing and the API provides no other way to specify the version, no? My personal choice would be to shy off direct version support by the native file system. It doesn't seem to have a reasonable solution without involving the user somehow to specify names or naming schemes. That kind of involvement just begs for a special utility and, once there, relieves the file system of the need to support any but the most recent version itself, anyway. --Lee On Sat, 2008-07-05 at 21:24 -0600, Peter Braam wrote: > > > On 7/4/08 8:37 AM, "Aurelien Degremont" wrote: > > > Peter Braam a écrit : > >> If there is more than one copy in the archive, it would be preferable if the > >> archive could maintain a mapping from the Lustre fid of the file to the > >> archived copies. Associated with the FID of the data would then be a list > >> of archived copies, timestamps etc. > > > > Do you mean that the HSM will be aware of various versions of one same > > file, identified in Lustre by a FID ? > > Or this will be masked by the archiving tool , doing some tricks to > > simulate it ? > > > >> Can that be done in HPSS? > > > > HPSS alone cannot do versioning on its files presently. > > But your archiving utility that copies from Lustre to HPSS can maintain > database of these objects - no need to store anything in Lustre. > > > > > > > >> If not, policy related operations like purging older files etc will become > >> very complex and not scalable. For example, a search to find older files in > >> the archive would require an e2scan operation to find the inodes and then > >> the objects in the archive. If the file system was not available anymore > >> (for whatever reason), it is not even clear that such a purge could still > >> happen. > >> > >> With an archive based database this can be an indexed search in the archive, > >> which is faster and more appropriate. > > > > By purgin do mean purging in Lustre or in the HSM? > > The HSM. > > > There's no issue with purging in Lustre because this do not imply the HSM. > > And removal of oldest copies in the HSM could be done asynchronously, > > slowly. > > There is a rule in Lustre - no scanning, ever. This rule will not be broken > by HSM. > > So, you have to move your management of ID's of the archvied copies outside > of Lustre, in some database. This will actually save you time - doing this > in the MDS will be no fun. > > The MDS should only get attributes to indicate if and what version of a file > is in the archive and a cursor (maybe other information) in relation with > ongoing restores. > > Peter > > > > > > I'm not sure I see what you mean here > > > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Peter.Braam at Sun.COM Sun Jul 6 22:53:02 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Sun, 06 Jul 2008 16:53:02 -0600 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: <1215372287.17129.66.camel@wheel> Message-ID: Lee - Thank you for this clear explanation. If solely the HSM can store multiple versions, we have already some difficulties. One might imagine setting a particular version in the HSM as the primary one, meaning that this primary one will be transparently restored or that a pre-staging utility will select this by default. If the file is fully absent in the file system staging or restoring it will work correctly. However, if a part of the file remains in the file system, this HSM versioning becomes complicated because the file will again have to remember what HSM versions the fragments belong to, and we are almost back where we were. I think the emails so far make it clear that we don't want to have one Lustre inode be associated with multiple objects in the HSM. If the HSM system is used as a backup then the restore operations will have user or operator involvement and this objection to storing multiple versions in the HSM does not apply. However, the we still don't' want to store a pointer to each version in the file system, that belongs in the HSM/backup metadata store. However, I don't want to end the discussion right here. With DMU (or otherwise) we will get file systems where snapshots become possible and common, and these snapshots will contain different versions of the same file. The way the namespace distinguishes these is that in the pair (fsid, fid) the fsid is different for each snapshot. So probably the id in the HSM should allow for an fsid component. Now DMU snapshot versions of one inode share blocks, and this leads to the question if/how we can efficiently share blocks in the HSM also. This discussion would probably equally apply to upcoming "dedup" efforts for the DMU, which the virtualization and "email attachment" community think are very important. Rick, Jeff - how will we handle this? Peter On 7/6/08 1:24 PM, "Lee Ward" wrote: > Are you all talking about HSM, really, or simply backup? > > If backup, read no further. > > If HSM, then, do you intend that the user be allowed to specify *which* > version of the file content is desired? > > If yes and you also want the standard API and utilities to function, > seamlessly, then the version must be exposed in the name space, no? I.e. > For any file named "foo" with 3 versions, for instance, there would be > foo;1, foo;2, foo;3, and "foo" which is an alias for "foo;1". > > If no, then, you'll have to craft a special API that will motivate > special tools. However, HPSS already has this API and set of tools so > what's the point? Wouldn't it be better to just modify HPSS to > understand versions? > > If HSM, then, do you intend that two users might be allowed to work with > two, or more, versions of the file content simultaneously? > > If yes then same problem as above since those two versions might need to > be in the same directory, at the same time, right? > > No matter what you do, you have problems that can't be resolved when > mixing a POSIX name space with file versions, I believe. Since POSIX > reserves no characters you can't pick a scheme that includes version > information in the name without at least being confusing and the API > provides no other way to specify the version, no? > > My personal choice would be to shy off direct version support by the > native file system. It doesn't seem to have a reasonable solution > without involving the user somehow to specify names or naming schemes. > That kind of involvement just begs for a special utility and, once > there, relieves the file system of the need to support any but the most > recent version itself, anyway. > > --Lee > > On Sat, 2008-07-05 at 21:24 -0600, Peter Braam wrote: >> >> >> On 7/4/08 8:37 AM, "Aurelien Degremont" wrote: >> >>> Peter Braam a écrit : >>>> If there is more than one copy in the archive, it would be preferable if >>>> the >>>> archive could maintain a mapping from the Lustre fid of the file to the >>>> archived copies. Associated with the FID of the data would then be a list >>>> of archived copies, timestamps etc. >>> >>> Do you mean that the HSM will be aware of various versions of one same >>> file, identified in Lustre by a FID ? >>> Or this will be masked by the archiving tool , doing some tricks to >>> simulate it ? >>> >>>> Can that be done in HPSS? >>> >>> HPSS alone cannot do versioning on its files presently. >> >> But your archiving utility that copies from Lustre to HPSS can maintain >> database of these objects - no need to store anything in Lustre. >> >> >>> >>> >>>> If not, policy related operations like purging older files etc will become >>>> very complex and not scalable. For example, a search to find older files >>>> in >>>> the archive would require an e2scan operation to find the inodes and then >>>> the objects in the archive. If the file system was not available anymore >>>> (for whatever reason), it is not even clear that such a purge could still >>>> happen. >>>> >>>> With an archive based database this can be an indexed search in the >>>> archive, >>>> which is faster and more appropriate. >>> >>> By purgin do mean purging in Lustre or in the HSM? >> >> The HSM. >> >>> There's no issue with purging in Lustre because this do not imply the HSM. >>> And removal of oldest copies in the HSM could be done asynchronously, >>> slowly. >> >> There is a rule in Lustre - no scanning, ever. This rule will not be broken >> by HSM. >> >> So, you have to move your management of ID's of the archvied copies outside >> of Lustre, in some database. This will actually save you time - doing this >> in the MDS will be no fun. >> >> The MDS should only get attributes to indicate if and what version of a file >> is in the archive and a cursor (maybe other information) in relation with >> ongoing restores. >> >> Peter >> >> >>> >>> I'm not sure I see what you mean here >>> >> >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> > > From Peter.Braam at Sun.COM Mon Jul 7 23:44:09 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Mon, 07 Jul 2008 17:44:09 -0600 Subject: [Lustre-devel] Lustre messages Message-ID: We spent a fair amount of time making our messages efficient ­ here is a different approach. Is it worth exploring? Peter http://code.google.com/apis/protocolbuffers/docs/overview.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From Tom.Wang at Sun.COM Tue Jul 8 01:00:53 2008 From: Tom.Wang at Sun.COM (Tom.Wang) Date: Mon, 07 Jul 2008 21:00:53 -0400 Subject: [Lustre-devel] Lustre ADIO collective write HLD Message-ID: <4872BC45.50403@sun.com> Hello, Here is the HLD for Lustre ADIO collective write driver, which is currently being co-developed by SUN Lustre group and Oak Ridge National Lab. You might be interested in some diagrams in this HLD. Welcome to any comments. -- Regards, Tom Wangdi -- Sun Lustre Group System Software Engineer http://www.sun.com -------------- next part -------------- A non-text attachment was scrubbed... Name: HLD-ADIO-LUSTRE-COLLECTIVE-WRITE.pdf Type: application/pdf Size: 587363 bytes Desc: not available URL: From aurelien.degremont at cea.fr Tue Jul 8 08:52:01 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Tue, 08 Jul 2008 10:52:01 +0200 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: <1215372287.17129.66.camel@wheel> References: <1215372287.17129.66.camel@wheel> Message-ID: <48732AB1.10802@cea.fr> Lee Ward a écrit : > If HSM, then, do you intend that the user be allowed to specify *which* > version of the file content is desired? User could say: "overwrite the current version of this file with this older copies which was made few time ago." -The current file content is lost. -That is the only way to access the older copies content. There is no namespace tricks, no huge API changes, always one version of a file in Lustre, just few functions added to 'lfs' command. The purpose is just, using the HSM infrastructure, simply add few feature to help people asking us for backup features, but this will not be a true backup system. This kind of utility requires much more development. -- Aurelien Degremont CEA From Richard.Matthews at Sun.COM Tue Jul 8 12:06:09 2008 From: Richard.Matthews at Sun.COM (Rick Matthews) Date: Tue, 08 Jul 2008 07:06:09 -0500 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: References: Message-ID: <48735831.9080508@Sun.COM> Peter and Lee, Lee, you are correct when pointing out the versioning of a file with a backup copy is a backup style function. One desirable to users of backup ans some HSM products, but still primarily driven by the "coincidence" that older copies remain, and reference to them may be desired. (In most instances, these references are used to either "restore this copy to that directory" or "restore this directory tree to its prior state".) So, while primarily a backup function, one that if an HSM is the basis for backup copies may be important in the future. HSM as the basis of backup copies is a desirable trait IMHO. The HSM is already retaining an instance of the file, one which could easily be captured as a "backup" copy. That said, HSM and snapshot seems to bring a better mix, particularly to the user. A snapshot of the file system presents a consistent view, and the backup would only need to include data (metadata) from files previously resident in the HSM. As for HSM and deduplication, I see the deduplication being an optimization tradeoff with consumed space. In a relatively expensive random access media (like disk), deduplication provides a reduced total data footprint while not affecting the retrieval rate significantly. When the media is sequentially oriented and relatively less expensive to have (like tape), deduplication seems to not make as much sense. So, I see deduplication as important on disk based archive copies, and not all that useful in tape archiving. Of course, tape striping is important, but is still a sequential store/retrieve. Also, if it is convenient to deduplicate full sequential images of a file (while not violating a numbers of copies policy), that should be done on the sequentially oriented media. There may also be some policy (sequential affinity) reasons where even the full image deduplication is not desirable. Thank you for letting me participate in this discussion. -- Rick Peter Braam wrote: > Lee - Thank you for this clear explanation. > > If solely the HSM can store multiple versions, we have already some > difficulties. One might imagine setting a particular version in the HSM as > the primary one, meaning that this primary one will be transparently > restored or that a pre-staging utility will select this by default. > > If the file is fully absent in the file system staging or restoring it will > work correctly. However, if a part of the file remains in the file system, > this HSM versioning becomes complicated because the file will again have to > remember what HSM versions the fragments belong to, and we are almost back > where we were. > > I think the emails so far make it clear that we don't want to have one > Lustre inode be associated with multiple objects in the HSM. > > If the HSM system is used as a backup then the restore operations will have > user or operator involvement and this objection to storing multiple versions > in the HSM does not apply. However, the we still don't' want to store a > pointer to each version in the file system, that belongs in the HSM/backup > metadata store. > > However, I don't want to end the discussion right here. > > With DMU (or otherwise) we will get file systems where snapshots become > possible and common, and these snapshots will contain different versions of > the same file. The way the namespace distinguishes these is that in the > pair (fsid, fid) the fsid is different for each snapshot. So probably the > id in the HSM should allow for an fsid component. > > Now DMU snapshot versions of one inode share blocks, and this leads to the > question if/how we can efficiently share blocks in the HSM also. This > discussion would probably equally apply to upcoming "dedup" efforts for the > DMU, which the virtualization and "email attachment" community think are > very important. > > Rick, Jeff - how will we handle this? > > Peter > > > > > > On 7/6/08 1:24 PM, "Lee Ward" wrote: > > >> Are you all talking about HSM, really, or simply backup? >> >> If backup, read no further. >> >> If HSM, then, do you intend that the user be allowed to specify *which* >> version of the file content is desired? >> >> If yes and you also want the standard API and utilities to function, >> seamlessly, then the version must be exposed in the name space, no? I.e. >> For any file named "foo" with 3 versions, for instance, there would be >> foo;1, foo;2, foo;3, and "foo" which is an alias for "foo;1". >> >> If no, then, you'll have to craft a special API that will motivate >> special tools. However, HPSS already has this API and set of tools so >> what's the point? Wouldn't it be better to just modify HPSS to >> understand versions? >> >> If HSM, then, do you intend that two users might be allowed to work with >> two, or more, versions of the file content simultaneously? >> >> If yes then same problem as above since those two versions might need to >> be in the same directory, at the same time, right? >> >> No matter what you do, you have problems that can't be resolved when >> mixing a POSIX name space with file versions, I believe. Since POSIX >> reserves no characters you can't pick a scheme that includes version >> information in the name without at least being confusing and the API >> provides no other way to specify the version, no? >> >> My personal choice would be to shy off direct version support by the >> native file system. It doesn't seem to have a reasonable solution >> without involving the user somehow to specify names or naming schemes. >> That kind of involvement just begs for a special utility and, once >> there, relieves the file system of the need to support any but the most >> recent version itself, anyway. >> >> --Lee >> >> On Sat, 2008-07-05 at 21:24 -0600, Peter Braam wrote: >> >>> On 7/4/08 8:37 AM, "Aurelien Degremont" wrote: >>> >>> >>>> Peter Braam a écrit : >>>> >>>>> If there is more than one copy in the archive, it would be preferable if >>>>> the >>>>> archive could maintain a mapping from the Lustre fid of the file to the >>>>> archived copies. Associated with the FID of the data would then be a list >>>>> of archived copies, timestamps etc. >>>>> >>>> Do you mean that the HSM will be aware of various versions of one same >>>> file, identified in Lustre by a FID ? >>>> Or this will be masked by the archiving tool , doing some tricks to >>>> simulate it ? >>>> >>>> >>>>> Can that be done in HPSS? >>>>> >>>> HPSS alone cannot do versioning on its files presently. >>>> >>> But your archiving utility that copies from Lustre to HPSS can maintain >>> database of these objects - no need to store anything in Lustre. >>> >>> >>> >>>> >>>>> If not, policy related operations like purging older files etc will become >>>>> very complex and not scalable. For example, a search to find older files >>>>> in >>>>> the archive would require an e2scan operation to find the inodes and then >>>>> the objects in the archive. If the file system was not available anymore >>>>> (for whatever reason), it is not even clear that such a purge could still >>>>> happen. >>>>> >>>>> With an archive based database this can be an indexed search in the >>>>> archive, >>>>> which is faster and more appropriate. >>>>> >>>> By purgin do mean purging in Lustre or in the HSM? >>>> >>> The HSM. >>> >>> >>>> There's no issue with purging in Lustre because this do not imply the HSM. >>>> And removal of oldest copies in the HSM could be done asynchronously, >>>> slowly. >>>> >>> There is a rule in Lustre - no scanning, ever. This rule will not be broken >>> by HSM. >>> >>> So, you have to move your management of ID's of the archvied copies outside >>> of Lustre, in some database. This will actually save you time - doing this >>> in the MDS will be no fun. >>> >>> The MDS should only get attributes to indicate if and what version of a file >>> is in the archive and a cursor (maybe other information) in relation with >>> ongoing restores. >>> >>> Peter >>> >>> >>> >>>> I'm not sure I see what you mean here >>>> >>>> >>> _______________________________________________ >>> Lustre-devel mailing list >>> Lustre-devel at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>> >>> >> > > > -- --------------------------------------------------------------------- Rick Matthews email: Rick.Matthews at sun.com Sun Microsystems, Inc. phone:+1(651) 554-1518 1270 Eagan Industrial Road phone(internal): 54418 Suite 160 fax: +1(651) 554-1540 Eagan, MN 55121-1231 USA main: +1(651) 554-1500 --------------------------------------------------------------------- From Eric.Mei at Sun.COM Tue Jul 8 17:27:40 2008 From: Eric.Mei at Sun.COM (Eric Mei) Date: Tue, 08 Jul 2008 11:27:40 -0600 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: <48728D97.1020704@psc.edu> References: <48728D97.1020704@psc.edu> Message-ID: <4873A38C.4040507@sun.com> Hi Ben, Benjamin Bennett wrote: > Hi Eric, > > If you could give me your input on something I'd greatly appreciate > it. Or, if I should just sent this to -devel let me know... Yes you can always send to -devel for open discussions. I'm CCing it. > For Lustre-WAN across TeraGrid we were hoping to distribute OSSs > across several resource providers (sites), leveraging existing kerberos > infrastructure, placing OSSs in each resource provider's local kerberos > realm, and the MDS in the teragrid realm. Unfortunately, MDT -> OST > connections will not allow the MDT and OST to be in different realms, > since an OSS considers an MDS to be anyone holding a lustre_mds > principal in their local realm. > > This also seems undesirable within a single-realm where multiple > lustre clusters may exist, as an OSS in cluster A will trust an MDS for > cluster B, and an OSS for cluster B will trust an MDS for cluster A. > > My first thought was to add functionality to tell the OSSs lsvcgssd > what the trusted MDS principals should be (local or not). Do you have > any thoughts on this? We just never thought the usage that OSS could locate in multiple realms. I agree with you in this case, we can make OSS configurable to only accept designated lustre_mds principals, local or remote. I'v questions just for curiosity: 1) is the benefit of cross site OSSs about bandwidth? 2) in the future with CMD (clustered metadata, multiple MDS nodes), would it be useful to distribute MDSs across multiple site as well? Thanks -- Eric From Peter.Braam at Sun.COM Tue Jul 8 17:41:10 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Tue, 08 Jul 2008 11:41:10 -0600 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: <48732AB1.10802@cea.fr> Message-ID: I think we have come to the following conclusions: 1. The HSM or a database associated with it implements a table to map FIDs to stored HSM versions of a file, with other metadata it may need to maintain its archives. 2. An HSM utility can query and learn about the versions stored for a fid (or file name). A "restore" function can copy any version out of the HSM and place it in the file system. This is similar to restoring a file from a backup archive. 3. The file system only has attributes to indicate the state of the primary archived copy (probably the last fully archived copy of the file), and can retrieve that file on demand (without user intervention). 4. The HSM database will allow files in snapshots to be encoded with (fsid, fid) or something similar. 5. for now we ignore block level dedup in the HSM Can the owner of the HLD make updates? Please also read on - I have some more questions below. On 7/8/08 2:52 AM, "Aurelien Degremont" wrote: > Lee Ward a écrit : >> If HSM, then, do you intend that the user be allowed to specify *which* >> version of the file content is desired? > > User could say: > "overwrite the current version of this file with this older copies > which was made few time ago." > > -The current file content is lost. > -That is the only way to access the older copies content. Yes, that is reasonable. > There is no namespace tricks, no huge API changes, always one version of > a file in Lustre, just few functions added to 'lfs' command. NO - this will not be an lfs command. This is an HSM command. > The purpose is just, using the HSM infrastructure, simply add few > feature to help people asking us for backup features, but this will not > be a true backup system. This kind of utility requires much more > development. I think it would be good to review one more time the following aspects of the design: 1. how is a bare metal restore arranged (ie. How is metadata moved into the HSM)? Can this restore put files in a file system different than Lustre? 2. how are small files grouped then "tar'd up" and how are we setting the attributes of the inodes of the files that have been placed in the HSM after this? How does the index entry for the fids in the HSM database function? 3. how are multiple coordinators and agents utilized to distribute load so that the HSM can keep up with massive small file creation? For all of these we have seen sketchy answers in the past, let's dig in and make sure that we have this right. Regards, Peter From Peter.Braam at Sun.COM Tue Jul 8 17:43:39 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Tue, 08 Jul 2008 11:43:39 -0600 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: <4873A38C.4040507@sun.com> Message-ID: Yes, it will be very important that we can separate OST's/MDT's widely. But placing them in different realms, I'm not sure about. Can PSC explain what administrative model warrants that? Why can a remote OST not be part of the realm of the MDS that controls it? Peter On 7/8/08 11:27 AM, "Eric Mei" wrote: > Hi Ben, > > Benjamin Bennett wrote: >> Hi Eric, >> >> If you could give me your input on something I'd greatly appreciate >> it. Or, if I should just sent this to -devel let me know... > > Yes you can always send to -devel for open discussions. I'm CCing it. > >> For Lustre-WAN across TeraGrid we were hoping to distribute OSSs >> across several resource providers (sites), leveraging existing kerberos >> infrastructure, placing OSSs in each resource provider's local kerberos >> realm, and the MDS in the teragrid realm. Unfortunately, MDT -> OST >> connections will not allow the MDT and OST to be in different realms, >> since an OSS considers an MDS to be anyone holding a lustre_mds >> principal in their local realm. >> >> This also seems undesirable within a single-realm where multiple >> lustre clusters may exist, as an OSS in cluster A will trust an MDS for >> cluster B, and an OSS for cluster B will trust an MDS for cluster A. >> >> My first thought was to add functionality to tell the OSSs lsvcgssd >> what the trusted MDS principals should be (local or not). Do you have >> any thoughts on this? > > We just never thought the usage that OSS could locate in multiple > realms. I agree with you in this case, we can make OSS configurable to > only accept designated lustre_mds principals, local or remote. > > I'v questions just for curiosity: 1) is the benefit of cross site OSSs > about bandwidth? 2) in the future with CMD (clustered metadata, multiple > MDS nodes), would it be useful to distribute MDSs across multiple site > as well? > > Thanks From ben at psc.edu Tue Jul 8 18:41:39 2008 From: ben at psc.edu (Benjamin Bennett) Date: Tue, 08 Jul 2008 14:41:39 -0400 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: References: Message-ID: <4873B4E3.2010609@psc.edu> Peter Braam wrote: > Yes, it will be very important that we can separate OST's/MDT's widely. > > But placing them in different realms, I'm not sure about. Can PSC explain > what administrative model warrants that? Why can a remote OST not be part > of the realm of the MDS that controls it? The OSTs will be distributed among several resource provider organizations, each with their own existing domain name space and kerberos realm. There is also a centrally managed teragrid realm which could be used to provide cross-realm transit between the resource provider realms. With this kerberos authentication infrastructure already in place the issue comes down to that of authorizing a principal as an MDS, the logic of which I believe should be reconsidered regardless of cross-realm issues. Currently an OSS's authz of an MDS is inherent in the name of the principal (lustre_mds/host) so AFAICT one cannot safely run two distinct lustre clusters within a single kerberos realm. Moreover, this makes the assumption that all kerberos admins are knowledgeable enough about lustre to only issue lustre_mds/host principals to entities that should have MDS privileges throughout the entire realm. Please do correct me if I'm wrong here. --ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: OpenPGP digital signature URL: From josephin at psc.edu Tue Jul 8 19:21:01 2008 From: josephin at psc.edu (Josephine Palencia) Date: Tue, 8 Jul 2008 15:21:01 -0400 (EDT) Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: <4873A38C.4040507@sun.com> References: <48728D97.1020704@psc.edu> <4873A38C.4040507@sun.com> Message-ID: On Tue, 8 Jul 2008, Eric Mei wrote: > Hi Ben, > > Benjamin Bennett wrote: >> Hi Eric, >> >> If you could give me your input on something I'd greatly appreciate >> it. Or, if I should just sent this to -devel let me know... > > Yes you can always send to -devel for open discussions. I'm CCing it. > >> For Lustre-WAN across TeraGrid we were hoping to distribute OSSs >> across several resource providers (sites), leveraging existing kerberos >> infrastructure, placing OSSs in each resource provider's local kerberos >> realm, and the MDS in the teragrid realm. Unfortunately, MDT -> OST >> connections will not allow the MDT and OST to be in different realms, >> since an OSS considers an MDS to be anyone holding a lustre_mds >> principal in their local realm. >> >> This also seems undesirable within a single-realm where multiple >> lustre clusters may exist, as an OSS in cluster A will trust an MDS for >> cluster B, and an OSS for cluster B will trust an MDS for cluster A. >> >> My first thought was to add functionality to tell the OSSs lsvcgssd >> what the trusted MDS principals should be (local or not). Do you have >> any thoughts on this? > > We just never thought the usage that OSS could locate in multiple > realms. I agree with you in this case, we can make OSS configurable to > only accept designated lustre_mds principals, local or remote. > > I'v questions just for curiosity: 1) is the benefit of cross site OSSs > about bandwidth? 2) in the future with CMD (clustered metadata, multiple > MDS nodes), would it be useful to distribute MDSs across multiple site > as well? At this point, we only want to be able to test functionality, i.e. create a lustre fs over the WAN with OSS contributions from various remote RP remote while kerb auth is enabled. Eventually one will have to deal with performance issues due to the setup but hopefully when that time comes, other mechanisms will become possible (distributed MDS) to manage the remote OSS's. Thanks, -j > > Thanks > -- > Eric > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Peter.Braam at Sun.COM Tue Jul 8 19:39:35 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Tue, 08 Jul 2008 13:39:35 -0600 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: <4873B4E3.2010609@psc.edu> Message-ID: Hmm. Perhaps there are implementation issues here that overshadow the architecture. To interact with MDS nodes that are part of one file system, the MDS needs to be part of a realm. The MDS performs authorization based on a principal to MDS (i.e. Lustre) user/group database. Within one Lustre file system each MDS MUST HAVE the same user group database. We will likely want to place MDS's distributedly in the longer term future, so take clear note of this: one Kerberos realm owns the entire MDS cluster for a file system. There can be multiple MDS clusters, i.e. Lustre file systems, in a single realm, each serving their own file system. Each Lustre file system can have its own user/group database. No restrictions here. For a given file system the MDS nodes produce capabilities which the OSS nodes use for authorization. It is important that the MDS can maken authenticated RPC's to the OSS nodes in its file system and for this we use Kerberos (this is not a "must have" - it could have been done with a different key sharing mechanism). ==> So the first issue you have to become clear about is how you authorize an MDS to contact one of its OSS nodes, wherever these are place. Similarly the Kerberos connections are used by the clients to connect to the OSS, but they are not used to authenticate anything (but optionally the node), they are used merely to provide privacy and/or authenticity for transporting data between the client and the OSS nodes. With relatively little effort this could be done without Kerberos at all, on the other hand, probably using Kerberos for this leads to a more easily understood architecture. So, to repeat, the authorization uses capabilities, which authenticate the requestor and contain authorization information, independent of a server user/group database on the OSS. ==> The second issue you need to be clear about is how you authenticate client NODES (NOT users) to OSS nodes. Peter On 7/8/08 12:41 PM, "Benjamin Bennett" wrote: > Peter Braam wrote: >> Yes, it will be very important that we can separate OST's/MDT's widely. >> >> But placing them in different realms, I'm not sure about. Can PSC explain >> what administrative model warrants that? Why can a remote OST not be part >> of the realm of the MDS that controls it? > > The OSTs will be distributed among several resource provider > organizations, each with their own existing domain name space and > kerberos realm. There is also a centrally managed teragrid realm which > could be used to provide cross-realm transit between the resource > provider realms. With this kerberos authentication infrastructure > already in place the issue comes down to that of authorizing a principal > as an MDS, the logic of which I believe should be reconsidered > regardless of cross-realm issues. > > Currently an OSS's authz of an MDS is inherent in the name of the > principal (lustre_mds/host) so AFAICT one cannot safely run two distinct > lustre clusters within a single kerberos realm. Moreover, this makes > the assumption that all kerberos admins are knowledgeable enough about > lustre to only issue lustre_mds/host principals to entities that should > have MDS privileges throughout the entire realm. Please do correct me > if I'm wrong here. > > > --ben > From adilger at sun.com Tue Jul 8 20:38:05 2008 From: adilger at sun.com (Andreas Dilger) Date: Tue, 08 Jul 2008 14:38:05 -0600 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: <4873A38C.4040507@sun.com> References: <48728D97.1020704@psc.edu> <4873A38C.4040507@sun.com> Message-ID: <20080708203805.GS6239@webber.adilger.int> Benjamin Bennett wrote: > For Lustre-WAN across TeraGrid we were hoping to distribute OSSs > across several resource providers (sites), leveraging existing kerberos > infrastructure, placing OSSs in each resource provider's local kerberos > realm, and the MDS in the teragrid realm. Unfortunately, MDT -> OST > connections will not allow the MDT and OST to be in different realms, > since an OSS considers an MDS to be anyone holding a lustre_mds > principal in their local realm. As an FYI, you should follow the work being done on OST pools (bug 14836 and its decendants), which will allow users to specify subsets of the OSTs to store files. This would allow users to save output files only to local OSTs for jobs being run at some site, but they could still access input files from any OSTs transparently (albeit more slowly over a WAN connection). This feature is expected to be in the 1.8 release. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From ben at psc.edu Tue Jul 8 20:38:21 2008 From: ben at psc.edu (Benjamin Bennett) Date: Tue, 08 Jul 2008 16:38:21 -0400 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: References: Message-ID: <4873D03D.5060002@psc.edu> Peter Braam wrote: > Hmm. Perhaps there are implementation issues here that overshadow the > architecture. > > To interact with MDS nodes that are part of one file system, the MDS needs > to be part of a realm. The MDS performs authorization based on a principal > to MDS (i.e. Lustre) user/group database. Within one Lustre file system > each MDS MUST HAVE the same user group database. We will likely want to > place MDS's distributedly in the longer term future, so take clear note of > this: one Kerberos realm owns the entire MDS cluster for a file system. Could you explain more on why this requires a single realm and not just consistent mappings across all MDSs? > There can be multiple MDS clusters, i.e. Lustre file systems, in a single > realm, each serving their own file system. Each Lustre file system can have > its own user/group database. No restrictions here. Well, that's the problem with multiple clusters in a single realm, lack of restriction... ;-) > For a given file system the MDS nodes produce capabilities which the OSS > nodes use for authorization. It is important that the MDS can maken > authenticated RPC's to the OSS nodes in its file system and for this we use > Kerberos (this is not a "must have" - it could have been done with a > different key sharing mechanism). With multiple clusters in a single realm an MDS from any cluster could authenticate and authorize as an MDS to an OSS in any cluster. This would allow an MDS in one cluster to change the key used for capabilities on the OSSs in another cluster, no? > ==> So the first issue you have to become clear about is how you authorize > an MDS to contact one of its OSS nodes, wherever these are place. I've changed lsvcgssd on the OSSs to take an arbitrary number of '-M lustre_mds/mdshost at REALM' and use this list to determine MDS authorization. Is there a way in which an OSS is already aware of its appropriate MDSs? > Similarly the Kerberos connections are used by the clients to connect to the > OSS, but they are not used to authenticate anything (but optionally the > node), they are used merely to provide privacy and/or authenticity for > transporting data between the client and the OSS nodes. With relatively > little effort this could be done without Kerberos at all, on the other hand, > probably using Kerberos for this leads to a more easily understood > architecture. > > So, to repeat, the authorization uses capabilities, which authenticate the > requestor and contain authorization information, independent of a server > user/group database on the OSS. > > ==> The second issue you need to be clear about is how you authenticate > client NODES (NOT users) to OSS nodes. Client nodes are issued lustre_root/host credentials from their local realm. This works just fine for Client -> OST since the only [kerberos-related] authorization check is a "lustre_root" service part. > Peter --ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: OpenPGP digital signature URL: From aurelien.degremont at cea.fr Wed Jul 9 13:25:38 2008 From: aurelien.degremont at cea.fr (Aurelien Degremont) Date: Wed, 09 Jul 2008 15:25:38 +0200 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: References: Message-ID: <4874BC52.10604@cea.fr> Peter Braam a écrit : > 1. The HSM or a database associated with it implements a table to map FIDs > to stored HSM versions of a file, with other metadata it may need to > maintain its archives. Ok > 2. An HSM utility can query and learn about the versions stored for a fid > (or file name). A "restore" function can copy any version out of the HSM > and place it in the file system. This is similar to restoring a file from a > backup archive. Ok, that's copy-in. > 3. The file system only has attributes to indicate the state of the primary > archived copy (probably the last fully archived copy of the file), and can > retrieve that file on demand (without user intervention). Ok. Still need to store the purge window on MDT and OST to raise cache misses. How Lustre will update this information if user can use a HSM command directly, by-passing Lustre? He can change the file copies present in the HSM without Lustre knowing it. > 4. The HSM database will allow files in snapshots to be encoded with (fsid, > fid) or something similar. Can we consider there is always a default snapshot? The ID will always be done with FSID+FID ? Or should we consider a special case when snapshotting is not enabled ? >> There is no namespace tricks, no huge API changes, always one version of >> a file in Lustre, just few functions added to 'lfs' command. > > NO - this will not be an lfs command. This is an HSM command. Could you present a use case of how user will explicitly make backups and restore an older copy using the HSM command and no Lustre component? Doing this, the client nodes should be able to communicate with the HSM infrastructure, using specific network protocols, and so on. You will need to set up your Lustre network and then your HSM network even if HSM just need to talk with the Lustre agent. > 1. how is a bare metal restore arranged (ie. How is metadata moved into the > HSM)? Can this restore put files in a file system different than Lustre? Until now, the metadata were stored inside Lustre, so this was not needed. Now, we must add a way for the archiving tool to "setattr" this data when restoring a file. About a different filesystem, this will depend on the features used by the archiving tool to copy back the data and metadata. If those are standard, the file could be put in a different filesystem. > 2. how are small files grouped then "tar'd up" and how are we setting the > attributes of the inodes of the files that have been placed in the HSM after > this? How does the index entry for the fids in the HSM database function? Presently, just the archiving tool was supposed to support such feature, to avoid having to recode them later (various tools will be needed for the various existing HSMs and their development won't be centralized) when we will add this kind of feature. There is no defined mechanism for grouping file into the HSM presently. > 3. how are multiple coordinators and agents utilized to distribute load so > that the HSM can keep up with massive small file creation? One coordinator per MDT. The coordinator deals only with its MDT files. The coordinator dispatches their requests on the agents with a round-robin. Agents can refuse requests if they cannot handle them (too busy). Coordinator try another one. If no agent are available, it postpones the request. -- Aurelien Degremont CEA From Peter.Braam at Sun.COM Wed Jul 9 13:49:25 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Wed, 09 Jul 2008 07:49:25 -0600 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: <4874BC52.10604@cea.fr> Message-ID: On 7/9/08 7:25 AM, "Aurelien Degremont" wrote: > >> 3. The file system only has attributes to indicate the state of the primary >> archived copy (probably the last fully archived copy of the file), and can >> retrieve that file on demand (without user intervention). > > Ok. Still need to store the purge window on MDT and OST to raise cache > misses. Yes. > How Lustre will update this information if user can use a HSM command > directly, by-passing Lustre? He can change the file copies present in > the HSM without Lustre knowing it. NO - we said that the only operation we do is placing an entire file into Lustre. > >> 4. The HSM database will allow files in snapshots to be encoded with (fsid, >> fid) or something similar. > > Can we consider there is always a default snapshot? > The ID will always be done with FSID+FID ? Or should we consider a > special case when snapshotting is not enabled ? Why would you? You need to make sure that the index field is large enough. Almost all our customers have more than one file system anyway, regardless of snapshots. > >>> There is no namespace tricks, no huge API changes, always one version of >>> a file in Lustre, just few functions added to 'lfs' command. >> >> NO - this will not be an lfs command. This is an HSM command. > > Could you present a use case of how user will explicitly make backups > and restore an older copy using the HSM command and no Lustre component? Hsm_copy_to_fs /mnt/lustre/braams_lost_file > Doing this, the client nodes should be able to communicate with the HSM > infrastructure, using specific network protocols, and so on. > You will need to set up your Lustre network and then your HSM network > even if HSM just need to talk with the Lustre agent. The utility for restore is not essentially different from what the agent invokes as a mover. > >> 1. how is a bare metal restore arranged (ie. How is metadata moved into the >> HSM)? Can this restore put files in a file system different than Lustre? > > Until now, the metadata were stored inside Lustre, so this was not > needed. Now, we must add a way for the archiving tool to "setattr" this > data when restoring a file. > > About a different filesystem, this will depend on the features used by > the archiving tool to copy back the data and metadata. If those are > standard, the file could be put in a different filesystem. Hmm. This description has no content. If you don't want to do this, say so, or describe the entire process in detail. > >> 2. how are small files grouped then "tar'd up" and how are we setting the >> attributes of the inodes of the files that have been placed in the HSM after >> this? How does the index entry for the fids in the HSM database function? > > Presently, just the archiving tool HOW? > was supposed to support such feature, > to avoid having to recode them later (various tools will be needed for > the various existing HSMs and their development won't be centralized) > when we will add this kind of feature. > There is no defined mechanism for grouping file into the HSM presently. You need to describe this in detail - so far you are just repeating my questions pretending they are answers. What events are generated for small files? How are they grouped into something that is "tarred up"? What happens to all the individual inodes when the tarball hits the HSM? >> 3. how are multiple coordinators and agents utilized to distribute load so >> that the HSM can keep up with massive small file creation? > > One coordinator per MDT. No - these must be independent considerations. A coordinator may be much slower than an MDS node in handling a single file. I say this because this has been the experience in the industry so far - with small files the HSM can not at all keep up. > The coordinator deals only with its MDT files. > The coordinator dispatches their requests on the agents with a > round-robin. No, I think a more sophisticated policy is needed. Eg. Small files to these agents, big files to others. > Agents can refuse requests if they cannot handle them (too > busy). NO NO NO. > Coordinator try another one. If no agent are available, it > postpones the request. > Please take time to respond with details. Thanks. Peter From Peter.Braam at Sun.COM Wed Jul 9 14:31:52 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Wed, 09 Jul 2008 08:31:52 -0600 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: <4873D03D.5060002@psc.edu> Message-ID: On 7/8/08 2:38 PM, "Benjamin Bennett" wrote: > Peter Braam wrote: >> Hmm. Perhaps there are implementation issues here that overshadow the >> architecture. >> >> To interact with MDS nodes that are part of one file system, the MDS needs >> to be part of a realm. The MDS performs authorization based on a principal >> to MDS (i.e. Lustre) user/group database. Within one Lustre file system >> each MDS MUST HAVE the same user group database. We will likely want to >> place MDS's distributedly in the longer term future, so take clear note of >> this: one Kerberos realm owns the entire MDS cluster for a file system. > > Could you explain more on why this requires a single realm and not just > consistent mappings across all MDSs? That MIGHT work ... But how would two domains guarantee consistent updates to the databases? However, the server - server trust across domains we need is new to me (and I am not sure if/how it works). > >> There can be multiple MDS clusters, i.e. Lustre file systems, in a single >> realm, each serving their own file system. Each Lustre file system can have >> its own user/group database. No restrictions here. > > Well, that's the problem with multiple clusters in a single realm, lack > of restriction... ;-) Restrict yourself, not me or Lustre :) > >> For a given file system the MDS nodes produce capabilities which the OSS >> nodes use for authorization. It is important that the MDS can maken >> authenticated RPC's to the OSS nodes in its file system and for this we use >> Kerberos (this is not a "must have" - it could have been done with a >> different key sharing mechanism). > > With multiple clusters in a single realm an MDS from any cluster could > authenticate and authorize as an MDS to an OSS in any cluster. Good point. If so that should be a bug. ===> Eric Mei, what is the story here? The key (which is manually generated) should authenticate an instance of an MDS, not a "cluster". The only case where this might become delicate is if one MDS node is the server for two file systems. > This > would allow an MDS in one cluster to change the key used for > capabilities on the OSSs in another cluster, no? > >> ==> So the first issue you have to become clear about is how you authorize >> an MDS to contact one of its OSS nodes, wherever these are place. > > I've changed lsvcgssd on the OSSs to take an arbitrary number of '-M > lustre_mds/mdshost at REALM' and use this list to determine MDS > authorization. Is there a way in which an OSS is already aware of its > appropriate MDSs? As you pointed out, we need that, and Eric Mei should help you get that. > >> Similarly the Kerberos connections are used by the clients to connect to the >> OSS, but they are not used to authenticate anything (but optionally the >> node), they are used merely to provide privacy and/or authenticity for >> transporting data between the client and the OSS nodes. With relatively >> little effort this could be done without Kerberos at all, on the other hand, >> probably using Kerberos for this leads to a more easily understood >> architecture. >> >> So, to repeat, the authorization uses capabilities, which authenticate the >> requestor and contain authorization information, independent of a server >> user/group database on the OSS. >> >> ==> The second issue you need to be clear about is how you authenticate >> client NODES (NOT users) to OSS nodes. > > Client nodes are issued lustre_root/host credentials from their local > realm. This works just fine for Client -> OST since the only > [kerberos-related] authorization check is a "lustre_root" service part. Good. Does it work across realms, because it seems we need that in any case? BTW, thank you for trying this all out in detail, that is very helpful. Perhaps Sheila could talk with you and Eric Mei and get a nice writeup done for the manual. Regards peter > >> Peter > > --ben > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From Eric.Mei at Sun.COM Wed Jul 9 17:25:15 2008 From: Eric.Mei at Sun.COM (Eric Mei) Date: Wed, 09 Jul 2008 11:25:15 -0600 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: References: Message-ID: <4874F47B.4010209@sun.com> Peter Braam wrote: > > > On 7/8/08 2:38 PM, "Benjamin Bennett" wrote: > >> Peter Braam wrote: >>> Hmm. Perhaps there are implementation issues here that overshadow the >>> architecture. >>> >>> To interact with MDS nodes that are part of one file system, the MDS needs >>> to be part of a realm. The MDS performs authorization based on a principal >>> to MDS (i.e. Lustre) user/group database. Within one Lustre file system >>> each MDS MUST HAVE the same user group database. We will likely want to >>> place MDS's distributedly in the longer term future, so take clear note of >>> this: one Kerberos realm owns the entire MDS cluster for a file system. >> Could you explain more on why this requires a single realm and not just >> consistent mappings across all MDSs? > > That MIGHT work ... But how would two domains guarantee consistent updates > to the databases? However, the server - server trust across domains we need > is new to me (and I am not sure if/how it works). Practically it's doable, of course. But as Peter pointed out the user database must be the same across all MDSs within a Luster FS. If 2 MDSs could share the user database, why bother putting them into different kerberos realms? So we assume all MDSs should be in a single realm. Does TeraGrid have different requirement? >>> There can be multiple MDS clusters, i.e. Lustre file systems, in a single >>> realm, each serving their own file system. Each Lustre file system can have >>> its own user/group database. No restrictions here. >> Well, that's the problem with multiple clusters in a single realm, lack >> of restriction... ;-) > > Restrict yourself, not me or Lustre :) > >>> For a given file system the MDS nodes produce capabilities which the OSS >>> nodes use for authorization. It is important that the MDS can maken >>> authenticated RPC's to the OSS nodes in its file system and for this we use >>> Kerberos (this is not a "must have" - it could have been done with a >>> different key sharing mechanism). >> With multiple clusters in a single realm an MDS from any cluster could >> authenticate and authorize as an MDS to an OSS in any cluster. > > > > Good point. If so that should be a bug. > > ===> Eric Mei, what is the story here? Yes Ben is right, currently in a same realm any MDS could authenticate with any MDS and OSS. But afaics the problem is nothing to do with Kerberos. It's because currently Lustre have no config information about the server cluster membership, each server target have no idea what other targets are. So solve this, we can either place the configuration on each MDS/OST nodes - as Ben proposed in last mail; or probably better centrally managed by MGS, thus MDT/OST would be able to get uptodate server cluster information. Would it work? > The key (which is manually generated) should authenticate an instance of an > MDS, not a "cluster". The only case where this might become delicate is if > one MDS node is the server for two file systems. GSS/Kerberos is for the a certain kind service on a node, we can tell it simply from the composition of Kerberos principal "service_name/hostname at REALM". As to Lustre, lustre_mds/hostname at REALM it's for MDS, not specific to MDT. So if two MDTs on a MDS serving two different file systems, GSS/Kerberos authentications are performed in the same way for them, further access control should be handled by each target (MDT/OST). >> This >> would allow an MDS in one cluster to change the key used for >> capabilities on the OSSs in another cluster, no? >> >>> ==> So the first issue you have to become clear about is how you authorize >>> an MDS to contact one of its OSS nodes, wherever these are place. >> I've changed lsvcgssd on the OSSs to take an arbitrary number of '-M >> lustre_mds/mdshost at REALM' and use this list to determine MDS >> authorization. Is there a way in which an OSS is already aware of its >> appropriate MDSs? > > As you pointed out, we need that, and Eric Mei should help you get that. Yes that works, probably as temporary solution. As described above, currently OSS don't know that info. we may need a more complete centrally controlled server membership authentication, maybe independent of GSS/Kerberos. >>> Similarly the Kerberos connections are used by the clients to connect to the >>> OSS, but they are not used to authenticate anything (but optionally the >>> node), they are used merely to provide privacy and/or authenticity for >>> transporting data between the client and the OSS nodes. With relatively >>> little effort this could be done without Kerberos at all, on the other hand, >>> probably using Kerberos for this leads to a more easily understood >>> architecture. >>> >>> So, to repeat, the authorization uses capabilities, which authenticate the >>> requestor and contain authorization information, independent of a server >>> user/group database on the OSS. >>> >>> ==> The second issue you need to be clear about is how you authenticate >>> client NODES (NOT users) to OSS nodes. >> Client nodes are issued lustre_root/host credentials from their local >> realm. This works just fine for Client -> OST since the only >> [kerberos-related] authorization check is a "lustre_root" service part. > > Good. Does it work across realms, because it seems we need that in any > case? Yes, Ben had a patch to make it work. > BTW, thank you for trying this all out in detail, that is very helpful. > Perhaps Sheila could talk with you and Eric Mei and get a nice writeup done > for the manual. -- Eric From ben at psc.edu Wed Jul 9 20:07:17 2008 From: ben at psc.edu (Benjamin Bennett) Date: Wed, 09 Jul 2008 16:07:17 -0400 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: <4874F47B.4010209@sun.com> References: <4874F47B.4010209@sun.com> Message-ID: <48751A75.1000102@psc.edu> Eric Mei wrote: > Peter Braam wrote: >> >> >> On 7/8/08 2:38 PM, "Benjamin Bennett" wrote: >> >>> Peter Braam wrote: >>>> Hmm. Perhaps there are implementation issues here that overshadow the >>>> architecture. >>>> >>>> To interact with MDS nodes that are part of one file system, the MDS >>>> needs >>>> to be part of a realm. The MDS performs authorization based on a >>>> principal >>>> to MDS (i.e. Lustre) user/group database. Within one Lustre file >>>> system >>>> each MDS MUST HAVE the same user group database. We will likely >>>> want to >>>> place MDS's distributedly in the longer term future, so take clear >>>> note of >>>> this: one Kerberos realm owns the entire MDS cluster for a file system. >>> Could you explain more on why this requires a single realm and not just >>> consistent mappings across all MDSs? >> >> That MIGHT work ... But how would two domains guarantee consistent >> updates >> to the databases? However, the server - server trust across domains >> we need >> is new to me (and I am not sure if/how it works). > > Practically it's doable, of course. But as Peter pointed out the user > database must be the same across all MDSs within a Luster FS. If 2 MDSs > could share the user database, why bother putting them into different > kerberos realms? So we assume all MDSs should be in a single realm. Does > TeraGrid have different requirement? TeraGrid has a central database of users which could be used to consistently generate mappings. The reason to bother putting MDSs in separate realms is that TeraGrid is composed of distinct organizations. We are trying to distribute a filesystem across several organizations, not simply implement a centralized fs accessed by several organizations. >>>> There can be multiple MDS clusters, i.e. Lustre file systems, in a >>>> single >>>> realm, each serving their own file system. Each Lustre file system >>>> can have >>>> its own user/group database. No restrictions here. >>> Well, that's the problem with multiple clusters in a single realm, lack >>> of restriction... ;-) >> >> Restrict yourself, not me or Lustre :) >> >>>> For a given file system the MDS nodes produce capabilities which the >>>> OSS >>>> nodes use for authorization. It is important that the MDS can maken >>>> authenticated RPC's to the OSS nodes in its file system and for this >>>> we use >>>> Kerberos (this is not a "must have" - it could have been done with a >>>> different key sharing mechanism). >>> With multiple clusters in a single realm an MDS from any cluster could >>> authenticate and authorize as an MDS to an OSS in any cluster. >> >> >> >> Good point. If so that should be a bug. >> >> ===> Eric Mei, what is the story here? > > Yes Ben is right, currently in a same realm any MDS could authenticate > with any MDS and OSS. But afaics the problem is nothing to do with > Kerberos. It's because currently Lustre have no config information about > the server cluster membership, each server target have no idea what > other targets are. > > So solve this, we can either place the configuration on each MDS/OST > nodes - as Ben proposed in last mail; or probably better centrally > managed by MGS, thus MDT/OST would be able to get uptodate server > cluster information. Would it work? Sounds like a good idea. If I understand correctly... A) An MDT/OST is explicitly given the MGS NID by a trusted entity (administrator) during mkfs. B) The MGS principal name would be derived from its NID (assuming lustre_mgs/mgsnode at REALM). Realm is determined from the usual kerberos dns -> realm mapping mechanism? C) MDT and OST (or just MDS, OSS) list retrieved via secured MGC -> MGS connection. D) MDS and OSS principal names are derived from MDS and OSS NIDs. Same realm determination as in B? >> The key (which is manually generated) should authenticate an instance >> of an >> MDS, not a "cluster". The only case where this might become delicate >> is if >> one MDS node is the server for two file systems. > > GSS/Kerberos is for the a certain kind service on a node, we can tell it > simply from the composition of Kerberos principal > "service_name/hostname at REALM". As to Lustre, lustre_mds/hostname at REALM > it's for MDS, not specific to MDT. So if two MDTs on a MDS serving two > different file systems, GSS/Kerberos authentications are performed in > the same way for them, further access control should be handled by each > target (MDT/OST). > >>> This would allow an MDS in one cluster to change the key used for >>> capabilities on the OSSs in another cluster, no? >>> >>>> ==> So the first issue you have to become clear about is how you >>>> authorize >>>> an MDS to contact one of its OSS nodes, wherever these are place. >>> I've changed lsvcgssd on the OSSs to take an arbitrary number of '-M >>> lustre_mds/mdshost at REALM' and use this list to determine MDS >>> authorization. Is there a way in which an OSS is already aware of its >>> appropriate MDSs? >> >> As you pointed out, we need that, and Eric Mei should help you get that. > > Yes that works, probably as temporary solution. As described above, > currently OSS don't know that info. we may need a more complete > centrally controlled server membership authentication, maybe independent > of GSS/Kerberos. If you're interested, the patch I have is at [1]. >>>> Similarly the Kerberos connections are used by the clients to >>>> connect to the >>>> OSS, but they are not used to authenticate anything (but optionally the >>>> node), they are used merely to provide privacy and/or authenticity for >>>> transporting data between the client and the OSS nodes. With >>>> relatively >>>> little effort this could be done without Kerberos at all, on the >>>> other hand, >>>> probably using Kerberos for this leads to a more easily understood >>>> architecture. >>>> >>>> So, to repeat, the authorization uses capabilities, which >>>> authenticate the >>>> requestor and contain authorization information, independent of a >>>> server >>>> user/group database on the OSS. >>>> >>>> ==> The second issue you need to be clear about is how you authenticate >>>> client NODES (NOT users) to OSS nodes. >>> Client nodes are issued lustre_root/host credentials from their local >>> realm. This works just fine for Client -> OST since the only >>> [kerberos-related] authorization check is a "lustre_root" service part. >> >> Good. Does it work across realms, because it seems we need that in any >> case? > > Yes, Ben had a patch to make it work. The foreign lustre_root principals have to be mapped on the MDS to allow mount. What are your thoughts on authorizing [squashed] mount to all, so as to not require mapping? >> BTW, thank you for trying this all out in detail, that is very helpful. >> Perhaps Sheila could talk with you and Eric Mei and get a nice writeup >> done >> for the manual. np :-) --ben [1] http://staff.psc.edu/ben/patches/lustre/lustre-explicit-mds-authz.patch -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: OpenPGP digital signature URL: From adilger at sun.com Wed Jul 9 20:29:38 2008 From: adilger at sun.com (Andreas Dilger) Date: Wed, 09 Jul 2008 14:29:38 -0600 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: <4874F47B.4010209@sun.com> References: <4874F47B.4010209@sun.com> Message-ID: <20080709202937.GB6239@webber.adilger.int> On Jul 09, 2008 11:25 -0600, Eric Mei wrote: > Yes Ben is right, currently in a same realm any MDS could authenticate > with any MDS and OSS. But afaics the problem is nothing to do with > Kerberos. It's because currently Lustre have no config information about > the server cluster membership, each server target have no idea what > other targets are. > > So solve this, we can either place the configuration on each MDS/OST > nodes - as Ben proposed in last mail; or probably better centrally > managed by MGS, thus MDT/OST would be able to get uptodate server > cluster information. Would it work? I think that MDT/OST addition to the filesystem needs to be managed properly at the MGS, regardless of whether Kerberos is in use or not. Please see bug 15827 with some details of the problem. For the non-kerberos case having administrator action at the MGS is the most secure. Enabling a shared secret key passed to mkfs.lustre like "--mgs-key e85021aee637f7250e482a9a5b23cb0d" sent from the MDT/OST to the MGS at first connect time at least provides some restriction on adding new devices to the filesystem. With Kerberos systems there could be principals for the OSTs stored inside their filesystems by mkfs.lustre or tunefs.lustre that can be loaded into the keyring at mount time. Having it inside the filesystem (instead of e.g /etc/{something}) ensures that it is always available to the MDT/OST if it can mount. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From bs at q-leap.de Wed Jul 9 21:10:13 2008 From: bs at q-leap.de (Bernd Schubert) Date: Wed, 9 Jul 2008 23:10:13 +0200 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: <20080709202937.GB6239@webber.adilger.int> References: <4874F47B.4010209@sun.com> <20080709202937.GB6239@webber.adilger.int> Message-ID: <20080709211013.GA8132@lanczos.q-leap.de> On Wed, Jul 09, 2008 at 02:29:38PM -0600, Andreas Dilger wrote: > On Jul 09, 2008 11:25 -0600, Eric Mei wrote: > > Yes Ben is right, currently in a same realm any MDS could authenticate > > with any MDS and OSS. But afaics the problem is nothing to do with > > Kerberos. It's because currently Lustre have no config information about > > the server cluster membership, each server target have no idea what > > other targets are. > > > > So solve this, we can either place the configuration on each MDS/OST > > nodes - as Ben proposed in last mail; or probably better centrally > > managed by MGS, thus MDT/OST would be able to get uptodate server > > cluster information. Would it work? > > I think that MDT/OST addition to the filesystem needs to be managed > properly at the MGS, regardless of whether Kerberos is in use or not. > > Please see bug 15827 with some details of the problem. > > For the non-kerberos case having administrator action at the MGS is > the most secure. Enabling a shared secret key passed to mkfs.lustre > like "--mgs-key e85021aee637f7250e482a9a5b23cb0d" sent from the > MDT/OST to the MGS at first connect time at least provides some > restriction on adding new devices to the filesystem. Hmm, is a secret key really neccessary, for me this sounds a bit like security by obscurity. Wouldn't it be better to to have a two way MDT/OST registration? 1.) As it is, simply mount the filesystem on the MDT/OST. But this will put this filesystem into a registered, but unconfirmed state on the MGS. 2.) Introduce a lctl command for the MGS to list all registered-but-unconfirmed systems. And another command to confirm the registration. On on the other hand, this approach might conflict with the writeconf concept. Cheers, Bernd From Eric.Mei at Sun.COM Thu Jul 10 16:45:16 2008 From: Eric.Mei at Sun.COM (Eric Mei) Date: Thu, 10 Jul 2008 10:45:16 -0600 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: <48751A75.1000102@psc.edu> References: <4874F47B.4010209@sun.com> <48751A75.1000102@psc.edu> Message-ID: <48763C9C.4000003@sun.com> Benjamin Bennett wrote: > Eric Mei wrote: >> Peter Braam wrote: >>> >>> >>> On 7/8/08 2:38 PM, "Benjamin Bennett" wrote: >>> >>>> Peter Braam wrote: >>>>> Hmm. Perhaps there are implementation issues here that overshadow the >>>>> architecture. >>>>> >>>>> To interact with MDS nodes that are part of one file system, the >>>>> MDS needs >>>>> to be part of a realm. The MDS performs authorization based on a >>>>> principal >>>>> to MDS (i.e. Lustre) user/group database. Within one Lustre file >>>>> system >>>>> each MDS MUST HAVE the same user group database. We will likely >>>>> want to >>>>> place MDS's distributedly in the longer term future, so take clear >>>>> note of >>>>> this: one Kerberos realm owns the entire MDS cluster for a file >>>>> system. >>>> Could you explain more on why this requires a single realm and not just >>>> consistent mappings across all MDSs? >>> >>> That MIGHT work ... But how would two domains guarantee consistent >>> updates >>> to the databases? However, the server - server trust across domains >>> we need >>> is new to me (and I am not sure if/how it works). >> >> Practically it's doable, of course. But as Peter pointed out the user >> database must be the same across all MDSs within a Luster FS. If 2 >> MDSs could share the user database, why bother putting them into >> different kerberos realms? So we assume all MDSs should be in a single >> realm. Does TeraGrid have different requirement? > > TeraGrid has a central database of users which could be used to > consistently generate mappings. > > The reason to bother putting MDSs in separate realms is that TeraGrid is > composed of distinct organizations. We are trying to distribute a > filesystem across several organizations, not simply implement a > centralized fs accessed by several organizations. I see, thanks for explanation. I think if the issue of server membership solved, there'll be no problem to do that as GSS/Kerberos's aspect. >>>>> There can be multiple MDS clusters, i.e. Lustre file systems, in a >>>>> single >>>>> realm, each serving their own file system. Each Lustre file system >>>>> can have >>>>> its own user/group database. No restrictions here. >>>> Well, that's the problem with multiple clusters in a single realm, lack >>>> of restriction... ;-) >>> >>> Restrict yourself, not me or Lustre :) >>> >>>>> For a given file system the MDS nodes produce capabilities which >>>>> the OSS >>>>> nodes use for authorization. It is important that the MDS can maken >>>>> authenticated RPC's to the OSS nodes in its file system and for >>>>> this we use >>>>> Kerberos (this is not a "must have" - it could have been done with a >>>>> different key sharing mechanism). >>>> With multiple clusters in a single realm an MDS from any cluster could >>>> authenticate and authorize as an MDS to an OSS in any cluster. >>> >>> >>> >>> Good point. If so that should be a bug. >>> >>> ===> Eric Mei, what is the story here? >> >> Yes Ben is right, currently in a same realm any MDS could authenticate >> with any MDS and OSS. But afaics the problem is nothing to do with >> Kerberos. It's because currently Lustre have no config information >> about the server cluster membership, each server target have no idea >> what other targets are. >> >> So solve this, we can either place the configuration on each MDS/OST >> nodes - as Ben proposed in last mail; or probably better centrally >> managed by MGS, thus MDT/OST would be able to get uptodate server >> cluster information. Would it work? > > Sounds like a good idea. If I understand correctly... > A) An MDT/OST is explicitly given the MGS NID by a trusted entity > (administrator) during mkfs. > > B) The MGS principal name would be derived from its NID (assuming > lustre_mgs/mgsnode at REALM). Realm is determined from the usual kerberos > dns -> realm mapping mechanism? > > C) MDT and OST (or just MDS, OSS) list retrieved via secured MGC -> > MGS connection. > > D) MDS and OSS principal names are derived from MDS and OSS NIDs. Same > realm determination as in B? Well I guess you're talking about secure connection of MGC->MGS. Yes we have plan to add that in the near future. As for the server membership control, I meant sysad need to teach MGS that a Lustre filesytem is comprised of what MDT/OSTs. And when a MDT/OST mounting, it can get the server list from MGS, thus it would know to prevent unwanted connection which pretend to be a MDT. And I think the membership management better be working for both with or without Kerberos. >>> The key (which is manually generated) should authenticate an instance >>> of an >>> MDS, not a "cluster". The only case where this might become >>> delicate is if >>> one MDS node is the server for two file systems. >> >> GSS/Kerberos is for the a certain kind service on a node, we can tell >> it simply from the composition of Kerberos principal >> "service_name/hostname at REALM". As to Lustre, lustre_mds/hostname at REALM >> it's for MDS, not specific to MDT. So if two MDTs on a MDS serving two >> different file systems, GSS/Kerberos authentications are performed in >> the same way for them, further access control should be handled by >> each target (MDT/OST). >> >>>> This would allow an MDS in one cluster to change the key used for >>>> capabilities on the OSSs in another cluster, no? >>>> >>>>> ==> So the first issue you have to become clear about is how you >>>>> authorize >>>>> an MDS to contact one of its OSS nodes, wherever these are place. >>>> I've changed lsvcgssd on the OSSs to take an arbitrary number of '-M >>>> lustre_mds/mdshost at REALM' and use this list to determine MDS >>>> authorization. Is there a way in which an OSS is already aware of its >>>> appropriate MDSs? >>> >>> As you pointed out, we need that, and Eric Mei should help you get that. >> >> Yes that works, probably as temporary solution. As described above, >> currently OSS don't know that info. we may need a more complete >> centrally controlled server membership authentication, maybe >> independent of GSS/Kerberos. > > If you're interested, the patch I have is at [1]. Thanks. >>>>> Similarly the Kerberos connections are used by the clients to >>>>> connect to the >>>>> OSS, but they are not used to authenticate anything (but optionally >>>>> the >>>>> node), they are used merely to provide privacy and/or authenticity for >>>>> transporting data between the client and the OSS nodes. With >>>>> relatively >>>>> little effort this could be done without Kerberos at all, on the >>>>> other hand, >>>>> probably using Kerberos for this leads to a more easily understood >>>>> architecture. >>>>> >>>>> So, to repeat, the authorization uses capabilities, which >>>>> authenticate the >>>>> requestor and contain authorization information, independent of a >>>>> server >>>>> user/group database on the OSS. >>>>> >>>>> ==> The second issue you need to be clear about is how you >>>>> authenticate >>>>> client NODES (NOT users) to OSS nodes. >>>> Client nodes are issued lustre_root/host credentials from their local >>>> realm. This works just fine for Client -> OST since the only >>>> [kerberos-related] authorization check is a "lustre_root" service part. >>> >>> Good. Does it work across realms, because it seems we need that in any >>> case? >> >> Yes, Ben had a patch to make it work. > > The foreign lustre_root principals have to be mapped on the MDS to allow > mount. What are your thoughts on authorizing [squashed] mount to all, > so as to not require mapping? It was original assumption we made is that "remote realm" means "different user database". That's why remote realm user have to be remapped to a local user. It seems in TeraGrid case that's not true anymore. The squashed mount, if I understand it correctly, it can be done by set a mapping entry in lustre/idmap.conf, to map "*@REALM" from NID "*" to a local user "U" - I don't remember the exact syntax though. As for the user mapping part, I always feel not confident whether the current implementation is what people really want or not, and not fully tested, that's why I didn't put the UID mapping information on the public wiki. I believe you are the first one outside of Lustre Group to try that :) any opinions are very welcome, but decisions to change need to be made by Peter Braam. >>> BTW, thank you for trying this all out in detail, that is very helpful. >>> Perhaps Sheila could talk with you and Eric Mei and get a nice >>> writeup done >>> for the manual. > > np :-) > > > --ben > > [1] http://staff.psc.edu/ben/patches/lustre/lustre-explicit-mds-authz.patch -- Eric From moibenko at fnal.gov Thu Jul 10 14:33:26 2008 From: moibenko at fnal.gov (Alexander Moibenko) Date: Thu, 10 Jul 2008 09:33:26 -0500 Subject: [Lustre-devel] How store HSM metadata in MDT Message-ID: <48761DB6.4010207@fnal.gov> ------------------------------------------------------------ HSM MetaData For Lustre HSM project, it will be needed to store, for each file, a list of information describing how many copies the file has in the HSM, what is their HSM ID, the copy date, and so on. This data could easily reach 500 bytes (I think we will need between 40 and 50 bytes per HSM copy, and we should be able to save at least 10 copies, surely more). The question is: where could we store this data on MDT, in which place (EA?) and how manage this. We had a discussion about this with Andreas and Nathan and it is not very clear what is the best solution here regarding to: - We must keep in mind that there is 2 available backends for MDT: ldiskfs and ZFS and both must be supported here. - EA space is not very wide on ldiskfs and quite used by several other features (stripping, ACL, ...) - Clients will need to read this data and so the RPC mechanism should be available and large enough to handle it. Moreover, we will store a purged data range on OST and MDT. This could easily fit in a EA. What is the possible solutions we have here ? -- Aurelien Degremont CEA --------------------------------------------- Hi Aurelien, I have just subscribed to the list this is why my reply has an unusual format. We at FNAL are considering to integrate lustre with our multipetabyte storage system Enstore: http://www-ccf.fnal.gov/enstore/ We had a preliminary discussion and some feasibility study and think that EA would be a good match. We though that it would be enough to have HSM ID(s) stored as EA the rest can be fetched directly from HSM referred by ID(s). Thanks, -- -------------------------------------------- Alexander N. Moibenko, Ph.D., CD, Fermilab Tel: (630)840-3937 email: moibenko at fnal.gov -------------------------------------------- From eeb at sun.com Fri Jul 11 09:05:02 2008 From: eeb at sun.com (Eric Barton) Date: Fri, 11 Jul 2008 10:05:02 +0100 Subject: [Lustre-devel] How store HSM metadata in MDT In-Reply-To: <48761DB6.4010207@fnal.gov> References: <48761DB6.4010207@fnal.gov> Message-ID: <00d001c8e335$341ff2e0$0281a8c0@ebpc> > For Lustre HSM project, it will be needed to store, for each file, a > list of information describing how many copies the file has in the HSM, > what is their HSM ID, the copy date, and so on. This data could easily > reach 500 bytes (I think we will need between 40 and 50 bytes per HSM > copy, and we should be able to save at least 10 copies, surely more). > The question is: where could we store this data on MDT, in which place > (EA?) and how manage this. Storing information about > 1 HSM copy violates one of our groundrules... > 3. The file system only has attributes to indicate the state of the primary > archived copy (probably the last fully archived copy of the file), and can > retrieve that file on demand (without user intervention). Cheers, Eric From jc.lafoucriere at cea.fr Fri Jul 11 14:31:41 2008 From: jc.lafoucriere at cea.fr (Jacques-Charles Lafoucriere) Date: Fri, 11 Jul 2008 16:31:41 +0200 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: References: Message-ID: <48776ECD.3000501@cea.fr> Peter Braam wrote: > > There is a rule in Lustre - no scanning, ever. This rule will not be broken > by HSM. > If a site wants to connect an HSM to an existing file system (close to be full) how do we do without fast scanning ? We cannot restrict the use of HSM binding only on new file systems JC -------------- next part -------------- An HTML attachment was scrubbed... URL: From jc.lafoucriere at cea.fr Fri Jul 11 14:32:48 2008 From: jc.lafoucriere at cea.fr (Jacques-Charles Lafoucriere) Date: Fri, 11 Jul 2008 16:32:48 +0200 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: References: Message-ID: <48776F10.60008@cea.fr> Peter Braam wrote: >>> 3. how are multiple coordinators and agents utilized to distribute load so >>> that the HSM can keep up with massive small file creation? >>> >> One coordinator per MDT. >> > > No - these must be independent considerations. A coordinator may be much > slower than an MDS node in handling a single file. I say this because this > has been the experience in the industry so far - with small files the HSM > can not at all keep up. > Why a coordinator should be on a slower node than a MDS ? Coordinator is a Lustre service like other Lustre services so it will be on a right hardware Do you mean a coordinator is not part of the Lustre cluster ? JC -------------- next part -------------- An HTML attachment was scrubbed... URL: From jc.lafoucriere at cea.fr Fri Jul 11 14:37:21 2008 From: jc.lafoucriere at cea.fr (Jacques-Charles Lafoucriere) Date: Fri, 11 Jul 2008 16:37:21 +0200 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: References: Message-ID: <48777021.4030001@cea.fr> Hello following latest discussions I understand a large change is coming on Lustre/HSM interactions. In the HLD the HSM is following Lustre requests : - lustre triggers copy-out and copy-in - all the copy requests are made through the coordinator control (so the Hsm_copy_to_fs command is a command line interface to the coordinator). Note that this Hsm_copy_to_fs is different from the copy tool. The central role of coordinator allows us to control all the requests and avoid duplicated requests to copy tool (and give a global view). Now it seems Lustre will have to be able to follow HSM requests to send file back in Lustre and so independently of the coordinator (in a previous email it was requested Hsm_copy_to_fs to trigger copy independently of Lustre) I do not agree on this change because HSM has to be seen as a backend storage for Lustre and the decisions to copy have to be in Lustre. Lustre must no suffer HSM but must use it To manage this new requirement the copy tool has to implement a central entity that will : - avoid duplicate requests - choose which agent has to make the copy This will have to be duplicated for each HSM (or backend) supported and also will duplicate coordinator role so I think it is better to have it in Lustre instead of in the copy tool. About file grouping, the planed features are : - for copy out: in one request a list of files can be provided to copy tool so it can choose to group them in one HSM "group archive". - for copy in: if a file is in a HSM "group archive", the copy tool will copy back this file in Lustre (and not all the archive file) - grouped request can come from a user request or a the space manager (for a generic policy) The space manager design is today on stand by because of the lack of information on changelogs, feeds, Lustre policy engine I think there is a very strong need in Lustre to have a generic policy database that can be used to allocate files, copy out files, purge files, choose which agent will copy files ..... One use case for this database is to provide users an interface to said : I want all my *.avi files to be striped on 6 OST and all other files to be not striped JC From adilger at sun.com Fri Jul 11 21:32:07 2008 From: adilger at sun.com (Andreas Dilger) Date: Fri, 11 Jul 2008 15:32:07 -0600 Subject: [Lustre-devel] GSS cross-realm on MDT -> OST In-Reply-To: <20080709211013.GA8132@lanczos.q-leap.de> References: <4874F47B.4010209@sun.com> <20080709202937.GB6239@webber.adilger.int> <20080709211013.GA8132@lanczos.q-leap.de> Message-ID: <20080711213207.GK6239@webber.adilger.int> On Jul 09, 2008 23:10 +0200, Bernd Schubert wrote: > On Wed, Jul 09, 2008 at 02:29:38PM -0600, Andreas Dilger wrote: > > Please see bug 15827 with some details of the problem. > > > > For the non-kerberos case having administrator action at the MGS is > > the most secure. Enabling a shared secret key passed to mkfs.lustre > > like "--mgs-key e85021aee637f7250e482a9a5b23cb0d" sent from the > > MDT/OST to the MGS at first connect time at least provides some > > restriction on adding new devices to the filesystem. > > Hmm, is a secret key really neccessary, for me this sounds a bit like > security by obscurity. Wouldn't it be better to to have a two way > MDT/OST registration? > > 1.) As it is, simply mount the filesystem on the MDT/OST. But this will > put this filesystem into a registered, but unconfirmed state on the MGS. > > 2.) Introduce a lctl command for the MGS to list all > registered-but-unconfirmed systems. And another command to confirm the > registration. Yes, this is defintely the minimum requirement, and it should be the default behaviour. For the case when completely automated configuration is needed (e.g. during automated regression testing) then having the --mgs-key mount option would still prevent "random" OSTs from joining the filesystem (as in bug 15827). > On on the other hand, this approach might conflict with the writeconf concept. No, I think 2-step authentication is the most secure, but there needs to some way to circumvent it, only if the MGS allows it of course. If the MGS is compromised then all bets are off... Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Peter.Braam at Sun.COM Fri Jul 11 21:57:30 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Fri, 11 Jul 2008 15:57:30 -0600 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: <48776ECD.3000501@cea.fr> Message-ID: A one time event to build a database or log is acceptable. However, the proposal so far would require frequent scans because the metadata would be in the wrong place, that is what I want to avoid. Maintaining that metadata is very simple, and can solve some problems that you cannot easily solve without maintaining a database in conjunction with the HSM (such as cleaning partial, aborted copies into the HSM). You¹d simply ask the moving script to contact a database and make a record. The database may not be able to keep up with all file creations in the FS but it can probably keep up with the activities that one moving node does to/from the HSM. Peter On 7/11/08 8:31 AM, "Jacques-Charles Lafoucriere" wrote: > > > Peter Braam wrote: >> >> >> There is a rule in Lustre - no scanning, ever. This rule will not be broken >> by HSM. >> > If a site wants to connect an HSM to an existing file system (close to be > full) > how do we do without fast scanning ? > We cannot restrict the use of HSM binding only on new file systems > > JC > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Peter.Braam at Sun.COM Fri Jul 11 22:03:18 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Fri, 11 Jul 2008 16:03:18 -0600 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: <48776F10.60008@cea.fr> Message-ID: On 7/11/08 8:32 AM, "Jacques-Charles Lafoucriere" wrote: > > > Peter Braam wrote: >> >>> >>>> >>>> 3. how are multiple coordinators and agents utilized to distribute load so >>>> that the HSM can keep up with massive small file creation? >>>> >>>> >>> >>> One coordinator per MDT. >>> >>> >> >> >> No - these must be independent considerations. A coordinator may be much >> slower than an MDS node in handling a single file. I say this because this >> has been the experience in the industry so far - with small files the HSM >> can not at all keep up. >> > > Why a coordinator should be on a slower node than a MDS ? > Coordinator is a Lustre service like other Lustre services so it will be on a > right hardware > I see no reason whatsoever to couple them to MDT¹s. Keeping them de-coupled is more flexible and can scale better. Am I missing issues here? As for performance the following: a coordinator may have a lot of work to do to track which files still need to be handled by agents and which are in progress already and may have a fair amount of interaction with agents (not with HSM to tape but when a coordinator is handling a re-striping migration it will). But its interaction with the MDT¹s is very limited. So we could degrade the performance of an MDS by placing this on the same node. Let¹s keep this flexible please. Peter > > > Do you mean a coordinator is not part of the Lustre cluster ? > > JC > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Peter.Braam at Sun.COM Fri Jul 11 22:12:36 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Fri, 11 Jul 2008 16:12:36 -0600 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: <48777021.4030001@cea.fr> Message-ID: On 7/11/08 8:37 AM, "Jacques-Charles Lafoucriere" wrote: > Hello > > following latest discussions I understand a large change is coming on > Lustre/HSM interactions. > In the HLD the HSM is following Lustre requests : > - lustre triggers copy-out and copy-in We can feed a list to the coordinator also to pre-stage ("primary versions") > - all the copy requests are made through the coordinator control (so the > Hsm_copy_to_fs command is a command line interface to the coordinator). No. I think we need to design this, but it will sipmly create a new file in the file system, that doesn't require a coordinator. > Note that this Hsm_copy_to_fs is different from the copy tool. > The central role of coordinator allows us to control all the requests > and avoid duplicated requests to copy tool (and give a global view). The coordinator will do copy in form kernel, automatically triggered by cache miss or by feeding a list to restore HSM primary copies. The hsm_copy_to_fs tool is ONLY needed when secondary copies held in the HSM are being restored. > > Now it seems Lustre will have to be able to follow HSM requests to send > file back in Lustre and so independently of the coordinator (in a > previous email it was requested Hsm_copy_to_fs to trigger copy > independently of Lustre) > I do not agree on this change because HSM has to be seen as a backend > storage for Lustre and the decisions to copy have to be in Lustre. > Lustre must no suffer HSM but must use it No - I think you perhaps misunderstood the proposal. > > To manage this new requirement the copy tool has to implement a central > entity that will : > - avoid duplicate requests > - choose which agent has to make the copy > This will have to be duplicated for each HSM (or backend) supported and > also will duplicate coordinator role so I think it is better to have it > in Lustre instead of in the copy tool. No. > > About file grouping, the planed features are : > - for copy out: in one request a list of files can be provided to copy > tool so it can choose to group them in one HSM "group archive". This again just rephrasing my question. HOW is a list of small files formed? > - for copy in: if a file is in a HSM "group archive", the copy tool will > copy back this file in Lustre (and not all the archive file) No, because with your proposal restoring a 1000 small files will cause 1000 tape actions to get the archive. I think the entire archive should come back in one blow. > - grouped request can come from a user request or a the space manager > (for a generic policy) Yes, how is the metadata handled? This is the case where the HSM DB does see significant load to make a mapping for each lustre fid to the archived file. > > The space manager design is today on stand by because of the lack of > information on changelogs, feeds, Lustre policy engine We will get there shortly. > > I think there is a very strong need in Lustre to have a generic policy > database that can be used to allocate files, copy out files, purge > files, choose which agent will copy files ..... Policy database yes, but NOT a database with HSM related data. There is a secondary side of policy which is how to treat the data that is held in the HSM and that doesn't belong in Lustre, but belongs in user space. Yet, this should be two parts of one policy management interface. If Lustre runs with Sun HSM it would be one tool, if Lustre runs with HPSS the tool would have two sides - one to Lustre from Sun and one to HPSS from the HPSS community. > One use case for this database is to provide users an interface to said > : I want all my *.avi files to be striped on 6 OST and all other files > to be not striped That is not an HSM policy, this should be a pool data placement policy and I agree we need it. The HSM should have sufficient metadata to restore files in this manner if bare metal restore takes place. The good news is that I see no serious disagreements, just some minor misunderstandings. Agree? Happy quartorze juillet!! Regards Peter > > JC > > From Peter.Braam at Sun.COM Sat Jul 12 03:56:07 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Fri, 11 Jul 2008 21:56:07 -0600 Subject: [Lustre-devel] Vector I/O api Message-ID: Tom - In a recent call with CERN the request came up to construct a call that can in parallel transfer an array of extents in a single file to a list of buffers and vice-versa. This call should be executed with read-ahead disabled, it will usually be made when the user is well informed of the I/O that is about to take place. Is this easy to get into the Lustre client (using our I/O library)? Do you have this already for MPI/IO use? Thanks. Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From Tom.Wang at Sun.COM Sat Jul 12 15:37:44 2008 From: Tom.Wang at Sun.COM (Tom.Wang) Date: Sat, 12 Jul 2008 11:37:44 -0400 Subject: [Lustre-devel] Vector I/O api In-Reply-To: References: Message-ID: <4878CFC8.3040203@sun.com> Peter Braam wrote: > Tom - > > In a recent call with CERN the request came up to construct a call > that can in parallel transfer an array of extents in a single file to > a list of buffers and vice-versa. > This call should be executed with read-ahead disabled, it will usually > be made when the user is well informed of the I/O that is about to > take place. > Is this easy to get into the Lustre client (using our I/O library)? > Do you have this already for MPI/IO use? > > Thanks. > > Peter Hello, Peter If you mean provide this list buffer read/write API in MPI by our library, it is easy. Because MPI already provide such API, you can define proper discontingous buf_type and file_type of these extents, and use (MPI_File_Write/read_all) to read/write these buffers in one call . We only need disable read-ahead here. So it should be easy to get into our I/O library. But if you mean provide such API in llite, I am not sure it is easy. because it seems we could only use ioctl to implement such non-posix API IMHO, which always has page-size limit for transferring buffers here? It is probably I misunderstand something here. Thanks WangDi > This kind of list buffers transferring can be implemented with proper MPI file_view > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > -- Regards, Tom Wangdi -- Sun Lustre Group System Software Engineer http://www.sun.com From eeb at sun.com Sat Jul 12 16:46:49 2008 From: eeb at sun.com (Eric Barton) Date: Sat, 12 Jul 2008 17:46:49 +0100 Subject: [Lustre-devel] Vector I/O api In-Reply-To: <4878CFC8.3040203@sun.com> References: <4878CFC8.3040203@sun.com> Message-ID: <00e401c8e43e$e1775aa0$0281a8c0@ebpc> Wangdi, There seems to be some momentum behind getting readx/writex adopted as posix standard system calls. That seems the right API to exploit (or anticipate if it's not implemented yet). Note that the memory and file descriptors are not required to be isomorphic (i.e. file and memory fragments don't have to correspond directly). struct iovec { void *iov_base; /* Starting address */ size_t iov_len; /* Number of bytes */ }; struct xtvec { off_t xtv_off; /* Starting file offset */ size_t xtv_len; /* Number of bytes */ }; ssize_t readx(int fd, const struct iovec *iov, size_t iov_count, struct xtvec *xtv, size_t xtv_count); ssize_t writex(int fd, const struct iovec *iov, size_t iov_count, struct xtvec *xtv, size_t xtv_count); Cheers, Eric > -----Original Message----- > From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Tom.Wang > Sent: 12 July 2008 4:38 PM > To: Peter Braam > Cc: lustre-devel > Subject: Re: [Lustre-devel] Vector I/O api > > > Peter Braam wrote: > > Tom - > > > > In a recent call with CERN the request came up to construct a call > > that can in parallel transfer an array of extents in a single file to > > a list of buffers and vice-versa. > > This call should be executed with read-ahead disabled, it will usually > > be made when the user is well informed of the I/O that is about to > > take place. > > Is this easy to get into the Lustre client (using our I/O library)? > > Do you have this already for MPI/IO use? > > > > Thanks. > > > > Peter > Hello, Peter > > If you mean provide this list buffer read/write API in MPI by our > library, it is easy. > Because MPI already provide such API, you can define proper > discontingous buf_type > and file_type of these extents, and use (MPI_File_Write/read_all) to > read/write these > buffers in one call . We only need disable read-ahead here. So it should > be easy to > get into our I/O library. > > But if you mean provide such API in llite, I am not sure it is easy. > because it seems we > could only use ioctl to implement such non-posix API IMHO, which always > has page-size > limit for transferring buffers here? It is probably I misunderstand > something here. > > Thanks > WangDi > > > > > This kind of list buffers transferring can be implemented with proper > MPI file_view > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Lustre-devel mailing list > > Lustre-devel at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-devel > > > > > -- > Regards, > Tom Wangdi > -- > Sun Lustre Group > System Software Engineer > http://www.sun.com > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Nikita.Danilov at Sun.COM Sat Jul 12 17:34:09 2008 From: Nikita.Danilov at Sun.COM (Nikita Danilov) Date: Sat, 12 Jul 2008 21:34:09 +0400 Subject: [Lustre-devel] Vector I/O api In-Reply-To: <4878CFC8.3040203@sun.com> References: <4878CFC8.3040203@sun.com> Message-ID: <18552.60177.444426.24266@gargle.gargle.HOWL> Tom.Wang writes: > > Peter Braam wrote: > > Tom - > > > > In a recent call with CERN the request came up to construct a call > > that can in parallel transfer an array of extents in a single file to > > a list of buffers and vice-versa. > > This call should be executed with read-ahead disabled, it will usually > > be made when the user is well informed of the I/O that is about to > > take place. > > Is this easy to get into the Lustre client (using our I/O library)? > > Do you have this already for MPI/IO use? > > > > Thanks. > > > > Peter > Hello, Peter > > If you mean provide this list buffer read/write API in MPI by our > library, it is easy. > Because MPI already provide such API, you can define proper > discontingous buf_type > and file_type of these extents, and use (MPI_File_Write/read_all) to > read/write these > buffers in one call . We only need disable read-ahead here. So it should > be easy to > get into our I/O library. > > But if you mean provide such API in llite, I am not sure it is easy. > because it seems we I think that if we are going to support this interface in a client, it makes sense to implement it in CLIO. CLIO design of readv and writev support allows upper layer (llite) to submit IO for multiple disjoint file extents within single system call. Nikita. From Tom.Wang at Sun.COM Sat Jul 12 18:15:25 2008 From: Tom.Wang at Sun.COM (Tom.Wang) Date: Sat, 12 Jul 2008 14:15:25 -0400 Subject: [Lustre-devel] Vector I/O api In-Reply-To: <00e401c8e43e$e1775aa0$0281a8c0@ebpc> References: <4878CFC8.3040203@sun.com> <00e401c8e43e$e1775aa0$0281a8c0@ebpc> Message-ID: <4878F4BD.9020403@sun.com> Hello, Yes, I just check source, we could use sys_readv here. But there are a limit of 1024 IO segments for each call, maybe it should not be a problem here. Actually, llite already include such api (ll_file_readv/writev). Then it should be easy to implement this by our lib. Sorry for the previous confuse reply. Thanks WangDi Eric Barton wrote: > Wangdi, > > There seems to be some momentum behind getting readx/writex > adopted as posix standard system calls. That seems the right > API to exploit (or anticipate if it's not implemented yet). > > Note that the memory and file descriptors are not required to > be isomorphic (i.e. file and memory fragments don't have to > correspond directly). > > struct iovec { > void *iov_base; /* Starting address */ > size_t iov_len; /* Number of bytes */ > }; > > struct xtvec { > off_t xtv_off; /* Starting file offset */ > size_t xtv_len; /* Number of bytes */ > }; > > ssize_t readx(int fd, const struct iovec *iov, size_t iov_count, > struct xtvec *xtv, size_t xtv_count); > > ssize_t writex(int fd, const struct iovec *iov, size_t iov_count, > struct xtvec *xtv, size_t xtv_count); > > Cheers, > Eric > > > >> -----Original Message----- >> From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Tom.Wang >> Sent: 12 July 2008 4:38 PM >> To: Peter Braam >> Cc: lustre-devel >> Subject: Re: [Lustre-devel] Vector I/O api >> >> >> Peter Braam wrote: >> >>> Tom - >>> >>> In a recent call with CERN the request came up to construct a call >>> that can in parallel transfer an array of extents in a single file to >>> a list of buffers and vice-versa. >>> This call should be executed with read-ahead disabled, it will usually >>> be made when the user is well informed of the I/O that is about to >>> take place. >>> Is this easy to get into the Lustre client (using our I/O library)? >>> Do you have this already for MPI/IO use? >>> >>> Thanks. >>> >>> Peter >>> >> Hello, Peter >> >> If you mean provide this list buffer read/write API in MPI by our >> library, it is easy. >> Because MPI already provide such API, you can define proper >> discontingous buf_type >> and file_type of these extents, and use (MPI_File_Write/read_all) to >> read/write these >> buffers in one call . We only need disable read-ahead here. So it should >> be easy to >> get into our I/O library. >> >> But if you mean provide such API in llite, I am not sure it is easy. >> because it seems we >> could only use ioctl to implement such non-posix API IMHO, which always >> has page-size >> limit for transferring buffers here? It is probably I misunderstand >> something here. >> >> Thanks >> WangDi >> >> This kind of list buffers transferring can be implemented with proper >> MPI file_view >> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> Lustre-devel mailing list >>> Lustre-devel at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>> >>> >> -- >> Regards, >> Tom Wangdi >> -- >> Sun Lustre Group >> System Software Engineer >> http://www.sun.com >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> >> > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > -- Regards, Tom Wangdi -- Sun Lustre Group System Software Engineer http://www.sun.com From Peter.Braam at Sun.COM Sat Jul 12 20:23:08 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Sat, 12 Jul 2008 14:23:08 -0600 Subject: [Lustre-devel] Vector I/O api In-Reply-To: <4878F4BD.9020403@sun.com> Message-ID: Hi - 1024 segments is fine. Readv is the wrong call - it reads contiguous areas from files. Readx/writex sound good, but making this available asap through our I/O library is important. It should be coded to somewhat minimize the number of round trips over the network to get the I/O done. So what are our options? On 7/12/08 12:15 PM, "Tom.Wang" wrote: > Hello, > > Yes, I just check source, we could use sys_readv here. > But there are a limit of 1024 IO segments for each call, maybe it > should not be a problem here. Actually, llite already include such > api (ll_file_readv/writev). Then it should be easy to implement this > by our lib. Sorry for the previous confuse reply. > > Thanks > WangDi > > Eric Barton wrote: >> Wangdi, >> >> There seems to be some momentum behind getting readx/writex >> adopted as posix standard system calls. That seems the right >> API to exploit (or anticipate if it's not implemented yet). >> >> Note that the memory and file descriptors are not required to >> be isomorphic (i.e. file and memory fragments don't have to >> correspond directly). >> >> struct iovec { >> void *iov_base; /* Starting address */ >> size_t iov_len; /* Number of bytes */ >> }; >> >> struct xtvec { >> off_t xtv_off; /* Starting file offset */ >> size_t xtv_len; /* Number of bytes */ >> }; >> >> ssize_t readx(int fd, const struct iovec *iov, size_t iov_count, >> struct xtvec *xtv, size_t xtv_count); >> >> ssize_t writex(int fd, const struct iovec *iov, size_t iov_count, >> struct xtvec *xtv, size_t xtv_count); >> >> Cheers, >> Eric >> >> >> >>> -----Original Message----- >>> From: lustre-devel-bounces at lists.lustre.org >>> [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Tom.Wang >>> Sent: 12 July 2008 4:38 PM >>> To: Peter Braam >>> Cc: lustre-devel >>> Subject: Re: [Lustre-devel] Vector I/O api >>> >>> >>> Peter Braam wrote: >>> >>>> Tom - >>>> >>>> In a recent call with CERN the request came up to construct a call >>>> that can in parallel transfer an array of extents in a single file to >>>> a list of buffers and vice-versa. >>>> This call should be executed with read-ahead disabled, it will usually >>>> be made when the user is well informed of the I/O that is about to >>>> take place. >>>> Is this easy to get into the Lustre client (using our I/O library)? >>>> Do you have this already for MPI/IO use? >>>> >>>> Thanks. >>>> >>>> Peter >>>> >>> Hello, Peter >>> >>> If you mean provide this list buffer read/write API in MPI by our >>> library, it is easy. >>> Because MPI already provide such API, you can define proper >>> discontingous buf_type >>> and file_type of these extents, and use (MPI_File_Write/read_all) to >>> read/write these >>> buffers in one call . We only need disable read-ahead here. So it should >>> be easy to >>> get into our I/O library. >>> >>> But if you mean provide such API in llite, I am not sure it is easy. >>> because it seems we >>> could only use ioctl to implement such non-posix API IMHO, which always >>> has page-size >>> limit for transferring buffers here? It is probably I misunderstand >>> something here. >>> >>> Thanks >>> WangDi >>> >>> This kind of list buffers transferring can be implemented with proper >>> MPI file_view >>> >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> Lustre-devel mailing list >>>> Lustre-devel at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>>> >>>> >>> -- >>> Regards, >>> Tom Wangdi >>> -- >>> Sun Lustre Group >>> System Software Engineer >>> http://www.sun.com >>> >>> _______________________________________________ >>> Lustre-devel mailing list >>> Lustre-devel at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>> >>> >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> > From Tom.Wang at Sun.COM Sat Jul 12 21:29:36 2008 From: Tom.Wang at Sun.COM (Tom.Wang) Date: Sat, 12 Jul 2008 17:29:36 -0400 Subject: [Lustre-devel] Vector I/O api In-Reply-To: References: Message-ID: <48792240.9090302@sun.com> Hello, Unfortunately, Readx/writex is still not included in linux kernel yet. So we may have these 2 options: 1) Use ioctl to transfer iovec and xvetc to llite, and then do read/write for each IO sec. Not sure if nikita's CLIO did something to minimize these IO's round trip. or 2) Provide such API in liblustreapi.a, then do each read/write for each IO sec there, where we can also use "read_all first, then copy the buffer to each seg" to minimize the number of round trips. But it depends on the distance between the disjoint extents. And also it may need extra buffer allocation, If putting this list buffer API to llite is a *must* requirement. Forget this option. Thanks WangDi Peter Braam wrote: > Hi - > > 1024 segments is fine. > b > Readv is the wrong call - it reads contiguous areas from files. > > Readx/writex sound good, but making this available asap through our I/O > library is important. > > It should be coded to somewhat minimize the number of round trips over the > network to get the I/O done. > > So what are our options? > > > On 7/12/08 12:15 PM, "Tom.Wang" wrote: > > >> Hello, >> >> Yes, I just check source, we could use sys_readv here. >> But there are a limit of 1024 IO segments for each call, maybe it >> should not be a problem here. Actually, llite already include such >> api (ll_file_readv/writev). Then it should be easy to implement this >> by our lib. Sorry for the previous confuse reply. >> >> Thanks >> WangDi >> >> Eric Barton wrote: >> >>> Wangdi, >>> >>> There seems to be some momentum behind getting readx/writex >>> adopted as posix standard system calls. That seems the right >>> API to exploit (or anticipate if it's not implemented yet). >>> >>> Note that the memory and file descriptors are not required to >>> be isomorphic (i.e. file and memory fragments don't have to >>> correspond directly). >>> >>> struct iovec { >>> void *iov_base; /* Starting address */ >>> size_t iov_len; /* Number of bytes */ >>> }; >>> >>> struct xtvec { >>> off_t xtv_off; /* Starting file offset */ >>> size_t xtv_len; /* Number of bytes */ >>> }; >>> >>> ssize_t readx(int fd, const struct iovec *iov, size_t iov_count, >>> struct xtvec *xtv, size_t xtv_count); >>> >>> ssize_t writex(int fd, const struct iovec *iov, size_t iov_count, >>> struct xtvec *xtv, size_t xtv_count); >>> >>> Cheers, >>> Eric >>> >>> >>> >>> >>>> -----Original Message----- >>>> From: lustre-devel-bounces at lists.lustre.org >>>> [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Tom.Wang >>>> Sent: 12 July 2008 4:38 PM >>>> To: Peter Braam >>>> Cc: lustre-devel >>>> Subject: Re: [Lustre-devel] Vector I/O api >>>> >>>> >>>> Peter Braam wrote: >>>> >>>> >>>>> Tom - >>>>> >>>>> In a recent call with CERN the request came up to construct a call >>>>> that can in parallel transfer an array of extents in a single file to >>>>> a list of buffers and vice-versa. >>>>> This call should be executed with read-ahead disabled, it will usually >>>>> be made when the user is well informed of the I/O that is about to >>>>> take place. >>>>> Is this easy to get into the Lustre client (using our I/O library)? >>>>> Do you have this already for MPI/IO use? >>>>> >>>>> Thanks. >>>>> >>>>> Peter >>>>> >>>>> >>>> Hello, Peter >>>> >>>> If you mean provide this list buffer read/write API in MPI by our >>>> library, it is easy. >>>> Because MPI already provide such API, you can define proper >>>> discontingous buf_type >>>> and file_type of these extents, and use (MPI_File_Write/read_all) to >>>> read/write these >>>> buffers in one call . We only need disable read-ahead here. So it should >>>> be easy to >>>> get into our I/O library. >>>> >>>> But if you mean provide such API in llite, I am not sure it is easy. >>>> because it seems we >>>> could only use ioctl to implement such non-posix API IMHO, which always >>>> has page-size >>>> limit for transferring buffers here? It is probably I misunderstand >>>> something here. >>>> >>>> Thanks >>>> WangDi >>>> >>>> This kind of list buffers transferring can be implemented with proper >>>> MPI file_view >>>> >>>> >>>>> ------------------------------------------------------------------------ >>>>> >>>>> _______________________________________________ >>>>> Lustre-devel mailing list >>>>> Lustre-devel at lists.lustre.org >>>>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>>>> >>>>> >>>>> >>>> -- >>>> Regards, >>>> Tom Wangdi >>>> -- >>>> Sun Lustre Group >>>> System Software Engineer >>>> http://www.sun.com >>>> >>>> _______________________________________________ >>>> Lustre-devel mailing list >>>> Lustre-devel at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>>> >>>> >>>> >>> _______________________________________________ >>> Lustre-devel mailing list >>> Lustre-devel at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>> >>> > > > -- Regards, Tom Wangdi -- Sun Lustre Group System Software Engineer http://www.sun.com From Jinshan.Xiong at Sun.COM Sun Jul 13 02:55:34 2008 From: Jinshan.Xiong at Sun.COM (jay) Date: Sun, 13 Jul 2008 10:55:34 +0800 Subject: [Lustre-devel] Vector I/O api In-Reply-To: <48792240.9090302@sun.com> References: <48792240.9090302@sun.com> Message-ID: <48796EA6.1060707@sun.com> Sounds like what customer needs is just a non-cache read/write? So how likely to implement it via directIO, which would be multithreaded driven, and each for a stripe to get better performance.. We don't need to put it in kernel side at all jay Tom.Wang wrote: > Hello, > > Unfortunately, Readx/writex is still not included in linux kernel yet. > > So we may have these 2 options: > > 1) Use ioctl to transfer iovec and xvetc to llite, and then do > read/write for > each IO sec. Not sure if nikita's CLIO did something to minimize these > IO's round trip. > > or > > 2) Provide such API in liblustreapi.a, then do each read/write for each > IO sec > there, where we can also use "read_all first, then copy the buffer > to each seg" to minimize the number of round trips. But it depends > on the > distance between the disjoint extents. And also it may need extra > buffer allocation, > If putting this list buffer API to llite is a *must* requirement. > Forget this option. > > Thanks > WangDi > > Peter Braam wrote: > >> Hi - >> >> 1024 segments is fine. >> b >> Readv is the wrong call - it reads contiguous areas from files. >> >> Readx/writex sound good, but making this available asap through our I/O >> library is important. >> >> It should be coded to somewhat minimize the number of round trips over the >> network to get the I/O done. >> >> So what are our options? >> >> >> On 7/12/08 12:15 PM, "Tom.Wang" wrote: >> >> >> >>> Hello, >>> >>> Yes, I just check source, we could use sys_readv here. >>> But there are a limit of 1024 IO segments for each call, maybe it >>> should not be a problem here. Actually, llite already include such >>> api (ll_file_readv/writev). Then it should be easy to implement this >>> by our lib. Sorry for the previous confuse reply. >>> >>> Thanks >>> WangDi >>> >>> Eric Barton wrote: >>> >>> >>>> Wangdi, >>>> >>>> There seems to be some momentum behind getting readx/writex >>>> adopted as posix standard system calls. That seems the right >>>> API to exploit (or anticipate if it's not implemented yet). >>>> >>>> Note that the memory and file descriptors are not required to >>>> be isomorphic (i.e. file and memory fragments don't have to >>>> correspond directly). >>>> >>>> struct iovec { >>>> void *iov_base; /* Starting address */ >>>> size_t iov_len; /* Number of bytes */ >>>> }; >>>> >>>> struct xtvec { >>>> off_t xtv_off; /* Starting file offset */ >>>> size_t xtv_len; /* Number of bytes */ >>>> }; >>>> >>>> ssize_t readx(int fd, const struct iovec *iov, size_t iov_count, >>>> struct xtvec *xtv, size_t xtv_count); >>>> >>>> ssize_t writex(int fd, const struct iovec *iov, size_t iov_count, >>>> struct xtvec *xtv, size_t xtv_count); >>>> >>>> Cheers, >>>> Eric >>>> >>>> >>>> >>>> >>>> >>>>> -----Original Message----- >>>>> From: lustre-devel-bounces at lists.lustre.org >>>>> [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Tom.Wang >>>>> Sent: 12 July 2008 4:38 PM >>>>> To: Peter Braam >>>>> Cc: lustre-devel >>>>> Subject: Re: [Lustre-devel] Vector I/O api >>>>> >>>>> >>>>> Peter Braam wrote: >>>>> >>>>> >>>>> >>>>>> Tom - >>>>>> >>>>>> In a recent call with CERN the request came up to construct a call >>>>>> that can in parallel transfer an array of extents in a single file to >>>>>> a list of buffers and vice-versa. >>>>>> This call should be executed with read-ahead disabled, it will usually >>>>>> be made when the user is well informed of the I/O that is about to >>>>>> take place. >>>>>> Is this easy to get into the Lustre client (using our I/O library)? >>>>>> Do you have this already for MPI/IO use? >>>>>> >>>>>> Thanks. >>>>>> >>>>>> Peter >>>>>> >>>>>> >>>>>> >>>>> Hello, Peter >>>>> >>>>> If you mean provide this list buffer read/write API in MPI by our >>>>> library, it is easy. >>>>> Because MPI already provide such API, you can define proper >>>>> discontingous buf_type >>>>> and file_type of these extents, and use (MPI_File_Write/read_all) to >>>>> read/write these >>>>> buffers in one call . We only need disable read-ahead here. So it should >>>>> be easy to >>>>> get into our I/O library. >>>>> >>>>> But if you mean provide such API in llite, I am not sure it is easy. >>>>> because it seems we >>>>> could only use ioctl to implement such non-posix API IMHO, which always >>>>> has page-size >>>>> limit for transferring buffers here? It is probably I misunderstand >>>>> something here. >>>>> >>>>> Thanks >>>>> WangDi >>>>> >>>>> This kind of list buffers transferring can be implemented with proper >>>>> MPI file_view >>>>> >>>>> >>>>> >>>>>> ------------------------------------------------------------------------ >>>>>> >>>>>> _______________________________________________ >>>>>> Lustre-devel mailing list >>>>>> Lustre-devel at lists.lustre.org >>>>>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>>>>> >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Regards, >>>>> Tom Wangdi >>>>> -- >>>>> Sun Lustre Group >>>>> System Software Engineer >>>>> http://www.sun.com >>>>> >>>>> _______________________________________________ >>>>> Lustre-devel mailing list >>>>> Lustre-devel at lists.lustre.org >>>>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>>>> >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Lustre-devel mailing list >>>> Lustre-devel at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>>> >>>> >>>> >> >> > > > From Peter.Braam at Sun.COM Sun Jul 13 04:30:14 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Sat, 12 Jul 2008 22:30:14 -0600 Subject: [Lustre-devel] Vector I/O api In-Reply-To: <48796EA6.1060707@sun.com> Message-ID: This seems fundamental enough that it should be implemented in the kernel. We would get far too many threads. The files may well be used again, so they should be cached. So I think direct I/O is not the right path to get there. peter On 7/12/08 8:55 PM, "jay" wrote: > > Sounds like what customer needs is just a non-cache read/write? So how > likely to implement it via directIO, which would be multithreaded > driven, and each for a stripe to get better performance.. We don't need > to put it in kernel side at all > > jay > > Tom.Wang wrote: >> Hello, >> >> Unfortunately, Readx/writex is still not included in linux kernel yet. >> >> So we may have these 2 options: >> >> 1) Use ioctl to transfer iovec and xvetc to llite, and then do >> read/write for >> each IO sec. Not sure if nikita's CLIO did something to minimize these >> IO's round trip. >> >> or >> >> 2) Provide such API in liblustreapi.a, then do each read/write for each >> IO sec >> there, where we can also use "read_all first, then copy the buffer >> to each seg" to minimize the number of round trips. But it depends >> on the >> distance between the disjoint extents. And also it may need extra >> buffer allocation, >> If putting this list buffer API to llite is a *must* requirement. >> Forget this option. >> >> Thanks >> WangDi >> >> Peter Braam wrote: >> >>> Hi - >>> >>> 1024 segments is fine. >>> b >>> Readv is the wrong call - it reads contiguous areas from files. >>> >>> Readx/writex sound good, but making this available asap through our I/O >>> library is important. >>> >>> It should be coded to somewhat minimize the number of round trips over the >>> network to get the I/O done. >>> >>> So what are our options? >>> >>> >>> On 7/12/08 12:15 PM, "Tom.Wang" wrote: >>> >>> >>> >>>> Hello, >>>> >>>> Yes, I just check source, we could use sys_readv here. >>>> But there are a limit of 1024 IO segments for each call, maybe it >>>> should not be a problem here. Actually, llite already include such >>>> api (ll_file_readv/writev). Then it should be easy to implement this >>>> by our lib. Sorry for the previous confuse reply. >>>> >>>> Thanks >>>> WangDi >>>> >>>> Eric Barton wrote: >>>> >>>> >>>>> Wangdi, >>>>> >>>>> There seems to be some momentum behind getting readx/writex >>>>> adopted as posix standard system calls. That seems the right >>>>> API to exploit (or anticipate if it's not implemented yet). >>>>> >>>>> Note that the memory and file descriptors are not required to >>>>> be isomorphic (i.e. file and memory fragments don't have to >>>>> correspond directly). >>>>> >>>>> struct iovec { >>>>> void *iov_base; /* Starting address */ >>>>> size_t iov_len; /* Number of bytes */ >>>>> }; >>>>> >>>>> struct xtvec { >>>>> off_t xtv_off; /* Starting file offset */ >>>>> size_t xtv_len; /* Number of bytes */ >>>>> }; >>>>> >>>>> ssize_t readx(int fd, const struct iovec *iov, size_t iov_count, >>>>> struct xtvec *xtv, size_t xtv_count); >>>>> >>>>> ssize_t writex(int fd, const struct iovec *iov, size_t iov_count, >>>>> struct xtvec *xtv, size_t xtv_count); >>>>> >>>>> Cheers, >>>>> Eric >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: lustre-devel-bounces at lists.lustre.org >>>>>> [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Tom.Wang >>>>>> Sent: 12 July 2008 4:38 PM >>>>>> To: Peter Braam >>>>>> Cc: lustre-devel >>>>>> Subject: Re: [Lustre-devel] Vector I/O api >>>>>> >>>>>> >>>>>> Peter Braam wrote: >>>>>> >>>>>> >>>>>> >>>>>>> Tom - >>>>>>> >>>>>>> In a recent call with CERN the request came up to construct a call >>>>>>> that can in parallel transfer an array of extents in a single file to >>>>>>> a list of buffers and vice-versa. >>>>>>> This call should be executed with read-ahead disabled, it will usually >>>>>>> be made when the user is well informed of the I/O that is about to >>>>>>> take place. >>>>>>> Is this easy to get into the Lustre client (using our I/O library)? >>>>>>> Do you have this already for MPI/IO use? >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> Peter >>>>>>> >>>>>>> >>>>>>> >>>>>> Hello, Peter >>>>>> >>>>>> If you mean provide this list buffer read/write API in MPI by our >>>>>> library, it is easy. >>>>>> Because MPI already provide such API, you can define proper >>>>>> discontingous buf_type >>>>>> and file_type of these extents, and use (MPI_File_Write/read_all) to >>>>>> read/write these >>>>>> buffers in one call . We only need disable read-ahead here. So it should >>>>>> be easy to >>>>>> get into our I/O library. >>>>>> >>>>>> But if you mean provide such API in llite, I am not sure it is easy. >>>>>> because it seems we >>>>>> could only use ioctl to implement such non-posix API IMHO, which always >>>>>> has page-size >>>>>> limit for transferring buffers here? It is probably I misunderstand >>>>>> something here. >>>>>> >>>>>> Thanks >>>>>> WangDi >>>>>> >>>>>> This kind of list buffers transferring can be implemented with proper >>>>>> MPI file_view >>>>>> >>>>>> >>>>>> >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Lustre-devel mailing list >>>>>>> Lustre-devel at lists.lustre.org >>>>>>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Tom Wangdi >>>>>> -- >>>>>> Sun Lustre Group >>>>>> System Software Engineer >>>>>> http://www.sun.com >>>>>> >>>>>> _______________________________________________ >>>>>> Lustre-devel mailing list >>>>>> Lustre-devel at lists.lustre.org >>>>>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>>>>> >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Lustre-devel mailing list >>>>> Lustre-devel at lists.lustre.org >>>>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>>>> >>>>> >>>>> >>> >>> >> >> >> > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From jc.lafoucriere at cea.fr Wed Jul 16 10:26:20 2008 From: jc.lafoucriere at cea.fr (Jacques-Charles Lafoucriere) Date: Wed, 16 Jul 2008 12:26:20 +0200 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: References: Message-ID: <487DCCCC.7060206@cea.fr> Space Manager needs are: 1) generate a candidate list for copy out (pre-migration) 2) generate a candidate list for purge For 1) the criteria is : not up to date in HSM and not recently modified For 2) the criteria is : up to date in HSM and not recently accessed Needed changelogs events are "modifications" like : - file creation - mtime change - atime change The things I do not like in events mode are: - if a file is created, filled and remove before copy-out (like a temporary file), we will have useless interaction with the spacemanger (and useless load) - if for some issue, events are missed we will have HSM unknown files in Lustre. To resume this issue we can use a scan or find a way to warranty we will never missed an event. This last point is a strong constraint because Lustre should be able to operate with a dead space manager. I agree, I not fond of scanning, but a low priority, background scan will solve these 2 issues. For me the spacemanager and it's DB are common to all HSM and will have no HSM specific information. HSM specific rules (like in which HSM internal storage class I will put a file) will be managed by HSM copy tool Do you agree ? JC From Peter.Braam at Sun.COM Wed Jul 16 18:10:03 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Wed, 16 Jul 2008 12:10:03 -0600 Subject: [Lustre-devel] Lustre HSM vs ADM HSM Message-ID: During the last few months we have received a few questions about our HSM architecture. This email explains the relationship and differences between an Open Source HSM (ADM from Sun) and the Lustre HSM design (which was done between CEA and some folks from the Lustre team). Likely this discussion applies equally well when comparing Lustre with other HSMs. Many thanks to Rick Matthews for educating me patiently. The conclusions at the end of this email may be of particular interest. References: Lustre HSM architecture: http://arch.lustre.org/index.php?title=HSM_Migration High level design: https://bugzilla.lustre.org/attachment.cgi?id=16341 CEA slides: attached (please post in a findable place on wiki.lustre.org) ADM: http://opensolaris.org/os/project/adm/WhatisADM/ Each of the following sections describes how elements of the Lustre HSM architecture relate to the ADM architecture Event management HSM¹s capture events in the file system for the following purposes: 1. archiving a file 2. restoring a file 3. policy management, such as purging less used files A general mechanism to manage such events is provided by DMAPI which effectively burdens a user space daemon to manage the events. Lustre chose not to use DMAPI. Lustre has multiple servers and events can be detected both on OSS and on MDS nodes. Detecting file IO on OSS nodes allows one to know exactly what region of a file will be read/written and adjust actions accordingly (e.g. If the first 4K of a purged file were left on the disk and that block is read, no HSM restore action is needed). Events are detected by initiators and logged transactionally, so that in case of a power outage no scans of the file system are required. Note that ADM is targeting ZFS DMU data stores, and in these stores searches in the file tree can dynamically generate event logs similar to Lustre logs (Lustre on ZFS will use these searches and abandon its own logging system). A precise mechanism is required to determine which search results are still relevant. Typically many events might be generated by a single system call, and handling this is called filtering. Some filters are always valuable, some filters are user policy (e.g. A decision not to archive mp3 files). The initiators will filter always un-necessary events, such as multiple clients triggering a restore on the same object, and send events to coordinators. The coordinators can be a (failover) collection of load balancing systems, organized in such a manner that events for one file all reach the same coordinator. The coordinator applies further filtering, for example if multiple OSS nodes request the restore of one file the coordinator will make sure that this leads to one action. The coordinator will also implement the optional filters arising from policy. The discussions on the lustre-devel mailing list have mentioned that we would simply extend the policy options by adding it to the coordinator(s). This is not sufficient. Coordinators dispatch events for HSM action to agents. Lustre allows multiple agents to collaborate on one event and the coordinator observes completion by each agents. Agents in turn invoke archiving tools, which might run in user space, to move files to and from the file system, and agents can also abort on-going actions at the request of a coordinator. Another way in which agents can be used is to deliver events to an event manager for a system like ADM. This was not previously considered in the Lustre architecture, but it seems to be a natural way to couple the two systems. If the coordinators handle separate subsets of the file system in a load balanced manner (e.g. By hashing the fids) this might be a good way to horizontally scale ADM. Some events need synchronous handling ­ principally to restore files. Lustre has not addressed how the adaptive timeout system in Lustre works with HSM to enable client wait until file restoration is complete. (For another timeout issue see space management below.) Omissions in the Lustre event management architecture are that (A) no mechanism was introduced to deliver the policy to all the coordinators (through the Lustre management server probably, using the standard configuration lock callbacks), and that (B) no attempt was made to define a policy language with a user interface. Lustre¹s logging of events seems very desirable, but the cancellation of log entries when no longer requires has not yet been architected (and there are other consumers of the logs). Strengths of this architecture are: transactional management of events to eliminate scanning, kernel based filtering close to the source and scalability through all elements of initiators, coordinators and agents. The ADM architecture has an DMAPI event handling mechanism, which is effectively single node, and did not address the multi server issues yet. ADM has a clear interface for managing policy, but the details for a policy language remain under discussion. ADM stores some events in a database in user space. Lustre retains them in the kernel generated logs. HSM metadata After discussion on this list we have decided to implement minimal metadata in the file system to locate archived copies and manage copy-in. This is not different from ADM in its design now, and allows for flexible management of storing multiple versions, and for searches among HSM objects to be performed in a database with suitable indexes (instead of through file system scans). The key bits of the attributes are: 1. an indicating that there is a copy in the archive 2. an indication that the copy in the archive is current 3. an indication that the file is being restored 4. an offset indicating what extent of the file has already been restored 5. file size and disk usage for use by stat(2) while the file is in the archive The HSM database will hold attributes to: 1. dates, owners etc associated with the file for HSM policy (see below) 2. map a lustre FID to a primary HSM object 3. a list of other HSM objects associated with the FID, which can be copied back into a new file in the file system. 4. striping metadata for use by restores like in (2) or bare metal restores This is similar among the two architectures, but Lustre will store this metadata transactionally on the node that manages the object to which the metadata is attached. Policy ­ small files The ADM policy manager (in user space) can instruct the work list generator to build an archive for a collection of small files. The archive can be transferred to tape, instead of the individual small files. In the HSM database FIDS for many small files will point to one object in the archive. Each of the FIDS in the file system will get its own EA to indicate that the file is archived; this is an undesirable issue as it involves a secondary update of the small files. Lustre currently has no design to build a collection of small files. One key issue with this is that the list needs to be retained until sufficiently many small files are available to form a good size archival file ­ doing this in-kernel may be problematic. Archiving many small files can be an Achilles heel for HSM systems. Lustre can distribute archival associated with rapid small file creation over many coordinating nodes. A single HSM database cannot scale in performance to keep up with multiple metadata servers creating new archival events, but multiple independent databases could easily be used and clustered as the coordinators do themselves. If Lustre is coupled to ADM by delivering events to the ADM event manager from a Lustre agent, then small files can be managed by ADM. Kernel based mechanisms to form filesets of small files to be archived might prove much more efficient. For Lustre¹s file set architecture see http://arch.lustre.org/index.php?title=Fileset Policy ­ space management Lustre plans ad-hoc policy for space management. Based on a scan or on least-recently-used kernel log files, files can be selected for purging. Note that such scans rarely need refreshing ­ a single scan should yields candidates for purging for a long time. The log is efficient to maintain and ZFS can likely search through its object tree to produce such lists. A better implementation would allow more flexible policies for space management to be expressed, using the policy language into this. Both lustre and ADM will migrate files typically before they need to be purged ­ this eliminates most performance issues during space management. Both systems have low and high water marks. However, when the file system is really full the space manager may have to invoke archiving. This should be done by asking the coordinator to archive certain files. Policy ­ HSM side A system like ADM will have interfaces to request action on the archived objects ­ e.g. Remove objects archived before 2002. Lustre did not consider such policies, as it intentionally does not want a tight coupling to any particular HSM. This decisions remains fine. An administrative interface is present in both systems to pre-stage files based on a work list. This is to populate the file system with restored copies of all files required by certain jobs. A language to express pre-stage lists is desirable (including user friendly syntax to state ³restore all files in this directory²). Conclusions 1. There are a few issues with both architectures. I will not speak for the ADM project here. 2. There is an excellent opportunity to couple Lustre¹s HSM with ADM 3. There is a interim, much simpler HSM architecture for Lustre that can work with ADM ­ see separate future email to this list 4. Lustre should address small file handling when not coupled to the ADM policy manager. 5. Lustre should define a policy language in relation with filtering and space management 6. Lustre should define a central way to dispatch policy (from the MGS) 7. Lustre should enable adaptive timeouts to assist with ³restore on demand² and ³space management requires archiving² events. 8. Lustre should define release events for log entries and also manage log entry cancellation in the presence of many consumers. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Lustre HSM v6.ppt Type: application/octet-stream Size: 965632 bytes Desc: not available URL: From Peter.Braam at Sun.COM Wed Jul 16 19:00:40 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Wed, 16 Jul 2008 13:00:40 -0600 Subject: [Lustre-devel] How store HSM metadata in MDT ? In-Reply-To: <487DCCCC.7060206@cea.fr> Message-ID: This continues the Lustre design discussion for HSM. On 7/16/08 4:26 AM, "Jacques-Charles Lafoucriere" wrote: > Space Manager needs are: > 1) generate a candidate list for copy out (pre-migration) > 2) generate a candidate list for purge > > For 1) the criteria is : not up to date in HSM and not recently modified > For 2) the criteria is : up to date in HSM and not recently accessed > > Needed changelogs events are "modifications" like : > - file creation > - mtime change > - atime change 1) the files are in the log (and in ZFS the log can be reconstructed through a fast search) The issue here that makes me worried is the following. Is the coordinator managing "archiving" or is the space manager? Whatever entity does it, it needs to WAIT until a file is quiescent for some time. ADM's event manager can do that, but how do we do it with HPSS? Now interestingly the Size On MDS (SOM) project does almost precisely this, it monitors a file going idle and transfers size from OSS's / clients to the MDS inode. So Lustre is pretty close, but this completes too quickly, commonly archiving is postponed 20 minutes or so. > > The things I do not like in events mode are: > - if a file is created, filled and remove before copy-out (like a > temporary file), we will have useless interaction with the spacemanger > (and useless load) A stat call to the file is quick and required anyway to eliminate race conditions. > - if for some issue, events are missed we will have HSM unknown files in > Lustre. To resume this issue we can use a scan or find a way to warranty > we will never missed an event. Lustre logs and ZFS searches are guaranteed NOT to miss anything. No finds are necessary. > This last point is a strong constraint because Lustre should be able to > operate with a dead space manager. > > I agree, I not fond of scanning, but a low priority, background scan > will solve these 2 issues. We only need the scan for 2), and as indicated earlier it can be a rare scan. I will not accept scanning for 1). > > For me the spacemanager and it's DB are common to all HSM and will have > no HSM specific information. And they will be a major bottleneck. I definitely want to avoid a DB. It is fair to state that all events belong with Lustre. Lustre should define adequate high performance features for distributed storage of events. The logs or ZFS searches are a good example, similarly file sets, collecting small files might be good examples. A key consideration for the design is that by integrating it into Lustre we control the performance of these event management systems much better than through upcalls and databases. Keep two systems in sync has proven to be the problem when archiving small files (a ridiculously small number of small files can be archived by current HSM's (100's) we need MILLIONS. So our architecture has to plan a major improvement here. The database should go away. > HSM specific rules (like in which HSM internal storage class I will put > a file) will be managed by HSM copy tool > Do you agree ? Yes. Peter > > JC > > From Peter.Braam at Sun.COM Wed Jul 16 19:29:31 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Wed, 16 Jul 2008 13:29:31 -0600 Subject: [Lustre-devel] Simplified HSM for Lustre? Message-ID: In the discussions Rick and I had we identified a simpler approach for HSM in Lustre which doesn¹t have all the niceties that the Lustre team planned, but might be achievable in short order. Nathan Rutman has implemented an event log for the purpose of replication. Nathan can add ³close² events to this, which are written when a file closes (or when the size on MDS system recovers file sizes from the OSS¹s). This is a sufficient indicator that a file that was opened for write should be archived. It is not necessary ­ we might occasionally re-archive a file that was not written to while open. Nathan also has a tool that retrieves the event log from the kernel for consumption by other agents ­ replication and HSM event managers could both use this. The second thing we need is restore on demand. This can be done when opening a file, instead of doing it when I/O is done to the file. This centralizes these events on the MDS and makes the architecture of this a little simpler for that reason (no initiator to coordinator communication needed). Of course the MDS should track if an open file has already been requested from the HSM to filter events, and it should manage adaptive timeouts for the opening clients. e2scan can be used to generate lists of files to be purged. This is essentially un-changed from the existing architecture. Finally we need a mover (possibly HSM specific ­ e.g. for ADM we would want to move from a Lustre client on Linux to an ADM mover node running Solaris). The mover needs to open files by FID (something that Nathan has implemented for his replication project) and the mover needs to manage EA¹s in the file system, both when archiving and when restoring files. This EA management is also needed when files are migrated or replicated within one file system. The easiest way is for Lustre to feed events to ADM¹s policy manager, since that is know to do most of the right things already. To combine Lustre with other HSM¹s this requires some glue. Yet this glue is not essentially different from what future initiator / coordinator / agent / space manager systems would do, and might be a valuable interim deliverable. We concluded that making this an interim focus for HSM could give Lustre basic HSM functionality long before the full project is complete, and perhaps nicely aligned with ADM¹s planned release(s) during the coming year. Our team¹s project managers will discuss this. Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From Richard.Matthews at Sun.COM Mon Jul 21 15:01:14 2008 From: Richard.Matthews at Sun.COM (Rick Matthews) Date: Mon, 21 Jul 2008 10:01:14 -0500 Subject: [Lustre-devel] Cross posting of ADM information on opensolaris.org Message-ID: <4884A4BA.3010409@Sun.COM> For reference in HSM discussions, I am cross-posing the location of the ADM (Automatic Data Migration) project page within OpenSolaris. Please note the following: 1) The OpenSolaris page refers to ADM as an HSM for ZFS and OpenSolaris based appliances using ZFS. No changes have been made to include potential Lustre consumables. 2) The project page seems to be missing a link to the ADM PSARC case (2007/210). We are working to get that corrected. The PSARC case documentation includes most of the detailed specifications for ADM components. The ADM project page is at: http://opensolaris.org/os/project/adm/ Interesting links from there are: http://opensolaris.org/os/project/adm/WhatisADM/ and the demo on that page. Of course, the link to the PSARC case will also be added. -- --------------------------------------------------------------------- Rick Matthews email: Rick.Matthews at sun.com Sun Microsystems, Inc. phone:+1(651) 554-1518 1270 Eagan Industrial Road phone(internal): 54418 Suite 160 fax: +1(651) 554-1540 Eagan, MN 55121-1231 USA main: +1(651) 554-1500 --------------------------------------------------------------------- From Tom.Wang at Sun.COM Mon Jul 21 18:14:14 2008 From: Tom.Wang at Sun.COM (Tom.Wang) Date: Mon, 21 Jul 2008 14:14:14 -0400 Subject: [Lustre-devel] Lustre IO discussion between CERN and Lustre Message-ID: <4884D1F6.7050109@sun.com> Hello, There is an interesting discussion between CERN and Lustre group about running CERN application on lustre recently. You might be interested The application description from CERN, "Our ROOT framework supports an object persistency system that, in a very first approximation) organizes a data set (say one file) like an RDBMS table, but being column-oriented instead of row-oriented. Our main data structure (Tree) may have several hundred, up to a few thousand branches (columns). The Tree with its N branches is filled from the objects coming from our collisions (called events) and we have zillions of collisions. Each branch is created with a buffer size around 32 KBytes. When the buffer is full, it is compressed (compression factors are typycally between 2 and 5) and written to the file. Files may be between 100 Mbytes and 10 GBytes and we have millions of files. The compression factor is pretty high because our branches contain similar data types for which the compression is typycally 30% better than compressing buffers with non homogeneous types. So a branch may have several thousand buffers in the file. When reading the Tree, in general only a small subset of the N branches is used. In the data structure for our branches,part of the tree header) we keep the file offsets and number of bytes corresponding to each compressed buffer. Our query mechanism (think to an SQL-like query) can pass a vector of pairs (offsets,nbytes) to the I/O sub-system." Interests from CERN group to Lustre 1)Implement a list vector read/write API(readx/writex) for this application. 2)For readx, it could read-ahead buffers from vectors of the pairs provided by users. "when reading .... Our query mechanism (think to an SQL-like query) can pass a vector of pairs (offsets,nbytes) to the I/O sub-system. We simply tell the I/O to return up to a maximum (say 10 Mbytes) of buffers (in general several hundred, a few thousand buffers). We expect the I/O to be clever enough to use the vector of pairs info to organize its internal read-ahead (via threads) such that our next request of 10 Mbytes can be satisfied immediately. " Suggestions from Lustre group. 1)Current lustre read_ahead mechanisms will only be triggered by contiguous or stride IO mode. But Lustre could update the read-ahead mechanisms to do RA according to the vector pairs. 2)For a single client, for each read request, current lustre read mechanism is basicly serialized(for each page), and it should be improved to fire off read request to OSTs parallel by implementing an async read loop on client(llite). 3)This kind of seek-heavy application model(read-size is about 30k, but might discontiguous on file offset)might hit the bottom of server disk IO, so a OSS read-cache might needed for this kind of IO pattern. Given that some OSS servers might concern about RAM, only special files will be enable for this read-cache features. Any other ideas? Thanks WangDi -- Regards, Tom Wangdi -- Sun Lustre Group System Software Engineer http://www.sun.com From Richard.Matthews at Sun.COM Mon Jul 21 20:29:19 2008 From: Richard.Matthews at Sun.COM (Rick Matthews) Date: Mon, 21 Jul 2008 15:29:19 -0500 Subject: [Lustre-devel] Cross posting of ADM information on opensolaris.org In-Reply-To: <4884A4BA.3010409@Sun.COM> References: <4884A4BA.3010409@Sun.COM> Message-ID: <4884F19F.40604@Sun.COM> The link to the PSARC documentation referenced below is: http://opensolaris.org/os/community/arc/policies/fma-event-protocol/ Rick Matthews wrote: > For reference in HSM discussions, I am cross-posing the location of the > ADM (Automatic Data Migration) project page within OpenSolaris. Please > note > the following: > 1) The OpenSolaris page refers to ADM as an HSM for ZFS and > OpenSolaris > based appliances using ZFS. No changes have been made to include > potential > Lustre consumables. > 2) The project page seems to be missing a link to the ADM PSARC > case (2007/210). > We are working to get that corrected. The PSARC case > documentation includes > most of the detailed specifications for ADM components. > > The ADM project page is at: > http://opensolaris.org/os/project/adm/ > > Interesting links from there are: > http://opensolaris.org/os/project/adm/WhatisADM/ > and the demo on that page. Of course, the link to the PSARC case will > also be added. > -- --------------------------------------------------------------------- Rick Matthews email: Rick.Matthews at sun.com Sun Microsystems, Inc. phone:+1(651) 554-1518 1270 Eagan Industrial Road phone(internal): 54418 Suite 160 fax: +1(651) 554-1540 Eagan, MN 55121-1231 USA main: +1(651) 554-1500 --------------------------------------------------------------------- From Tom.Wang at Sun.COM Tue Jul 22 21:44:54 2008 From: Tom.Wang at Sun.COM (Tom.Wang) Date: Tue, 22 Jul 2008 17:44:54 -0400 Subject: [Lustre-devel] [Fwd: Re: Lustre IO discussion between CERN and Lustre] Message-ID: <488654D6.7010303@sun.com> -- Regards, Tom Wangdi -- Sun Lustre Group System Software Engineer http://www.sun.com -------------- next part -------------- An embedded message was scrubbed... From: "Teige, Scott W" Subject: Re: [Lustre-devel] Lustre IO discussion between CERN and Lustre Date: Mon, 21 Jul 2008 16:17:39 -0400 Size: 131334 URL: From josephin at psc.edu Wed Jul 23 01:02:11 2008 From: josephin at psc.edu (Josephine Palencia) Date: Tue, 22 Jul 2008 21:02:11 -0400 (EDT) Subject: [Lustre-devel] Lustre version patchless kernel? In-Reply-To: <4884F19F.40604@Sun.COM> References: <4884A4BA.3010409@Sun.COM> <4884F19F.40604@Sun.COM> Message-ID: For the lustre version, the kernel portion indicates patchless... Should it be something else? [root at operon22 lustre]# more /proc/fs/lustre/version > lustre: 1.9.50 > kernel: patchless > build: 1.9.50-19691231190000-PRISTINE-.extra.linux-2.6.18-92.1.6-2.6.18-92.1.6-lustre-1.9.50 Thanks, josephin From adilger at sun.com Wed Jul 23 01:20:28 2008 From: adilger at sun.com (Andreas Dilger) Date: Tue, 22 Jul 2008 19:20:28 -0600 Subject: [Lustre-devel] Lustre version patchless kernel? In-Reply-To: References: <4884A4BA.3010409@Sun.COM> <4884F19F.40604@Sun.COM> Message-ID: <20080723012028.GC20749@webber.adilger.int> On Jul 22, 2008 21:02 -0400, Josephine Palencia wrote: > > For the lustre version, the kernel portion indicates patchless... > Should it be something else? > > [root at operon22 lustre]# more /proc/fs/lustre/version > > lustre: 1.9.50 > > kernel: patchless > > build: 1.9.50-19691231190000-PRISTINE-.extra.linux-2.6.18-92.1.6-2.6.18-92.1.6-lustre-1.9.50 This is correct. For patched kernels there will be a version number here, indicating the VFS patch API version number. This hasn't changed in a long time because we are no longer changing the kernel VFS patches, and only a few older kernels still need them. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Alex.Zhuravlev at Sun.COM Thu Jul 24 10:19:43 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Thu, 24 Jul 2008 14:19:43 +0400 Subject: [Lustre-devel] statahead feature Message-ID: <4888573F.4010506@sun.com> Hi, due to some experiments with dcache related code we've been doing with shadow and others, it became clear that statahead code is quite complicated. probably for no reason. the most hard part to follow is interaction with dcache. the feature does number of complex things and make other parts (like ll_lookup_it()) harder to follow too. after amount of discussions with people we'd like to share our vision on the feature and propose slightly different solution. we think statahead should do nothing with dcache. it's about inodes and attributes only. thus, it would be good to decouple it from dcache. the only thing statahead should do is: 1) detect statahead is needed (policy, out of the message's scope) 2) scan part of directory (probably using readdir(), skip RPCs) 3) finds/creates inodes for found fids 4) lock these inodes (notice we propose to use inodes as a serialization point so that lockless getattr can be used) 5) issue getattr RPCs (probably lockless) 6) unlock inodes upon getattr's completion then stat(2) is called, it first has to lookup fid by name. for this we can use pagecache just filled with MDS_READDIR. if directory isn't being modified at the time, then entries will be there and we can create dentries in the dcache. they will be valid till UPDATE lock is cancelled - no even LOOKUP lock is needed. another possible thing for optimization is lockless getattr. given most of supported kernel don't pass intent to ->getattr(), it's possible that stat(2) needs two RPCs: one in ll_lookup_it() and another in ll_getattr_it() as lock is released between them. stat(2) gives no warranty about attributes, it gives a shot of them. attributes can change right before userspace application get them. so, why don't we introduce some simple mechanism making attributes valid for short time at least for process executed lookup. this could help statahead as well, we think. comments? suggestions? thanks, Alex From Alexey.Lyashkov at Sun.COM Thu Jul 24 11:35:06 2008 From: Alexey.Lyashkov at Sun.COM (Alex Lyashkov) Date: Thu, 24 Jul 2008 14:35:06 +0300 Subject: [Lustre-devel] statahead feature In-Reply-To: <4888573F.4010506@sun.com> References: <4888573F.4010506@sun.com> Message-ID: <1216899306.16594.68.camel@bear.shadowland> On Thu, 2008-07-24 at 14:19 +0400, Alex Zhuravlev wrote: > we think statahead should do nothing with dcache. it's about inodes > and attributes > only. thus, it would be good to decouple it from dcache. the only > thing statahead > should do is: > 1) detect statahead is needed (policy, out of the message's scope) > 2) scan part of directory (probably using readdir(), skip RPCs) > 3) finds/creates inodes for found fids > 4) lock these inodes (notice we propose to use inodes as a > serialization point > so that lockless getattr can be used) > 5) issue getattr RPCs (probably lockless) > 6) unlock inodes upon getattr's completion other point view - statahead never create dentry or inode, this function (or thread) only send all need lock enqueue to MDS and put granted lock to cache. user thread will call ->lookup and will be sleep until lock is granted on mdc_enqueue so none new wait queue need. At this case statahead will be very simple - call ->readdir() in background, in fill callback skip dot/dotdot files, skip hidden files if it not need, possible something other. in result new rpc added to ptlrpcd or own set. and very simple interpret callback which only process granted lock. -- Alex Lyashkov Lustre Group, Sun Microsystems From Peter.Braam at Sun.COM Thu Jul 24 16:27:28 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Thu, 24 Jul 2008 10:27:28 -0600 Subject: [Lustre-devel] statahead feature In-Reply-To: <4888573F.4010506@sun.com> Message-ID: I strongly agree with this. A good way to verify if we have a favorable implementation is to see if it can be ported to other OS's. Peter On 7/24/08 4:19 AM, "Alex Zhuravlev" wrote: > Hi, > > due to some experiments with dcache related code we've been doing with shadow > and others, it became clear that statahead code is quite complicated. probably > for no reason. the most hard part to follow is interaction with dcache. the > feature does number of complex things and make other parts (like > ll_lookup_it()) > harder to follow too. > > after amount of discussions with people we'd like to share our vision on the > feature and propose slightly different solution. > > we think statahead should do nothing with dcache. it's about inodes and > attributes > only. thus, it would be good to decouple it from dcache. the only thing > statahead > should do is: > 1) detect statahead is needed (policy, out of the message's scope) > 2) scan part of directory (probably using readdir(), skip RPCs) > 3) finds/creates inodes for found fids > 4) lock these inodes (notice we propose to use inodes as a serialization point > so that lockless getattr can be used) > 5) issue getattr RPCs (probably lockless) > 6) unlock inodes upon getattr's completion > > then stat(2) is called, it first has to lookup fid by name. for this we can > use > pagecache just filled with MDS_READDIR. if directory isn't being modified at > the > time, then entries will be there and we can create dentries in the dcache. > they > will be valid till UPDATE lock is cancelled - no even LOOKUP lock is needed. > > > another possible thing for optimization is lockless getattr. given most of > supported > kernel don't pass intent to ->getattr(), it's possible that stat(2) needs two > RPCs: > one in ll_lookup_it() and another in ll_getattr_it() as lock is released > between them. > stat(2) gives no warranty about attributes, it gives a shot of them. > attributes can > change right before userspace application get them. so, why don't we introduce > some > simple mechanism making attributes valid for short time at least for process > executed > lookup. this could help statahead as well, we think. > > comments? suggestions? > > thanks, Alex > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From Alex.Zhuravlev at Sun.COM Thu Jul 24 18:56:39 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Thu, 24 Jul 2008 22:56:39 +0400 Subject: [Lustre-devel] statahead feature In-Reply-To: References: Message-ID: <4888D067.204@sun.com> it's actually very interesting point - how much of lustre internals we should expose outside? I mean flexibility vs. complexity issue in context of portable API. thanks, Alex Peter Braam wrote: > I strongly agree with this. A good way to verify if we have a favorable > implementation is to see if it can be ported to other OS's. > > Peter > > > On 7/24/08 4:19 AM, "Alex Zhuravlev" wrote: > >> Hi, >> >> due to some experiments with dcache related code we've been doing with shadow >> and others, it became clear that statahead code is quite complicated. probably >> for no reason. the most hard part to follow is interaction with dcache. the >> feature does number of complex things and make other parts (like >> ll_lookup_it()) >> harder to follow too. >> >> after amount of discussions with people we'd like to share our vision on the >> feature and propose slightly different solution. >> >> we think statahead should do nothing with dcache. it's about inodes and >> attributes >> only. thus, it would be good to decouple it from dcache. the only thing >> statahead >> should do is: >> 1) detect statahead is needed (policy, out of the message's scope) >> 2) scan part of directory (probably using readdir(), skip RPCs) >> 3) finds/creates inodes for found fids >> 4) lock these inodes (notice we propose to use inodes as a serialization point >> so that lockless getattr can be used) >> 5) issue getattr RPCs (probably lockless) >> 6) unlock inodes upon getattr's completion >> >> then stat(2) is called, it first has to lookup fid by name. for this we can >> use >> pagecache just filled with MDS_READDIR. if directory isn't being modified at >> the >> time, then entries will be there and we can create dentries in the dcache. >> they >> will be valid till UPDATE lock is cancelled - no even LOOKUP lock is needed. >> >> >> another possible thing for optimization is lockless getattr. given most of >> supported >> kernel don't pass intent to ->getattr(), it's possible that stat(2) needs two >> RPCs: >> one in ll_lookup_it() and another in ll_getattr_it() as lock is released >> between them. >> stat(2) gives no warranty about attributes, it gives a shot of them. >> attributes can >> change right before userspace application get them. so, why don't we introduce >> some >> simple mechanism making attributes valid for short time at least for process >> executed >> lookup. this could help statahead as well, we think. >> >> comments? suggestions? >> >> thanks, Alex >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel > > From Yong.Fan at Sun.COM Fri Jul 25 04:34:52 2008 From: Yong.Fan at Sun.COM (Yong Fan) Date: Fri, 25 Jul 2008 12:34:52 +0800 Subject: [Lustre-devel] statahead feature In-Reply-To: <4888573F.4010506@sun.com> References: <4888573F.4010506@sun.com> Message-ID: <488957EC.1040701@sun.com> Alex Zhuravlev 写道: > Hi, > > due to some experiments with dcache related code we've been doing with > shadow > and others, it became clear that statahead code is quite complicated. > probably > for no reason. the most hard part to follow is interaction with > dcache. the > feature does number of complex things and make other parts (like > ll_lookup_it()) > harder to follow too. > > after amount of discussions with people we'd like to share our vision > on the > feature and propose slightly different solution. > > we think statahead should do nothing with dcache. it's about inodes > and attributes > only. thus, it would be good to decouple it from dcache. the only > thing statahead > should do is: > 1) detect statahead is needed (policy, out of the message's scope) > 2) scan part of directory (probably using readdir(), skip RPCs) > 3) finds/creates inodes for found fids > 4) lock these inodes (notice we propose to use inodes as a > serialization point > so that lockless getattr can be used) > 5) issue getattr RPCs (probably lockless) > 6) unlock inodes upon getattr's completion > > then stat(2) is called, it first has to lookup fid by name. for this > we can use > pagecache just filled with MDS_READDIR. if directory isn't being > modified at the > time, then entries will be there and we can create dentries in the > dcache. they > will be valid till UPDATE lock is cancelled - no even LOOKUP lock is > needed. > > > another possible thing for optimization is lockless getattr. given > most of supported > kernel don't pass intent to ->getattr(), it's possible that stat(2) > needs two RPCs: > one in ll_lookup_it() and another in ll_getattr_it() as lock is > released between them. > stat(2) gives no warranty about attributes, it gives a shot of them. > attributes can > change right before userspace application get them. so, why don't we > introduce some > simple mechanism making attributes valid for short time at least for > process executed > lookup. this could help statahead as well, we think. Lockless getattr RPC for "ls -l" is a good idea. But whether all the getattr RPC will be lockless or not? If not, how to distinguish them from "stat(2)" case? by intent or something else? Statahead is based on forecast, it maybe wrong. We should guarantee the proper lock(s) is held even if the statahead is wrong. Another optimization (maybe) to be considered is that whether it is necessary to start one statahead thread for each "ls -l" operation or not? As said by Nikita, maybe we can use a single thread for all the statahead. Thanks! -- Fan Yong > > comments? suggestions? > > thanks, Alex > From Alex.Zhuravlev at Sun.COM Fri Jul 25 04:41:30 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Fri, 25 Jul 2008 08:41:30 +0400 Subject: [Lustre-devel] statahead feature In-Reply-To: <488957EC.1040701@sun.com> References: <4888573F.4010506@sun.com> <488957EC.1040701@sun.com> Message-ID: <4889597A.6060109@sun.com> Yong Fan wrote: > Lockless getattr RPC for "ls -l" is a good idea. But whether all the > getattr RPC will be lockless or not? > If not, how to distinguish them from "stat(2)" case? by intent or > something else? > Statahead is based on forecast, it maybe wrong. We should guarantee the > proper lock(s) is held even > if the statahead is wrong. no, I don't think we need lock always: you just did getattr and fill inode with fresh attributes, then you release lock, somebody changes attributes on the server and only then userspace application gets attributes (already non-fresh). so, what lock gives you in this case? > Another optimization (maybe) to be considered is that whether it is > necessary to start one statahead thread > for each "ls -l" operation or not? As said by Nikita, maybe we can use a > single thread for all the statahead. I'm not sure we need any statahead thread at all. what's wrong with issuing number of async RPCs from ll_getattr_it()? this way user's application drives statahead directly: each time stat(2) is called you tune statahead window and send few more RPCs again -- like data read-ahead does. thanks, Alex From Alex.Zhuravlev at Sun.COM Fri Jul 25 04:44:22 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Fri, 25 Jul 2008 08:44:22 +0400 Subject: [Lustre-devel] statahead feature In-Reply-To: <488957EC.1040701@sun.com> References: <4888573F.4010506@sun.com> <488957EC.1040701@sun.com> Message-ID: <48895A26.8010006@sun.com> Yong Fan wrote: > Lockless getattr RPC for "ls -l" is a good idea. But whether all the > getattr RPC will be lockless or not? forgot to answer this ... I believe it's to be decided by the server who knows how given inode is contended. we do the same for i_size in glimpse path. thanks, Alex From Alex.Zhuravlev at Sun.COM Fri Jul 25 05:17:04 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Fri, 25 Jul 2008 09:17:04 +0400 Subject: [Lustre-devel] statahead feature In-Reply-To: <37e3b1470807242204x6ab9037bif384e8a37595b6c6@mail.gmail.com> References: <4888573F.4010506@sun.com> <488957EC.1040701@sun.com> <4889597A.6060109@sun.com> <37e3b1470807242204x6ab9037bif384e8a37595b6c6@mail.gmail.com> Message-ID: <488961D0.9060709@sun.com> Ragnar Kjørstad wrote: > On Thu, Jul 24, 2008 at 9:41 PM, Alex Zhuravlev wrote: >> no, I don't think we need lock always: you just did getattr and fill inode >> with fresh attributes, then you release lock, somebody changes attributes >> on the server and only then userspace application gets attributes (already >> non-fresh). so, what lock gives you in this case? > > If I have an application that does: > 1. Update stat data on one node. > 2. Barrier > 3. Read the same stat data on a different node > > I should be guaranteed to get the correct data, right? > So stat should return data that was valid either when the system call > started, when it ended, or somewhere in between. (not data that was > invalid even before the systemcall started) indeed. we can't use this trick in all the cases. I think it's a subject of discussion what we consider a barrier, for regular stat(2) it can be start of system call. for ls -l case it can be open(2). isn't this a reasonable cost for significant performance improvement of ls -l ? my understanding is that for NFS with readdir+ support, the barrier will be open(2). thanks for this good point, Alex From Alexey.Lyashkov at Sun.COM Fri Jul 25 09:37:56 2008 From: Alexey.Lyashkov at Sun.COM (Alex Lyashkov) Date: Fri, 25 Jul 2008 12:37:56 +0300 Subject: [Lustre-devel] statahead feature In-Reply-To: <4889597A.6060109@sun.com> References: <4888573F.4010506@sun.com> <488957EC.1040701@sun.com> <4889597A.6060109@sun.com> Message-ID: <1216978676.16594.104.camel@bear.shadowland> On Fri, 2008-07-25 at 08:41 +0400, Alex Zhuravlev wrote: > > > Another optimization (maybe) to be considered is that whether it is > > necessary to start one statahead thread > > for each "ls -l" operation or not? As said by Nikita, maybe we can use a > > single thread for all the statahead. > > I'm not sure we need any statahead thread at all. what's wrong with issuing > number of async RPCs from ll_getattr_it()? this way user's application drives > statahead directly: each time stat(2) is called you tune statahead window and > send few more RPCs again -- like data read-ahead does. > not need any statahead thread. as alex say: if we have UPDATE lock to parent - we can create valid dentry without lookup and add new locked inode to him. --- this looks easy to implement - call ll_readdir with i_mutex held and pass custom fill callback. in callback we allocate new dentry + empty locked inode (or attach new dentry to inode if we have only update lock /ll_find_alias/ ?? ), also submit async getattr rpc into ptlrpcd if need. rpc completion callback set md_lock to inode and unlock inode. also ll_getattr_it should call wait_on_inode for be sure statahead is finished. -- Alex Lyashkov Lustre Group, Sun Microsystems From Alex.Zhuravlev at Sun.COM Fri Jul 25 12:29:07 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Fri, 25 Jul 2008 16:29:07 +0400 Subject: [Lustre-devel] statahead feature In-Reply-To: <1216978676.16594.104.camel@bear.shadowland> References: <4888573F.4010506@sun.com> <488957EC.1040701@sun.com> <4889597A.6060109@sun.com> <1216978676.16594.104.camel@bear.shadowland> Message-ID: <4889C713.7050801@sun.com> Alex Lyashkov wrote: > this looks easy to implement - call ll_readdir with i_mutex held and > pass custom fill callback. > in callback we allocate new dentry + empty locked inode (or attach new > dentry to inode if we have only update lock /ll_find_alias/ ?? ), also > submit async getattr rpc into ptlrpcd if need. > rpc completion callback set md_lock to inode and unlock inode. > also ll_getattr_it should call wait_on_inode for be sure statahead is > finished. I think it'd be better to get common infrastructure in ll_lookup_it() which can resolve name to fid from readdir's cache. having that in lookup path means any syscall can benefit from it, not statahead only. for example, we can do open by fid on mds and mds won't need to resolve name by himself. probably, this infrastructure could be used for WBC. thanks, Alex From adilger at sun.com Fri Jul 25 19:41:09 2008 From: adilger at sun.com (Andreas Dilger) Date: Fri, 25 Jul 2008 15:41:09 -0400 Subject: [Lustre-devel] statahead feature In-Reply-To: <1216978676.16594.104.camel@bear.shadowland> References: <4888573F.4010506@sun.com> <488957EC.1040701@sun.com> <4889597A.6060109@sun.com> <1216978676.16594.104.camel@bear.shadowland> Message-ID: <20080725194109.GV5346@webber.adilger.int> On Jul 25, 2008 12:37 +0300, Alexey Lyashkov wrote: > On Fri, 2008-07-25 at 08:41 +0400, Alex Zhuravlev wrote: > > > Another optimization (maybe) to be considered is that whether it is > > > necessary to start one statahead thread > > > for each "ls -l" operation or not? As said by Nikita, maybe we can use a > > > single thread for all the statahead. > > > > I'm not sure we need any statahead thread at all. what's wrong with issuing > > number of async RPCs from ll_getattr_it()? this way user's application > > drives statahead directly: each time stat(2) is called you tune statahead > > window and send few more RPCs again -- like data read-ahead does. I think it is very useful to consider these "lockless MDT getattr" RPCs as "glimpse" requests. That idea has served us very well on OST attributes, and I think the same would true with MDT attributes. We should always send these getattr requests with a special flag (is a "getattr intent" enough, or do we need a different flag). We should ideally make this DLM RPC work the same way for both OST glimpses and MDT glimpses - client can optionally be granted a lock if inode has not been changed recently, but only the attributes returned if the lock is busy. > not need any statahead thread. > as alex say: > if we have UPDATE lock to parent - we can create valid dentry without > lookup and add new locked inode to him. > --- > this looks easy to implement - call ll_readdir with i_mutex held and > pass custom fill callback. One problem with this idea is that we cannot do the callbacks in the context of the ptlrpcd thread - see bug 15927. > in callback we allocate new dentry + empty locked inode (or attach new > dentry to inode if we have only update lock /ll_find_alias/ ?? ), also > submit async getattr rpc into ptlrpcd if need. > rpc completion callback set md_lock to inode and unlock inode. > also ll_getattr_it should call wait_on_inode for be sure statahead is > finished. IMHO it would also be useful to start the OST glimpse in the callback as soon as the MDT "glimpse" is returned. Without a change like this we are limited to <= 2x speedup for statahead until size-on-MDS is done. That is because we are hiding the MDT RPC latency, but still have to wait for the parallel OST RPC latency, so: speedup = (old time / new time) = (MDT RPC + max(OST RPC)) / max(OST RPC) Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Alex.Zhuravlev at Sun.COM Fri Jul 25 19:50:13 2008 From: Alex.Zhuravlev at Sun.COM (Alex Zhuravlev) Date: Fri, 25 Jul 2008 23:50:13 +0400 Subject: [Lustre-devel] statahead feature In-Reply-To: <20080725194109.GV5346@webber.adilger.int> References: <4888573F.4010506@sun.com> <488957EC.1040701@sun.com> <4889597A.6060109@sun.com> <1216978676.16594.104.camel@bear.shadowland> <20080725194109.GV5346@webber.adilger.int> Message-ID: <488A2E75.80901@sun.com> Andreas Dilger wrote: > IMHO it would also be useful to start the OST glimpse in the callback as > soon as the MDT "glimpse" is returned. Without a change like this we > are limited to <= 2x speedup for statahead until size-on-MDS is done. > That is because we are hiding the MDT RPC latency, but still have to wait > for the parallel OST RPC latency, so: > > speedup = (old time / new time) = (MDT RPC + max(OST RPC)) / max(OST RPC) definitely thanks, Alex From bob.kossey at hp.com Wed Jul 23 03:22:15 2008 From: bob.kossey at hp.com (Kossey, Robert) Date: Tue, 22 Jul 2008 23:22:15 -0400 Subject: [Lustre-devel] Lustre version patchless kernel? In-Reply-To: <4886A29C.2000009@hp.com> References: <4886A29C.2000009@hp.com> Message-ID: <4886A3E7.6050904@hp.com> We also see this on recent releases like 1.6.5.1. Why do you bother to distribute kernel-lustre-smp RPMs if the kernels are not patched? Thanks, Bob >> / > />/ For the lustre version, the kernel portion indicates patchless... > />/ Should it be something else? > />/ />/ [root at operon22 > lustre]# more > /proc/fs/lustre/version > />/ > lustre: 1.9.50 > />/ > kernel: patchless > />/ > build: > 1.9.50-19691231190000-PRISTINE-.extra.linux-2.6.18-92.1.6-2.6.18-92.1.6-lustre-1.9.50 > > / > This is correct. For patched kernels there will be a version number > here, > indicating the VFS patch API version number. This hasn't changed in a > long time because we are no longer changing the kernel VFS patches, and > only a few older kernels still need them. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > From rk at ragnarkjorstad.com Fri Jul 25 05:04:53 2008 From: rk at ragnarkjorstad.com (=?ISO-8859-1?Q?Ragnar_Kj=F8rstad?=) Date: Thu, 24 Jul 2008 22:04:53 -0700 Subject: [Lustre-devel] statahead feature In-Reply-To: <4889597A.6060109@sun.com> References: <4888573F.4010506@sun.com> <488957EC.1040701@sun.com> <4889597A.6060109@sun.com> Message-ID: <37e3b1470807242204x6ab9037bif384e8a37595b6c6@mail.gmail.com> On Thu, Jul 24, 2008 at 9:41 PM, Alex Zhuravlev wrote: > no, I don't think we need lock always: you just did getattr and fill inode > with fresh attributes, then you release lock, somebody changes attributes > on the server and only then userspace application gets attributes (already > non-fresh). so, what lock gives you in this case? If I have an application that does: 1. Update stat data on one node. 2. Barrier 3. Read the same stat data on a different node I should be guaranteed to get the correct data, right? So stat should return data that was valid either when the system call started, when it ended, or somewhere in between. (not data that was invalid even before the systemcall started) -- Ragnar Kjørstad From Brian.Murrell at Sun.COM Mon Jul 28 13:03:03 2008 From: Brian.Murrell at Sun.COM (Brian J. Murrell) Date: Mon, 28 Jul 2008 09:03:03 -0400 Subject: [Lustre-devel] Lustre version patchless kernel? In-Reply-To: <4886A3E7.6050904@hp.com> References: <4886A29C.2000009@hp.com> <4886A3E7.6050904@hp.com> Message-ID: <1217250183.22114.110.camel@pc.ilinx> On Tue, 2008-07-22 at 23:22 -0400, Kossey, Robert wrote: > We also see this on recent releases like 1.6.5.1. Why do you bother to > distribute kernel-lustre-smp RPMs if the kernels are not patched? MDS and OSS kernels *are* patched. My 1.6.5 here does not report it's patchless: lustre: 1.6.5 kernel: 47 build: 1.6.5-19691231190000-PRISTINE-.cache.OLDRPMS.20080513015618.linux-smp-2.6.9-67.0.7.EL_lustre.1.6.5.i686-2.6.9-67.0.7.EL_lustre.1.6.5smp If you are using our kernel on your MDS/OSS and it's reporting patchless, this indeed is a bug. Could you please search our bugzilla to see if there is a similar bug and if not found, file a bug in our bugzilla about it? b. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From adilger at sun.com Mon Jul 28 16:07:49 2008 From: adilger at sun.com (Andreas Dilger) Date: Mon, 28 Jul 2008 10:07:49 -0600 Subject: [Lustre-devel] Lustre version patchless kernel? In-Reply-To: <1217250183.22114.110.camel@pc.ilinx> References: <4886A29C.2000009@hp.com> <4886A3E7.6050904@hp.com> <1217250183.22114.110.camel@pc.ilinx> Message-ID: <20080728160749.GB3342@webber.adilger.int> On Jul 28, 2008 09:03 -0400, Brian J. Murrell wrote: > On Tue, 2008-07-22 at 23:22 -0400, Kossey, Robert wrote: > > We also see this on recent releases like 1.6.5.1. Why do you bother to > > distribute kernel-lustre-smp RPMs if the kernels are not patched? > > MDS and OSS kernels *are* patched. My 1.6.5 here does not report it's > patchless: > > lustre: 1.6.5 > kernel: 47 > build: 1.6.5-19691231190000-PRISTINE-.cache.OLDRPMS.20080513015618.linux-smp-2.6.9-67.0.7.EL_lustre.1.6.5.i686-2.6.9-67.0.7.EL_lustre.1.6.5smp > > If you are using our kernel on your MDS/OSS and it's reporting > patchless, this indeed is a bug. Could you please search our bugzilla > to see if there is a similar bug and if not found, file a bug in our > bugzilla about it? Brian, you are being confused by exactly what "patchless" means. In the context here, "patchless" means "no VFS patches" and that "clients can run on patchless kernels". This has nothing to do with whether the kernel has any of our patches on it. Any server kernel HAS to have patches in order to run. The fact that your kernel reports a "kernel: 47" is because that is the version of the VFS patch API that is being used on your RHEL4 2.6.9 kernel. Newer kernels (RHEL5, SLES10, vanilla 2.6.16+) will report "patchless" here. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From adilger at sun.com Mon Jul 28 22:02:58 2008 From: adilger at sun.com (Andreas Dilger) Date: Mon, 28 Jul 2008 16:02:58 -0600 Subject: [Lustre-devel] statahead feature In-Reply-To: <37e3b1470807242204x6ab9037bif384e8a37595b6c6@mail.gmail.com> References: <4888573F.4010506@sun.com> <488957EC.1040701@sun.com> <4889597A.6060109@sun.com> <37e3b1470807242204x6ab9037bif384e8a37595b6c6@mail.gmail.com> Message-ID: <20080728220258.GE3342@webber.adilger.int> On Jul 24, 2008 22:04 -0700, Ragnar Kj�rstad wrote: > On Thu, Jul 24, 2008 at 9:41 PM, Alex Zhuravlev wrote: > > no, I don't think we need lock always: you just did getattr and fill inode > > with fresh attributes, then you release lock, somebody changes attributes > > on the server and only then userspace application gets attributes (already > > non-fresh). so, what lock gives you in this case? > > If I have an application that does: > 1. Update stat data on one node. > 2. Barrier > 3. Read the same stat data on a different node > > I should be guaranteed to get the correct data, right? > So stat should return data that was valid either when the system call > started, when it ended, or somewhere in between. (not data that was > invalid even before the systemcall started) That depends on what stat data was changed. If it was e.g. atime then there is no guarantee that the other node will get an updated copy. If there is a file write that updates mtime then there is no guarantee that other clients will get an updated mtime if they already have a file lock. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From bob.kossey at hp.com Mon Jul 28 14:38:08 2008 From: bob.kossey at hp.com (Kossey, Robert) Date: Mon, 28 Jul 2008 10:38:08 -0400 Subject: [Lustre-devel] Lustre version patchless kernel? In-Reply-To: <488DD7C1.7010807@hp.com> References: <488DD7C1.7010807@hp.com> Message-ID: <488DD9D0.5080902@hp.com> Yes, we see patchless reported on our RHEL5U1 Opteron servers with 1.6.5.1and 1.6.4.3: root at njxc4ad ~]# cat /proc/fs/lustre/version lustre: 1.6.5.1 kernel: patchless build: 1.6.5.1-19691231190000-PRISTINE-.cache.OLDRPMS.20080618233454.linux-smp- 2.6.18-53.1.14.el5_lustre.1.6.5.1.x86_64-2.6.18-53.1.14.el5_lustre.1.6.5.1smp [root at njxc4ad ~]# uname -a Linux njxc4ad.usa.hp.com 2.6.18-53.1.14.el5_lustre.1.6.5.1smp #1 SMP Wed Jun 18 19:45:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux [root at njxc4ad ~]# [root at ichpcsfsd1 ~]# cat /proc/fs/lustre/version lustre: 1.6.5 kernel: patchless build: 1.6.5-19691231190000-PRISTINE-.usr.src.linux-2.6.18-53.1.13.el5_lustre.1 .6.4.3-2.6.18-53.1.13.el5_lustre.1.6.4.3smp [root at ichpcsfsd1 ~]# uname -a Linux ichpcsfsd1 2.6.18-53.1.13.el5_lustre.1.6.4.3smp #1 SMP Sun Feb 17 08:38:44 EST 2008 x86_64 x86_64 x86_64 GNU/Linux [root at ichpcsfsd1 ~]# In both cases, these nodes are running kernels from the kernel-lustre-smp RPMs downloaded from the Sun Lustre site. I don't see a bug reported for this, so I will submit one. Thanks, Bob > >> / We also see this on recent releases like 1.6.5.1. Why do you >> bother to > />/ distribute kernel-lustre-smp RPMs if the kernels are not patched? > / > MDS and OSS kernels *are* patched. My 1.6.5 here does not report it's > patchless: > > lustre: 1.6.5 > kernel: 47 > build: > 1.6.5-19691231190000-PRISTINE-.cache.OLDRPMS.20080513015618.linux-smp-2.6.9-67.0.7.EL_lustre.1.6.5.i686-2.6.9-67.0.7.EL_lustre.1.6.5smp > > > If you are using our kernel on your MDS/OSS and it's reporting > patchless, this indeed is a bug. Could you please search our bugzilla > to see if there is a similar bug and if not found, file a bug in our > bugzilla about it? > > b. > > From lesovo at gmail.com Tue Jul 29 18:25:54 2008 From: lesovo at gmail.com (SoVo Le) Date: Tue, 29 Jul 2008 11:25:54 -0700 Subject: [Lustre-devel] Centos 5.2 with Lustre 1.6.5.1 Message-ID: <52ce537f0807291125r5d27fcf1m5710838a8e27e93c@mail.gmail.com> Hi, If someone could help me out on this: Why can't I mount lustre ? [root at lustre_server ~]# mkfs.lustre --reformat --device-size=300000 --mdt --mgs --param lov.stripcount=3 --param lov.stripesize=4M /dev/sdb Permanent disk data: Target: lustre-MDTffff Index: unassigned Lustre FS: lustre Mount type: ldiskfs Flags: 0x75 (MDT MGS needs_index first_time update ) Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr Parameters: lov.stripcount=3 lov.stripesize=4M mdt.group_upcall=/usr/sbin/l_getgroups 2 6 18 formatting backing filesystem ldiskfs on /dev/sdb target name lustre-MDTffff 4k blocks 75000 options -i 4096 -I 512 -q -O dir_index,uninit_groups -F mkfs_cmd = mkfs.ext2 -j -b 4096 -L lustre-MDTffff -i 4096 -I 512 -q -O dir_index,uninit_groups -F /dev/sdb 75000 Writing CONFIGS/mountdata [root at lustre_server ~]# mount -t lustre /dev/sdb /mnt/mdt [root at lustre_server ~]# mkdir /mnt/mdt [root at lustre_server ~]# mount -t lustre /dev/sdb /mnt/mdt mount.lustre: mount /dev/sdb at /mnt/mdt failed: *Operation not supported * /var/log/messages shows: Jul 29 11:14:47 lustre_server kernel: Lustre: Skipped 1 previous similar message Jul 29 11:14:47 lustre_server kernel: Lustre: *** setting obd lustre-MDT0000 device 'unknown-block(8,16)' read-only *** Jul 29 11:14:47 lustre_server kernel: Turning device sdb (0x800010) read-only Jul 29 11:14:47 lustre_server kernel: Lustre: lustre-MDT0000: shutting down for failover; client state will be preserved. Jul 29 11:14:47 lustre_server kernel: Lustre: MDT lustre-MDT0000 has stopped. Jul 29 11:14:47 lustre_server kernel: LustreError: 14992:0:(ldlm_request.c:986:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway Jul 29 11:14:47 lustre_server kernel: LustreError: 14992:0:(ldlm_request.c:1575:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Jul 29 11:14:47 lustre_server kernel: Lustre: MGS has stopped. Jul 29 11:14:47 lustre_server kernel: Removing read-only on sdb (0x800010) Jul 29 11:14:47 lustre_server kernel: Lustre: server umount lustre-MDT0000 complete TIA PS: I have no problem mounting lustre on RedHat 5.2 with Lustre 1.6.5.1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at sun.com Wed Jul 30 09:35:50 2008 From: adilger at sun.com (Andreas Dilger) Date: Wed, 30 Jul 2008 03:35:50 -0600 Subject: [Lustre-devel] Lustre version patchless kernel? In-Reply-To: <488DD9D0.5080902@hp.com> References: <488DD7C1.7010807@hp.com> <488DD9D0.5080902@hp.com> Message-ID: <20080730093550.GS3342@webber.adilger.int> On Jul 28, 2008 10:38 -0400, Kossey, Robert wrote: > Yes, we see patchless reported on our RHEL5U1 Opteron servers with > 1.6.5.1and 1.6.4.3: > > In both cases, these nodes are running kernels from the > kernel-lustre-smp RPMs downloaded from the Sun Lustre site. I don't see > a bug reported for this, so I will submit one. You missed my reply. This is normal, please do not file a bug. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Brian.Murrell at Sun.COM Wed Jul 30 14:41:20 2008 From: Brian.Murrell at Sun.COM (Brian J. Murrell) Date: Wed, 30 Jul 2008 10:41:20 -0400 Subject: [Lustre-devel] Centos 5.2 with Lustre 1.6.5.1 In-Reply-To: <52ce537f0807291125r5d27fcf1m5710838a8e27e93c@mail.gmail.com> References: <52ce537f0807291125r5d27fcf1m5710838a8e27e93c@mail.gmail.com> Message-ID: <1217428880.13569.247.camel@pc.ilinx> On Tue, 2008-07-29 at 11:25 -0700, SoVo Le wrote: > Hi, > > If someone could help me out on this: Why can't I mount lustre ? This list is for discussions of lustre development only. Please use lustre-discuss for questions such as this. I have an answer for your question but in order to prevent any further follow-up to this list of this topic I will hold off answering it until you post your question to lustre-discuss. Cheers, b. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From adilger at Sun.COM Wed Jul 30 21:19:26 2008 From: adilger at Sun.COM (Andreas Dilger) Date: Wed, 30 Jul 2008 15:19:26 -0600 Subject: [Lustre-devel] Lustre version patchless kernel? In-Reply-To: <93703EDD1F125544A59BD25384B175621F5A36DCF0@G1W0485.americas.hpqcorp.net> References: <488DD7C1.7010807@hp.com> <488DD9D0.5080902@hp.com> <20080730093550.GS3342@webber.adilger.int> <93703EDD1F125544A59BD25384B175621F5A36DCF0@G1W0485.americas.hpqcorp.net> Message-ID: <20080730211926.GA3342@webber.adilger.int> On Jul 30, 2008 15:52 +0000, Kossey, Robert wrote: > Thanks for the explanation. I actually sent my response, and created > bug 16516, before you sent your response, but my posting was delayed > by the list moderator since I am not a list subscriber. I modifed the > bug to request that the wording of the program output be changed to > something like: > > kernel: no VFS patches > > This could reduce the confusion that Brian and I and probably others > would have with the current wording. Yes, the wording will be changed to "kernel: patchless client", because several of the regression tests depend on "patchless" being there. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From bob.kossey at hp.com Wed Jul 30 15:52:26 2008 From: bob.kossey at hp.com (Kossey, Robert) Date: Wed, 30 Jul 2008 15:52:26 +0000 Subject: [Lustre-devel] Lustre version patchless kernel? In-Reply-To: <20080730093550.GS3342@webber.adilger.int> References: <488DD7C1.7010807@hp.com> <488DD9D0.5080902@hp.com> <20080730093550.GS3342@webber.adilger.int> Message-ID: <93703EDD1F125544A59BD25384B175621F5A36DCF0@G1W0485.americas.hpqcorp.net> Hi Andreas, Thanks for the explanation. I actually sent my response, and created bug 16516, before you sent your response, but my posting was delayed by the list moderator since I am not a list subscriber. I modifed the bug to request that the wording of the program output be changed to something like: kernel: no VFS patches This could reduce the confusion that Brian and I and probably others would have with the current wording. Thanks, Bob > -----Original Message----- > From: Andreas.Dilger at sun.com [mailto:Andreas.Dilger at sun.com] > On Behalf Of Andreas Dilger > Sent: Wednesday, July 30, 2008 5:36 AM > To: Kossey, Robert > Cc: lustre-devel at lists.lustre.org > Subject: Re: [Lustre-devel] Lustre version patchless kernel? > > On Jul 28, 2008 10:38 -0400, Kossey, Robert wrote: > > Yes, we see patchless reported on our RHEL5U1 Opteron servers with > > 1.6.5.1and 1.6.4.3: > > > > In both cases, these nodes are running kernels from the > > kernel-lustre-smp RPMs downloaded from the Sun Lustre site. I don't > > see a bug reported for this, so I will submit one. > > You missed my reply. This is normal, please do not file a bug. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > >