From Ricardo.M.Correia at Sun.COM Mon Apr 14 19:14:04 2008 From: Ricardo.M.Correia at Sun.COM (Ricardo M. Correia) Date: Mon, 14 Apr 2008 20:14:04 +0100 Subject: [Lustre-devel] RFC: DMU Force Export arch page Message-ID: <1208200444.16993.27.camel@localhost> Hi Eric, Peter, Andreas, Alex and anyone else interested, Can you review the DMU Force Export architecture page to see if anything is wrong or missing? This is my first attempt at writing an arch page, so any kind of feedback would be great. I have tried to describe the use cases from Lustre's point of view, as opposed to how one would write an arch page if one were concerned about more specific DMU and ZFS implementation issues. The URL is http://arch.lustre.org/index.php?title=DMU_Force_Export (BTW, I don't know how to link this page in the index of arch pages). Thanks, Ricardo -- Ricardo Manuel Correia Lustre Engineering Sun Microsystems, Inc. Portugal Phone +351.214134023 / x58723 Mobile +351.912590825 Email Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From Ricardo.M.Correia at Sun.COM Mon Apr 14 19:14:04 2008 From: Ricardo.M.Correia at Sun.COM (Ricardo M. Correia) Date: Mon, 14 Apr 2008 20:14:04 +0100 Subject: [Lustre-devel] RFC: DMU Force Export arch page Message-ID: <1208200444.16993.27.camel@localhost> Hi Eric, Peter, Andreas, Alex and anyone else interested, Can you review the DMU Force Export architecture page to see if anything is wrong or missing? This is my first attempt at writing an arch page, so any kind of feedback would be great. I have tried to describe the use cases from Lustre's point of view, as opposed to how one would write an arch page if one were concerned about more specific DMU and ZFS implementation issues. The URL is http://arch.lustre.org/index.php?title=DMU_Force_Export (BTW, I don't know how to link this page in the index of arch pages). Thanks, Ricardo -- Ricardo Manuel Correia Lustre Engineering Sun Microsystems, Inc. Portugal Phone +351.214134023 / x58723 Mobile +351.912590825 Email Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From eeb at sun.com Tue Apr 15 00:06:51 2008 From: eeb at sun.com (Eric Barton) Date: Tue, 15 Apr 2008 02:06:51 +0200 Subject: [Lustre-devel] DMU Force Export arch page In-Reply-To: <1208200444.16993.27.camel@localhost> Message-ID: <018501c89e8c$9c16a940$0182a8c0@bartonsoftware.com> Ricardo, Yes, I'll review - but the end of this week is the earliest I can provide feedback... _____ From: Ricardo.M.Correia at Sun.COM [mailto:Ricardo.M.Correia at Sun.COM] Sent: 14 April 2008 9:14 PM To: Eric Barton; Peter J Braam; Andreas Dilger; Alex Zhuravlev Cc: lustre-elk-team at Sun.COM; lustre-devel Subject: RFC: DMU Force Export arch page Hi Eric, Peter, Andreas, Alex and anyone else interested, Can you review the DMU Force Export architecture page to see if anything is wrong or missing? This is my first attempt at writing an arch page, so any kind of feedback would be great. I have tried to describe the use cases from Lustre's point of view, as opposed to how one would write an arch page if one were concerned about more specific DMU and ZFS implementation issues. The URL is http://arch.lustre.org/index.php?title=DMU_Force_Export (BTW, I don't know how to link this page in the index of arch pages). Thanks, Ricardo -- Ricardo Manuel Correia Lustre Engineering Sun Microsystems, Inc. Portugal Phone +351.214134023 / x58723 Mobile +351.912590825 Email Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From Peter.Braam at Sun.COM Wed Apr 16 03:20:41 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Tue, 15 Apr 2008 21:20:41 -0600 Subject: [Lustre-devel] pCIFS file layout questions Message-ID: Hi Matt - I finally had time to read your document about pCIFS and CTDB more carefully and I now understand the problems you are trying to address better. I still want to ask a few questions, to check that my understanding is more or less correct and make some suggestions. Regards, Peter A. The CIFS client recovery is unclear to me. If a Samba node disappears (a) does the client know to try to re-establish the connection? I think this is based on a timeout (b) if a request was sent from the client to the servers, how can the client re-construct the reply or know that the server never executed the request? My claim is that the CIFS protocol is not strong enough to shield the applications from errors and the recovery is an approximate recovery, of the type ³things started to work again². B. On the Samba CTDB nodes, how does the clustering software interact with software that monitors the functioning of the cluster file system? For example, if Samba gets errors doing I/O to Lustre how is a failover initiated? C. Now focus on the ³Lustre clients on the OSS approach² (which customers want ­ they don¹t want extra Lustre clients) My thought is that with pCIFS we in fact do not want to use CTDB in the normal manner on the OSS nodes at all. We do want it for metadata nodes. Assuming that Samba is reasonably fast (we will discover this over the coming weeks) there is one optimal Lustre node to read/write data from, namely the OSS node that holds the data. If that node fails for whatever reason, Lustre/heartbeat will create a new node mountain the target and heartbeat can arrange the IP takeover. So all we need is a Samba server that fails over from the old OSS to the new OSS. Every other solution would cause OSS-to-OSS cross talk. Is this correct? D. Finally another question to verify my understanding. If we take a normal CTDB setup, then many clients can open the SAME file CONCURRENTLY for I/O provided they use a windows share mode that allows this? But in Samba (and probably in CIFS) there is no re-direction protocol that we can use to tell an unmodified client to use different Samba servers to fetch different parts of the file. E. Some architectural thoughts. I believe that if clients read unique pieces of files, the CTDB model without pCIFS is highly sub-optimal. pCIFS which can force clients to do I/O with the right node is much preferable. However, there are some extremely interesting exceptions to the rule. I want to illustrate my thoughts. If each client reads its own file (this is called the ³file per process² I/O model in HPC) CTDB without pCIFS is most unfortunate (with the current Lustre data model, which would recommend to store such a file on one node). The chance that the client has connected to the correct client node is small and almost always we will needlessly pull or push the data from the Samba node to the OSS node that has the data. For an HPC job where all nodes read a single file fully (another very common scenario), the CTDB model D works out great. The Samba nodes all act as a read cache. The OSS to OSS transfer is not so expensive in this case, the overhead is more or less #OSS nodes / #clients, typically 1-5%. But for writing things are completely different. In many (almost all in fact) HPC jobs when files are written they are written they are written as disjoint pieces, so if Lustre was more clever and we used it with CTDB, it could accept data from all writers and write it into the local OSS and simply tell the MDS what the layout of the file should be. This could also be applied to Lustre without CIFS exports: clients that don¹t run on an OSS and write to a file would be told to do all writes through a certain OSS, nicely load balanced over all OSSs. There are many implementations that can lead to the layout management in the previous paragraph. One is to start using a single lock manager for an entire file (not a per-oss lock manager for stripes) and to let the lock manager build the layout based on the extents it is seeing in requests. This is possible provided the cluster has a liveness mechanism. A second implementation is a hierarchical protocol where the OSS negotiates layouts with the MDS as it goes along (and performs re-directs if the I/O must go somewhere else). -------------- next part -------------- An HTML attachment was scrubbed... URL: From Qiang.Wu at Sun.COM Wed Apr 16 13:57:38 2008 From: Qiang.Wu at Sun.COM (Matt Wu) Date: Wed, 16 Apr 2008 21:57:38 +0800 Subject: [Lustre-devel] pCIFS file layout questions In-Reply-To: References: Message-ID: <480605D2.3090902@Sun.COM> Peter, > A. The CIFS client recovery is unclear to me. If a Samba node > disappears (a) does the client know to try to re-establish the > connection? I think this is based on a timeout (b) if a request was sent > from the client to the servers, how can the client re-construct the > reply or know that the server never executed the request? My claim is > that the CIFS protocol is not strong enough to shield the applications > from errors and the recovery is an approximate recovery, of the type > “things started to work again”. The original request will be canceled if it timeouts or connection is broken. Windows CIFS client won't re-send this request. But it will try to reconnect when there comes new requests from user. There are two scenarios: 1, network is broken when windows CIFS client sends request packet in this case the client will try to reconnect and then resend the request package to server. if even the reconnection couldn't work, it just fails the request. 2, network is broken when client thread waits on reply the original request will be canceled with a network error code returned to user. When the request is failed, pCIFS can detect the failure and then retry it once more or try another CIFS server. pCIFS re-send is being done above CIFS protocol and it relies on windows CIFS client driver to reconnect in case the failed request is to be sent to the same server. > B. On the Samba CTDB nodes, how does the clustering software interact > with software that monitors the functioning of the cluster file system? > For example, if Samba gets errors doing I/O to Lustre how is a failover > initiated? > The recovery master node uses a timer to detect all other CTDB nodes. Once there's a node down, the recovery master will issue a recovery process to re-assign the dead node's ip and clients connections to another node. When a Samba process (Samba process acts as a CTDB client) crashes, the CTDB will get acknowledged from the closure of unix-socket, but here it only clean up all the client context since it might be a normal quiting. Then CTDB's monitor process will discover the fact that nothing is servicing on Samba port and then change node status to trigger a recovery process. > C. Now focus on the “Lustre clients on the OSS approach” (which > customers want – they don’t want extra Lustre clients) My thought is > that with pCIFS we in fact do not want to use CTDB in the normal manner > on the OSS nodes at all. We do want it for metadata nodes. Assuming > that Samba is reasonably fast (we will discover this over the coming > weeks) there is one optimal Lustre node to read/write data from, namely > the OSS node that holds the data. If that node fails for whatever > reason, Lustre/heartbeat will create a new node mountain the target and > heartbeat can arrange the IP takeover. So all we need is a Samba server > that fails over from the old OSS to the new OSS. Every other solution > would cause OSS-to-OSS cross talk. Is this correct? > yes. Both pCIFS re-send and CTDB takeover will migrate client's requests to another OSS node. After the stand-by OSS node starts, new requests can be sent to this node and thus to be processed as normally. > D. Finally another question to verify my understanding. If we take a > normal CTDB setup, then many clients can open the SAME file CONCURRENTLY > for I/O provided they use a windows share mode that allows this? But in > Samba (and probably in CIFS) there is no re-direction protocol that we > can use to tell an unmodified client to use different Samba servers to > fetch different parts of the file. We can let Samba ignore the share modes to grant exclusive requests. Let Lustre clients harmonize their concurrent access. This issue is addressed in HLD/ctdb_share_conflict.lyx > E. Some architectural thoughts. > > I believe that if clients read unique pieces of files, the CTDB model > without pCIFS is highly sub-optimal. pCIFS which can force clients to > do I/O with the right node is much preferable. However, there are some > extremely interesting exceptions to the rule. I want to illustrate my > thoughts. > > If each client reads its own file (this is called the “file per process” > I/O model in HPC) CTDB without pCIFS is most unfortunate (with the > current Lustre data model, which would recommend to store such a file on > one node). The chance that the client has connected to the correct > client node is small and almost always we will needlessly pull or push > the data from the Samba node to the OSS node that has the data. > This case we could also redirect metadata operations though these operations will finally be done by the MDS node. But the OSS/client node could cache everything. > For an HPC job where all nodes read a single file fully (another very > common scenario), the CTDB model D works out great. The Samba nodes all > act as a read cache. The OSS to OSS transfer is not so expensive in > this case, the overhead is more or less #OSS nodes / #clients, > typically 1-5%. > > But for writing things are completely different. > > In many (almost all in fact) HPC jobs when files are written they are > written they are written as disjoint pieces, so if Lustre was more > clever and we used it with CTDB, it could accept data from all writers > and write it into the local OSS and simply tell the MDS what the layout > of the file should be. This could also be applied to Lustre without > CIFS exports: clients that don’t run on an OSS and write to a file would > be told to do all writes through a certain OSS, nicely load balanced > over all OSSs. > > There are many implementations that can lead to the layout management in > the previous paragraph. One is to start using a single lock manager for > an entire file (not a per-oss lock manager for stripes) and to let the > lock manager build the layout based on the extents it is seeing in > requests. This is possible provided the cluster has a liveness > mechanism. A second implementation is a hierarchical protocol where the > OSS negotiates layouts with the MDS as it goes along (and performs > re-directs if the I/O must go somewhere else). That's like something enhanced join-file (current pCIFS doesn't support join file). One lock manager can ease file size/extents operations but all these locks are invisible to pCIFS. When writing to file end, we need send a "SET_LENGTH" request to lock manager to alloc the necessary extent on a spare OSS target. Regards, Matt From Peter.Braam at Sun.COM Wed Apr 16 15:37:04 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Wed, 16 Apr 2008 08:37:04 -0700 Subject: [Lustre-devel] Failover & Force export for the DMU Message-ID: ³Force export² for the DMU serves a similar purpose as a feature we added for block devices in Linux in relation to exports. When failover is initiated, the OSS/MDS servers stop sending replies and requests that are still being processed interact with the block devices in a model where the devices discard write commands WITHOUT returning errors. This is different from merely declaring the device READONLY in which case errors are returned. The latter is a default feature in the Linux kernel, what we did is a patch (but could be a mapper module). The thinking behind this approach was (many years ago) that this avoids exposing the server layers to errors (caused by writes to read only devices) from the block devices which might cause the server to panic, thereby taking out other targets inadvertently. However, the approach is flawed. It is (theoretically, but not so likely) possible for the server to write something, believe it has been done, and read it back getting the wrong data (because it wasn¹t written), and still panic. So I would like to suggest that for the DMU we do this differently and rely on a normal read only device. So, the server, during recovery, will be using standard read only devices (and similar under the DMU). If the file system or DMU returns errors because writes cannot be performed for requests that are in progress during the failover event, then these errors should be handled gracefully (without panics). Note that the errors will never reach the client, not over the network and not through reply reconstruction, because failover was initiated before they happened. The hacked feature retains value because it can generate an artificially large amount of rollback data, which is useful for testing the replay recovery mechanisms in Lustre. However, with DMU snapshots this can easily be simulated in a different manner. Nikita, Alex ­ I think the key issue here is that the error handling in the new servers that you have written needs to be resilient enough to handle this. Can you think about it? Ricardo ­ for the DMU all you need to do is make sure you can quickly turn a device read only below the DMU and the DMU can handle that (its like doing ³mount ­o remount, ro²). Regards Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ricardo.M.Correia at Sun.COM Wed Apr 16 16:40:36 2008 From: Ricardo.M.Correia at Sun.COM (Ricardo M. Correia) Date: Wed, 16 Apr 2008 17:40:36 +0100 Subject: [Lustre-devel] Failover & Force export for the DMU In-Reply-To: References: Message-ID: <1208364036.15849.133.camel@localhost> On Qua, 2008-04-16 at 08:37 -0700, Peter Braam wrote: > However, the approach is flawed. It is (theoretically, but not so > likely) possible for the server to write something, believe it has > been done, and read it back getting the wrong data (because it wasn’t > written), and still panic. With the DMU there is a similar problem, but its behavior is more sane and much more interesting. If a write is discarded without the DMU's knowledge, when the data is read back the checksum will necessarily fail due to the ZFS design's cleverness of storing the checksum on the block pointer (in the parent block, which itself has its checksum on its parent block, and so on up until the uberblock). So if the checksum fails 2 things can happen: - If a read is a normal read, the caller will get an ECKSUM error, propagating the error back to the DMU's consumer (this is what is used for all data reads). - If a read if a special "must succeed" read, then the behavior will depend on the "failmode" property of the pool (explained below). A "must succeed" read, like the name indicates, is a critical read which always succeeds (caller is blocked until it does), used in situations where failure would lead to data loss. It is only used for some metadata reads. > So I would like to suggest that for the DMU we do this differently and > rely on a normal read only device. So, the server, during recovery, > will be using standard read only devices (and similar under the DMU). > If the file system or DMU returns errors because writes cannot be > performed for requests that are in progress during the failover event, > then these errors should be handled gracefully (without panics). Note > that the errors will never reach the client, not over the network and > not through reply reconstruction, because failover was initiated > before they happened. I agree, but I'm not so sure we should still continue to send read requests to the storage devices when we are failing over. One of the reasons the failover could be happening is due to a failure somewhere in the server -> storage path, and if this is happening we may experience delays of 30 or 60 seconds for the IOs to timeout, especially if we're doing synchronous I/O in the ZIO threads like we are doing now. So I think returning EIO for reads on the backend storage might be more appropriate during a failover. > Ricardo – for the DMU all you need to do is make sure you can quickly > turn a device read only below the DMU and the DMU can handle that (its > like doing “mount –o remount, ro”). Well, it's a bit more complicated than that.. If there is a fatal failure to write to the backend devices, the error will be returned to the ZIO pipeline and the DMU's behavior will again depend on the "failmode" property of the pool, which can have 3 different values: - wait mode: I/O is blocked until the administrator corrects the problem manually. This is useful for regular ZFS pools, because the administrator has a chance to replace the device that is experiencing IO failures and therefore prevent any data loss. - continue mode: (quoting) "Returns EIO to any new write I/O requests" (in the transaction phase) ".. but allows reads to any of the remaining healthy devices. Any write requests that have yet to be committed to disk would be blocked." - panic mode: in userspace, we do an abort(). This would be a good solution for Lustre if we didn't have multiple ZFS pools in the same userspace server, but it's not useful at all in that case. The big problem here is that neither the "wait" mode nor the "continue" mode allow a pool with dirty data to be exported if the backend devices are returning errors in the pwrite() calls (be it EROFS, EIO, or any other), due to ZFS's insistence on preserving data integrity (which I think is very well designed). I have thought a lot about this, and my conclusion is that when force-exporting a pool we should make the DMU discard all writes to the backend storage, make reads (even "must succeed" reads) return EIO, and then go through the normal DMU export process. I believe this is the only sane way of successfully getting rid of dirty data in the DMU without any loss of transactional integrity or weird failures, but it will also require changing the DMU to gracefully handle failures in "must succeed" reads, which will not be easy.. The consequence for Lustre is that the OSS/MDS servers *must* be able to handle errors gracefully because the DMU could return a lot of EIOs during failover. Cheers, Ricardo -- Ricardo Manuel Correia Lustre Engineering Sun Microsystems, Inc. Portugal Phone +351.214134023 / x58723 Mobile +351.912590825 Email Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From Peter.Braam at Sun.COM Thu Apr 17 00:18:51 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Wed, 16 Apr 2008 17:18:51 -0700 Subject: [Lustre-devel] Failover & Force export for the DMU In-Reply-To: <1208364036.15849.133.camel@localhost> Message-ID: On 4/16/08 9:40 AM, "Ricardo M. Correia" wrote: ... SNIP > I agree, but I'm not so sure we should still continue to send read requests to > the storage devices when we are failing over. One of the reasons the failover > could be happening is due to a failure somewhere in the server -> storage > path, and if this is happening we may experience delays of 30 or 60 seconds > for the IOs to timeout, especially if we're doing synchronous I/O in the ZIO > threads like we are doing now. > > So I think returning EIO for reads on the backend storage might be more > appropriate during a failover. > I think that is fine ­ again, the key issue is not to kill the server while it gets these errors. It may well be that the server needs a special ³I¹m recovering be gentle with errors² mode to avoid reasonable panics. > >> Ricardo ­ for the DMU all you need to do is make sure you can quickly turn a >> device read only below the DMU and the DMU can handle that (its like doing >> ³mount ­o remount, ro²). > > Well, it's a bit more complicated than that.. > If there is a fatal failure to write to the backend devices, the error will be > returned to the ZIO pipeline and the DMU's behavior will again depend on the > "failmode" property of the pool, which can have 3 different values: > > - wait mode: I/O is blocked until the administrator corrects the problem > manually. This is useful for regular ZFS pools, because the administrator has > a chance to replace the device that is experiencing IO failures and therefore > prevent any data loss. > > - continue mode: (quoting) "Returns EIO to any new write I/O requests" (in the > transaction phase) ".. but allows reads to any of the remaining healthy > devices. Any write requests that have yet to be committed to disk would be > blocked." > > - panic mode: in userspace, we do an abort(). This would be a good solution > for Lustre if we didn't have multiple ZFS pools in the same userspace server, > but it's not useful at all in that case. > Well yes, the problem is that controlled failovers are required, for example when you fail back. > > > The big problem here is that neither the "wait" mode nor the "continue" mode > allow a pool with dirty data to be exported if the backend devices are > returning errors in the pwrite() calls (be it EROFS, EIO, or any other), due > to ZFS's insistence on preserving data integrity (which I think is very well > designed). > Please explain why we want to export such a pool and on which node we want to export it, in fact what is ³export² (it should be similar to unmount)? If things are failing, then, on the node that is failing, we don¹t need this pool anymore, we need to shut things down, in most cases for a reboot. We need the pool on the failover node. In fact there is a very useful distinction to make. There are two failover scenarios: 1. fail over to move services away from failures on the OSS. In this case a reboot/panic is not really harmful. 2. fail over from a fully functioning OSS/DMU to redistribute services. In this case we need a control mechanism to turn the device read-only and clean up the DMU. Unfortunately we cannot consider mandating that there is only one file system per OSS because then we need an idle node to act as the failover node. We must handle the problem of shutting ³one of more² down, but only in the clean case (2). > > I have thought a lot about this, and my conclusion is that when > force-exporting a pool we should make the DMU discard all writes to the > backend storage, make reads (even "must succeed" reads) return EIO, and then > go through the normal DMU export process. I believe this is the only sane way > of successfully getting rid of dirty data in the DMU without any loss of > transactional integrity or weird failures, but it will also require changing > the DMU to gracefully handle failures in "must succeed" reads, which will not > be easy.. > Sun already has products (a CIFS server) that can failover on ZFS. It might be interesting to ask them if they can handle failing over one ZFS file system while keeping others, because this is essentially the same problem as we have from a DMU perspective. Peter > > The consequence for Lustre is that the OSS/MDS servers *must* be able to > handle errors gracefully because the DMU could return a lot of EIOs during > failover. > > Cheers, > Ricardo > -- > Ricardo Manuel Correia > Lustre Engineering > > Sun Microsystems, Inc. > Portugal > Phone +351.214134023 / x58723 > Mobile +351.912590825 > Email Ricardo.M.Correia at Sun.COM > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From Ricardo.M.Correia at Sun.COM Thu Apr 17 16:10:31 2008 From: Ricardo.M.Correia at Sun.COM (Ricardo M. Correia) Date: Thu, 17 Apr 2008 17:10:31 +0100 Subject: [Lustre-devel] Failover & Force export for the DMU In-Reply-To: References: Message-ID: <1208448631.6677.82.camel@localhost> Hi Peter, Please see my comments. On Qua, 2008-04-16 at 17:18 -0700, Peter Braam wrote: > I think that is fine – again, the key issue is not to kill the server > while it gets these errors. It may well be that the server needs a > special “I’m recovering be gentle with errors” mode to avoid > reasonable panics. I would say any error returned by the filesystem even in normal operation should be handled gently :) > Please explain why we want to export such a pool and on which node we > want to export it, in fact what is “export” (it should be similar to > unmount)? If things are failing, then, on the node that is failing, > we don’t need this pool anymore, we need to shut things down, in most > cases for a reboot. We need the pool on the failover node. The DMU has the notion of importing and exporting a pool, which is different from mounting/unmounting a filesystem inside the pool. Basically, an import consists in scanning and reading the labels of all the devices of a pool to find out the pool configuration. After this process, the pool transitions to the imported state, which means that the DMU knows about the pool (has the pool configuration cached) and the user can perform any operation he desires on the pool. Usually after an import ZFS also mounts the filesystems inside the pool automatically, but this is not relevant here. In ZFS, an export consists of unmounting any filesystem belonging to the pool, flushing dirty data, marking the pool as exported on-disk and then removing the pool configuration from the cache. In Lustre/ZFS, strictly speaking there are no filesystems mounted so we don't do that, but of course the export would fail if Lustre has an open objset, so we need to close them first. After this, the user can only operate/manipulate the pool if he re-imports it. So basically, what we need to do when things are failing (in the node that is failing) is to close the filesystems and export the pool. The big problem is that the DMU cannot export a pool if the devices are experiencing fatal write failures, which is why we need a force-export mechanism. After that, we need to import the pool on the failover node and mount all the MDTs/OSTs that were stored there, do recovery, etc (I'm sure you understand this process much better than I do :) > In fact there is a very useful distinction to make. There are two > failover scenarios: > 1. fail over to move services away from failures on the OSS. In > this case a reboot/panic is not really harmful. That's why when I heard about the need for this feature, I immediately proposed doing a panic, which wouldn't have any consequences assuming Lustre recovery does its job. But it's not useful in a "multiple pools in the same server" scenario. > 1. fail over from a fully functioning OSS/DMU to redistribute > services. In this case we need a control mechanism to turn > the device read-only and clean up the DMU. Why do we need to turn the device read-only in this case? Why can't we do a clean unmount/export if the devices are fully functioning? Andreas has told me before that with ldiskfs, doing a clean unmount could take a lot of time if there's a lot of dirty data, but I don't believe this will be true with the DMU. Even if such a problem were to arise, in the DMU it's trivial to limit the transaction group size and therefore limit the time it takes to sync a txg. > Unfortunately we cannot consider mandating that there is only one file > system per OSS because then we need an idle node to act as the > failover node. We must handle the problem of shutting “one of more” > down, but only in the clean case (2). In the clean case, we don't need force-export. Force-export is only really needed if all of the following conditions are true: 1) We have more than 1 filesystem (MDT/OST) running in the same userspace process (note how I didn't say "same server". Also note that for Lustre 2.0, we will have a limitation of 1 userspace process per server). 2) The MDTs/OSTs are stored in more than 1 ZFS pool (note how I didn't say "more than 1 device". A single ZFS pool can use multiple disk devices.). 3) One or more, but not all of the ZFS pools are suffering from fatal IO failures. 4) We only want to failover the MDTs/OSTs stored on the pools that are suffering IO failures, but we still want to keep the remaining MDTs/OSTs working in the same server. If there is a requirement of supporting a scenario where all of these conditions are true, then we need force-export. From my latest discussion with Andreas about this, we do need that. If not all of the conditions are true, we could either do a clean export or do a panic, depending on the situation. At least, that is my understanding :) Thanks, Ricardo -- Ricardo Manuel Correia Lustre Engineering Sun Microsystems, Inc. Portugal Phone +351.214134023 / x58723 Mobile +351.912590825 Email Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From Peter.Braam at Sun.COM Thu Apr 17 17:53:17 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Thu, 17 Apr 2008 10:53:17 -0700 Subject: [Lustre-devel] Failover & Force export for the DMU In-Reply-To: <1208448631.6677.82.camel@localhost> Message-ID: On 4/17/08 9:10 AM, "Ricardo M. Correia" wrote: > >> In fact there is a very useful distinction to make. There are two failover >> scenarios: >> 1. fail over to move services away from failures on the OSS. In this case a >> reboot/panic is not really harmful. > > That's why when I heard about the need for this feature, I immediately > proposed doing a panic, which wouldn't have any consequences assuming Lustre > recovery does its job. But it's not useful in a "multiple pools in the same > server" scenario. > I don¹t think this is valid reasoning. If one pool is hosed, it is just as well to reboot the node. At best what you are proposing is a ³nice to have refinement² but not necessary for proper management of Lustre clusters. Following my proposal seems to eliminate the requirement for very complicated work. > >> >> 1. fail over from a fully functioning OSS/DMU to redistribute services. In >> this case we need a control mechanism to turn the device read-only and clean >> up the DMU. > > Why do we need to turn the device read-only in this case? Why can't we do a > clean unmount/export if the devices are fully functioning? > Andreas has told me before that with ldiskfs, doing a clean unmount could take > a lot of time if there's a lot of dirty data, but I don't believe this will be > true with the DMU. > Even if such a problem were to arise, in the DMU it's trivial to limit the > transaction group size and therefore limit the time it takes to sync a txg. > >> Unfortunately we cannot consider mandating that there is only one file >> system per OSS because then we need an idle node to act as the failover node. >> We must handle the problem of shutting ³one of more² down, but only in the >> clean case (2). > > In the clean case, we don't need force-export. > > Force-export is only really needed if all of the following conditions are > true: > > 1) We have more than 1 filesystem (MDT/OST) running in the same userspace > process (note how I didn't say "same server". Also note that for Lustre 2.0, > we will have a limitation of 1 userspace process per server). > > 2) The MDTs/OSTs are stored in more than 1 ZFS pool (note how I didn't say > "more than 1 device". A single ZFS pool can use multiple disk devices.). > > 3) One or more, but not all of the ZFS pools are suffering from fatal IO > failures. > > 4) We only want to failover the MDTs/OSTs stored on the pools that are > suffering IO failures, but we still want to keep the remaining MDTs/OSTs > working in the same server. > Yes. But this is not a requirement, because for example 4) is not necessary for customer happiness. > > If there is a requirement of supporting a scenario where all of these > conditions are true, then we need force-export. From my latest discussion with > Andreas about this, we do need that. > No we do not. Andreas, please get in touch with me. I think this is a ³nice to have² but not important enough. -Peter - > > If not all of the conditions are true, we could either do a clean export or do > a panic, depending on the situation. > > At least, that is my understanding :) > > Thanks, > Ricardo > > -- > Ricardo Manuel Correia > Lustre Engineering > > Sun Microsystems, Inc. > Portugal > Phone +351.214134023 / x58723 > Mobile +351.912590825 > Email Ricardo.M.Correia at Sun.COM > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From Peter.Braam at Sun.COM Thu Apr 17 17:56:48 2008 From: Peter.Braam at Sun.COM (Peter Braam) Date: Thu, 17 Apr 2008 10:56:48 -0700 Subject: [Lustre-devel] Failover & Force export for the DMU In-Reply-To: <1208448631.6677.82.camel@localhost> Message-ID: I forgot one other comment/question: shutdown of Lustre servers was traditionally sometimes very slow because of timeouts ­ however with the Sandia ³kill the export features² is this still true? - peter - On 4/17/08 9:10 AM, "Ricardo M. Correia" wrote: > Hi Peter, > > Please see my comments. > > On Qua, 2008-04-16 at 17:18 -0700, Peter Braam wrote: >> I think that is fine ­ again, the key issue is not to kill the server while >> it gets these errors. It may well be that the server needs a special ³I¹m >> recovering be gentle with errors² mode to avoid reasonable panics. > > I would say any error returned by the filesystem even in normal operation > should be handled gently :) > >> Please explain why we want to export such a pool and on which node we want >> to export it, in fact what is ³export² (it should be similar to unmount)? If >> things are failing, then, on the node that is failing, we don¹t need this >> pool anymore, we need to shut things down, in most cases for a reboot. We >> need the pool on the failover node. > > The DMU has the notion of importing and exporting a pool, which is different > from mounting/unmounting a filesystem inside the pool. > > Basically, an import consists in scanning and reading the labels of all the > devices of a pool to find out the pool configuration. > After this process, the pool transitions to the imported state, which means > that the DMU knows about the pool (has the pool configuration cached) and the > user can perform any operation he desires on the pool. > > Usually after an import ZFS also mounts the filesystems inside the pool > automatically, but this is not relevant here. > > In ZFS, an export consists of unmounting any filesystem belonging to the pool, > flushing dirty data, marking the pool as exported on-disk and then removing > the pool configuration from the cache. > In Lustre/ZFS, strictly speaking there are no filesystems mounted so we don't > do that, but of course the export would fail if Lustre has an open objset, so > we need to close them first. > After this, the user can only operate/manipulate the pool if he re-imports it. > > So basically, what we need to do when things are failing (in the node that is > failing) is to close the filesystems and export the pool. The big problem is > that the DMU cannot export a pool if the devices are experiencing fatal write > failures, which is why we need a force-export mechanism. > > After that, we need to import the pool on the failover node and mount all the > MDTs/OSTs that were stored there, do recovery, etc (I'm sure you understand > this process much better than I do :) > > >> In fact there is a very useful distinction to make. There are two failover >> scenarios: >> 1. fail over to move services away from failures on the OSS. In this case a >> reboot/panic is not really harmful. > > That's why when I heard about the need for this feature, I immediately > proposed doing a panic, which wouldn't have any consequences assuming Lustre > recovery does its job. But it's not useful in a "multiple pools in the same > server" scenario. > >> >> 1. fail over from a fully functioning OSS/DMU to redistribute services. In >> this case we need a control mechanism to turn the device read-only and clean >> up the DMU. > > Why do we need to turn the device read-only in this case? Why can't we do a > clean unmount/export if the devices are fully functioning? > Andreas has told me before that with ldiskfs, doing a clean unmount could take > a lot of time if there's a lot of dirty data, but I don't believe this will be > true with the DMU. > Even if such a problem were to arise, in the DMU it's trivial to limit the > transaction group size and therefore limit the time it takes to sync a txg. > >> Unfortunately we cannot consider mandating that there is only one file >> system per OSS because then we need an idle node to act as the failover node. >> We must handle the problem of shutting ³one of more² down, but only in the >> clean case (2). > > In the clean case, we don't need force-export. > > Force-export is only really needed if all of the following conditions are > true: > > 1) We have more than 1 filesystem (MDT/OST) running in the same userspace > process (note how I didn't say "same server". Also note that for Lustre 2.0, > we will have a limitation of 1 userspace process per server). > > 2) The MDTs/OSTs are stored in more than 1 ZFS pool (note how I didn't say > "more than 1 device". A single ZFS pool can use multiple disk devices.). > > 3) One or more, but not all of the ZFS pools are suffering from fatal IO > failures. > > 4) We only want to failover the MDTs/OSTs stored on the pools that are > suffering IO failures, but we still want to keep the remaining MDTs/OSTs > working in the same server. > > If there is a requirement of supporting a scenario where all of these > conditions are true, then we need force-export. From my latest discussion with > Andreas about this, we do need that. > If not all of the conditions are true, we could either do a clean export or do > a panic, depending on the situation. > > At least, that is my understanding :) > > Thanks, > Ricardo > > -- > Ricardo Manuel Correia > Lustre Engineering > > Sun Microsystems, Inc. > Portugal > Phone +351.214134023 / x58723 > Mobile +351.912590825 > Email Ricardo.M.Correia at Sun.COM > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From adilger at sun.com Sat Apr 19 00:22:17 2008 From: adilger at sun.com (Andreas Dilger) Date: Fri, 18 Apr 2008 18:22:17 -0600 Subject: [Lustre-devel] New Pools DLD In-Reply-To: <18373.20125.724151.421930@gargle.gargle.HOWL> References: <20080225100425.GC3108@webber.adilger.int> <18373.20125.724151.421930@gargle.gargle.HOWL> Message-ID: <20080419002217.GC2775@webber.adilger.int> On Feb 27, 2008 14:50 +0300, Nikita Danilov wrote: > Andreas Dilger writes: > > are you aware of any desirable LOV EA changes that would be good to > > include with the changes for the v3 "pool" EA (attached)? Are there > > any changes that are desirable for e.g. FIDs or similar? > > I think that if we are introducing incompatible LOV EA format, we can as > well go forward with changes hinted to at > > http://arch.lustre.org/index.php?title=MDS_striping_format#Future_developments Nikita, sorry to take a long time to get back to this issue, but I think it is quite valuable to pursue if we are already going to change the on-disk format. Since we are already need to change the LOV_MAGIC value, we may as well do as you suggest and have 0x0BD3ssss instead of 0x0BD30BD0, where ssss = size of EA in bytes. That would limit us to a 65536-byte striping EA, but it still larger than what is supported today, and the plans for wide striping also do not call for larger EAs. Even supporting 64kB EAs would be an issue with the current nifrastructure because the client always has to preallocate a receive buffer large enough for the largest EA because it does not know the EA size in advance. The question is whether you think we should also add a magic + size to the lov_ost_data_v1 structure, which is currently the same for all EA types. Adding a per-stripe magic + size would reduce the number of stripes we can allocate per file, and the 160-stripe limit is already a problem for some systems with more than 160 OSTs. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Nikita.Danilov at Sun.COM Mon Apr 21 11:08:18 2008 From: Nikita.Danilov at Sun.COM (Nikita Danilov) Date: Mon, 21 Apr 2008 15:08:18 +0400 Subject: [Lustre-devel] New Pools DLD In-Reply-To: <20080419002217.GC2775@webber.adilger.int> References: <20080225100425.GC3108@webber.adilger.int> <18373.20125.724151.421930@gargle.gargle.HOWL> <20080419002217.GC2775@webber.adilger.int> Message-ID: <18444.30114.732662.996066@gargle.gargle.HOWL> Andreas Dilger writes: > On Feb 27, 2008 14:50 +0300, Nikita Danilov wrote: > > Andreas Dilger writes: > > > are you aware of any desirable LOV EA changes that would be good to > > > include with the changes for the v3 "pool" EA (attached)? Are there > > > any changes that are desirable for e.g. FIDs or similar? > > > > I think that if we are introducing incompatible LOV EA format, we can as > > well go forward with changes hinted to at > > > > http://arch.lustre.org/index.php?title=MDS_striping_format#Future_developments > > Nikita, sorry to take a long time to get back to this issue, but I think > it is quite valuable to pursue if we are already going to change the > on-disk format. > > Since we are already need to change the LOV_MAGIC value, we may as well > do as you suggest and have 0x0BD3ssss instead of 0x0BD30BD0, where > ssss = size of EA in bytes. > > That would limit us to a 65536-byte striping EA, but it still larger than If that happens to be a limiting factor, we can interpret ssss as a number of __u32's or __u64's in EA. > what is supported today, and the plans for wide striping also do not call > for larger EAs. Even supporting 64kB EAs would be an issue with the > current nifrastructure because the client always has to preallocate a > receive buffer large enough for the largest EA because it does not know > the EA size in advance. > > The question is whether you think we should also add a magic + size to the > lov_ost_data_v1 structure, which is currently the same for all EA types. > Adding a per-stripe magic + size would reduce the number of stripes we > can allocate per file, and the 160-stripe limit is already a problem for > some systems with more than 160 OSTs. I think we need an ability to have fully general files layout, so that, for example, a stripe can in turn be a striped file. This can be something like struct lov_mds_md_v3 { __u32 lmm_magic; /* 0x0BD3ssss */ __u32 lmm_pattern; /* LOV_PATTERN_RAID0, LOV_PATTERN_RAID1 */ __u64 lmm_object_id; /* LOV object ID */ __u64 lmm_object_gr; /* LOV object group */ __u32 lmm_stripe_size; /* size of stripe in bytes */ __u32 lmm_stripe_count; /* num stripes in use for this object */ }; followed by a sequence of stripe layout descriptors each starting with __u32 magic; /* 0xLLLLSSSS. where LLLL is an identifier of a * layout type (e.g., 0bd3 is raid0 or raid1), and * SSSS is a size. */ But for a common case of a striped file where all stripes have the same layout, we implement a short-cut: struct lov_mds_md_v4 { __u32 lmm_magic; /* 0x0BD4ssss */ __u32 lmm_pattern; /* LOV_PATTERN_RAID0, LOV_PATTERN_RAID1 */ __u64 lmm_object_id; /* LOV object ID */ __u64 lmm_object_gr; /* LOV object group */ __u32 lmm_stripe_size; /* size of stripe in bytes */ __u32 lmm_stripe_count; /* num stripes in use for this object */ __u32 lmm_stripe_magic; /* 0xLLLLSSSS for all stripes */ }; followed by an array of stripe layout descriptor, stripped of their magics. Or we can go one step further and assume particular value of lmm_stripe_magic for a particular lmm_magic. In this case, ->lmm_stripe_magic field can be removed. > > Cheers, Andreas > -- Nikita. From eeb at sun.com Tue Apr 22 13:06:10 2008 From: eeb at sun.com (Eric Barton) Date: Tue, 22 Apr 2008 14:06:10 +0100 Subject: [Lustre-devel] New Pools DLD In-Reply-To: <20080419002217.GC2775@webber.adilger.int> References: <20080225100425.GC3108@webber.adilger.int> <18373.20125.724151.421930@gargle.gargle.HOWL> <20080419002217.GC2775@webber.adilger.int> Message-ID: <019701c8a479$a2e42d40$0281a8c0@ebpc> Comment in-line below... > -----Original Message----- > From: lustre-devel-bounces at lists.lustre.org > [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of > Andreas Dilger > Sent: 19 April 2008 1:22 AM > To: Lustre Development Mailing List > Cc: Mike Pershin; Nikita Danilov; Yuriy Umanets > Subject: Re: [Lustre-devel] New Pools DLD > > On Feb 27, 2008 14:50 +0300, Nikita Danilov wrote: > > Andreas Dilger writes: > > > are you aware of any desirable LOV EA changes that would be good > > > to include with the changes for the v3 "pool" EA (attached)? > > > Are there any changes that are desirable for e.g. FIDs or > > > similar? > > > > I think that if we are introducing incompatible LOV EA format, we > > can as well go forward with changes hinted to at > > > > > http://arch.lustre.org/index.php?title=MDS_striping_format#Future_developments > > Nikita, sorry to take a long time to get back to this issue, but I > think it is quite valuable to pursue if we are already going to > change the on-disk format. > > Since we are already need to change the LOV_MAGIC value, we may as > well do as you suggest and have 0x0BD3ssss instead of 0x0BD30BD0, > where ssss = size of EA in bytes. That seems a hack - why overload magic like this rather than have a separate field? > That would limit us to a 65536-byte striping EA, but it still larger > than what is supported today, and the plans for wide striping also > do not call for larger EAs. Even supporting 64kB EAs would be an > issue with the current nifrastructure because the client always has > to preallocate a receive buffer large enough for the largest EA > because it does not know the EA size in advance. > > The question is whether you think we should also add a magic + size > to the lov_ost_data_v1 structure, which is currently the same for > all EA types. Adding a per-stripe magic + size would reduce the > number of stripes we can allocate per file, and the 160-stripe limit > is already a problem for some systems with more than 160 OSTs. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Nikita.Danilov at Sun.COM Tue Apr 22 17:03:59 2008 From: Nikita.Danilov at Sun.COM (Nikita Danilov) Date: Tue, 22 Apr 2008 21:03:59 +0400 Subject: [Lustre-devel] New Pools DLD In-Reply-To: <019701c8a479$a2e42d40$0281a8c0@ebpc> References: <20080225100425.GC3108@webber.adilger.int> <18373.20125.724151.421930@gargle.gargle.HOWL> <20080419002217.GC2775@webber.adilger.int> <019701c8a479$a2e42d40$0281a8c0@ebpc> Message-ID: <18446.6783.368819.504719@gargle.gargle.HOWL> Eric Barton writes: > Comment in-line below... > > > -----Original Message----- > > From: lustre-devel-bounces at lists.lustre.org > > [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of > > Andreas Dilger [...] > > Since we are already need to change the LOV_MAGIC value, we may as > > well do as you suggest and have 0x0BD3ssss instead of 0x0BD30BD0, > > where ssss = size of EA in bytes. > > That seems a hack - why overload magic like this rather than have a > separate field? Historically space in EA's was very scarce resource. Also, even a slight increase in EA size might result in significant performance changes, e.g., when EA's stop fitting into inode, and additional seek+read is added to every inode load. So I tend to agree with Andreas that it makes sense to replace __u32 magic with __u16 layout_type_id; __u16 layout_descriptor_size; Nikita. From Ricardo.M.Correia at Sun.COM Wed Apr 23 21:15:35 2008 From: Ricardo.M.Correia at Sun.COM (Ricardo M. Correia) Date: Wed, 23 Apr 2008 22:15:35 +0100 Subject: [Lustre-devel] Lustre and zfs on solaris In-Reply-To: References: <6C2A6456-4B2C-412C-A6C2-7B2066D2C438@sun.com> Message-ID: <1208985335.7065.65.camel@localhost> Hi James,  > > > On Apr 18, 2008, at 6:42 PM, James Hartley wrote: > > > > > > > I have a REAL need to get involved with this. Do you have a > > > > beta test > > > > program or can I join the development of Lustre and ZFS on > > > > solaris. I > > > > am willing to do what is necessary to come up to speed and > > > > either work > > > > directly or test this product. (snip) > > > > If there is a current beta in the works please inform me how I > > > > can get > > > > a copy and help. I don't know what is your background with Lustre, but here are a few links to get you started: * Lustre operations manual: http://manual.lustre.org/index.php?title=Main_Page * Lustre build instructions: http://wiki.lustre.org/index.php?title=BuildLustre * Access instructions for the CVS repository: http://wiki.lustre.org/index.php?title=Open_CVS * How to build/setup the userspace OSS (this is what you want the most): http://wiki.lustre.org/index.php?title=Lustre_OSS/MDS_with_ZFS_DMU Note that currently only the userspace OSS can be used in Solaris (and with ZFS). Also note that when using the uOSS, you have to use an MDS compiled from the HEAD branch. Please let me know if you have any questions. Cheers, Ricardo -- Ricardo Manuel Correia Lustre Engineering Sun Microsystems, Inc. Portugal Phone +351.214134023 / x58723 Mobile +351.912590825 Email Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From mihai.stanescu at gmail.com Thu Apr 24 12:30:46 2008 From: mihai.stanescu at gmail.com (Mihai"Stanescu) Date: Thu, 24 Apr 2008 15:30:46 +0300 Subject: [Lustre-devel] Interested in cluster membership services Message-ID: Hello I am a developer working on a project involving a cluster and i researched the development of cluster services (node discovery, group membership, group mantainance, election, consensul) but i still have many roads i can take with no experience feedback If someone of you could give me some practical hints, have a little chat would be great. Thanks Mihai From adilger at sun.com Thu Apr 24 17:13:57 2008 From: adilger at sun.com (Andreas Dilger) Date: Thu, 24 Apr 2008 11:13:57 -0600 Subject: [Lustre-devel] Interested in cluster membership services In-Reply-To: References: Message-ID: <20080424171357.GQ3095@webber.adilger.int> On Apr 24, 2008 15:30 +0300, MihaiStanescu wrote: > I am a developer working on a project involving a cluster and i > researched the development of cluster services (node discovery, group > membership, group mantainance, election, consensul) but i still have > many roads i can take with no experience feedback > > If someone of you could give me some practical hints, have a little > chat would be great. Lustre doesn't use any of these mechanisms (group membership) so you won't get much help here. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From lustre at ragnark.vestdata.no Fri Apr 25 08:48:04 2008 From: lustre at ragnark.vestdata.no (Ragnar =?iso-8859-15?Q?Kj=F8rstad?=) Date: Fri, 25 Apr 2008 10:48:04 +0200 Subject: [Lustre-devel] Interested in cluster membership services In-Reply-To: References: Message-ID: <20080425084803.GW26061@vestdata.no> On Thu, Apr 24, 2008 at 03:30:46PM +0300, MihaiStanescu wrote: > Hello > > I am a developer working on a project involving a cluster and i > researched the development of cluster services (node discovery, group > membership, group mantainance, election, consensul) but i still have > many roads i can take with no experience feedback > > If someone of you could give me some practical hints, have a little > chat would be great. I would suggest that you start by checking out the Open Cluster Framework (http://opencf.org/) and Service Availability Forum (http://www.saforum.org/) specifications for such cluster services. The opencf mailinglist would be a good place for further questions or discussion. -- Ragnar Kjørstad Software Engineer Platform Computing From adilger at sun.com Sun Apr 27 05:13:05 2008 From: adilger at sun.com (Andreas Dilger) Date: Sat, 26 Apr 2008 23:13:05 -0600 Subject: [Lustre-devel] New Pools DLD In-Reply-To: <18444.30114.732662.996066@gargle.gargle.HOWL> References: <20080225100425.GC3108@webber.adilger.int> <18373.20125.724151.421930@gargle.gargle.HOWL> <20080419002217.GC2775@webber.adilger.int> <18444.30114.732662.996066@gargle.gargle.HOWL> Message-ID: <20080427051305.GH3047@webber.adilger.int> On Apr 21, 2008 15:08 +0400, Nikita Danilov wrote: > Andreas Dilger writes: > > Since we are already need to change the LOV_MAGIC value, we may as well > > do as you suggest and have 0x0BD3ssss instead of 0x0BD30BD0, where > > ssss = size of EA in bytes. > > > > That would limit us to a 65536-byte striping EA, but it still larger than > > If that happens to be a limiting factor, we can interpret ssss as a > number of __u32's or __u64's in EA. The EA size being a multiple of at least __u32 is certainly true, and the current EAs are also a multiple of __u64 so I don't think this is a bad idea at all. This gives us an extra 8x larger EAs (512kB). > I think we need an ability to have fully general files layout, so that, > for example, a stripe can in turn be a striped file. This can be > something like > > struct lov_mds_md_v3 { > __u32 lmm_magic; /* 0x0BD3ssss */ > __u32 lmm_pattern; /* LOV_PATTERN_RAID0, LOV_PATTERN_RAID1 */ > __u64 lmm_object_id; /* LOV object ID */ > __u64 lmm_object_gr; /* LOV object group */ > __u32 lmm_stripe_size; /* size of stripe in bytes */ > __u32 lmm_stripe_count; /* num stripes in use for this object */ > }; > > followed by a sequence of stripe layout descriptors each starting with > > __u32 magic; /* 0xLLLLSSSS. where LLLL is an identifier of a > * layout type (e.g., 0bd3 is raid0 or raid1), and > * SSSS is a size. */ > > But for a common case of a striped file where all stripes have the same > layout, we implement a short-cut: > > struct lov_mds_md_v4 { > __u32 lmm_magic; /* 0x0BD4ssss */ > __u32 lmm_pattern; /* LOV_PATTERN_RAID0, LOV_PATTERN_RAID1 */ > __u64 lmm_object_id; /* LOV object ID */ > __u64 lmm_object_gr; /* LOV object group */ > __u32 lmm_stripe_size; /* size of stripe in bytes */ > __u32 lmm_stripe_count; /* num stripes in use for this object */ > __u32 lmm_stripe_magic; /* 0xLLLLSSSS for all stripes */ > }; > > followed by an array of stripe layout descriptor, stripped of their > magics. > > Or we can go one step further and assume particular value of > lmm_stripe_magic for a particular lmm_magic. In this case, > ->lmm_stripe_magic field can be removed. This is essentially the mechanism we use today, which is OK for the very common case of a single record format. I do like the idea of being able to have a heirarchical LOV EA format, and have been thinking about that for a long time. The other thing that I noticed in the pNFS layout is the ability to have "joined" files in a very simple manner. The "header" part of the layout (equivalent to our lov_mds_md) also contains the length in bytes of that part of the file, and then allows mutliple EAs to be concatenated together. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.