From gshipman at ornl.gov Tue Apr 5 20:57:23 2011 From: gshipman at ornl.gov (Shipman, Galen M.) Date: Tue, 05 Apr 2011 16:57:23 -0400 Subject: [Lustre-devel] Lustre User Group 2011 - Registration ends this Friday (4/8/2011) Message-ID: <7A2F3E97-9555-471C-B6D6-C4E15DE13B98@ornl.gov> Registration for the Lustre User Group (LUG) 2011 ends this Friday (4/8/2011) at noon EDT. We very limited space remaining (10 slots). If you plan on attending please register today! LUG 2011 will be held in Orlando, Florida from 8:30 AM Tuesday, April 12, 2011 through 12:00 noon April 14, 2011 at the Marriott World Center Golf and Spa resort. This two-and-a-half-day event is the primary venue for discussion and seminars on open source parallel file system technologies with a unique focus on the Lustre parallel file system. The conference is generously supported by the following corporate sponsors: Bull, DataDirect Networks, Dell, HP, LSI, Oracle, SGI, Terascala, Whamcloud, and Xyratex. The LUG program committee has put together a great agenda for this event, featuring presentations on Lustre features, upcoming enhancements, site-specific experiences, lessons learned, community organizations, and much more. You can find the entire LUG agenda via the LUG website at http://www.olcf.ornl.gov/event/lug-2011/ (click on the agenda tab) REGISTER TODAY You can register for LUG 2011 via the LUG website at http://www.olcf.ornl.gov/event/lug-2011/ (click on the registration tab) Standard registration is $550 per person for the entire two-and-a-half-day event. (Apologies if you are receiving this message more than once, it has been sent to multiple lists) From aurelien.degremont at cea.fr Wed Apr 6 16:58:47 2011 From: aurelien.degremont at cea.fr (DEGREMONT Aurelien) Date: Wed, 06 Apr 2011 18:58:47 +0200 Subject: [Lustre-devel] llog API, locking and concurrency Message-ID: <4D9C9BC7.5050603@cea.fr> Hello I'm wondering what is the correct way to use llog_ API regarding locking and concurrencies? When looking at code, it seems llog already deal with proper locking when several thread access the same llog. The codes that use this king of log do not seem to add more locking when using it. Is this correct? llog_* call can be called without taking care of concurrency? Thanks Aurélien From bzzz at whamcloud.com Wed Apr 6 17:33:16 2011 From: bzzz at whamcloud.com (Alex Zhuravlev) Date: Wed, 6 Apr 2011 21:33:16 +0400 Subject: [Lustre-devel] llog API, locking and concurrency In-Reply-To: <4D9C9BC7.5050603@cea.fr> References: <4D9C9BC7.5050603@cea.fr> Message-ID: correct, llog is doing serialization internally. thanks, Alex On Wed, Apr 6, 2011 at 8:58 PM, DEGREMONT Aurelien < aurelien.degremont at cea.fr> wrote: > Hello > > I'm wondering what is the correct way to use llog_ API regarding locking > and concurrencies? > > When looking at code, it seems llog already deal with proper locking > when several thread access the same llog. > The codes that use this king of log do not seem to add more locking when > using it. > > Is this correct? llog_* call can be called without taking care of > concurrency? > > > Thanks > > Aurélien > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Norman.Morse at OpenSFS.org Fri Apr 8 21:58:16 2011 From: Norman.Morse at OpenSFS.org (Norman Morse) Date: Fri, 8 Apr 2011 16:58:16 -0500 Subject: [Lustre-devel] Open Scalable File Systems (OpenSFS) Request for Proposals (RFP) Message-ID: Dear Lustre Community, OpenSFS is pleased to announce a request for proposals for “OpenSFS Phase 1 Lustre development” (Solicitation Number CX589). The purpose of this RFP is to meet the highest priority Lustre requirements gathered from the community by the OpenSFS Technical Working Group with participation from OpenSFS participants and non-participants alike. The results of the requirements gathering process are available in the OpenSFS TWG Lustre Requirements document TWG Requirements 2011-03-22< http://www.opensfs.org/wp-content/uploads/2011/03/OpenSFSTWGRequirements_2011-03-22.pdf >. Interested offerors are advised to review the RFP documentation available at the OpenSFS website RFP Components: - Proposal Preparation Instructions - Metadata Performance and Scalability Technical Specification - Space Quota Accounting and Enforcement Technical Specification - Evaluation Criteria Interested offerors are encouraged to indicate their intent to submit a proposal no later than April 15th, 2011 via email to proposals at lists.opensfs.org. Full proposals are due April 22nd, 2011 at 11:59 PM EDT as detailed in the Proposal Preparation Instructions. Interested offerors must submit all communication (questions, comments, etc.) to proposals at lists.opensfs.org. While the RFP is in play, the OpenSFS technical community will not discuss the RFP except through proposals at lists.opensfs.org. Regular business activities that do not pertain to this RFP will be conducted in the normal manner. All trademarks are the property of their respective owners. With best regards, Norman Morse, President Open Scalable File Systems -------------- next part -------------- An HTML attachment was scrubbed... URL: From yidingz at yidingz.com Tue Apr 12 04:41:57 2011 From: yidingz at yidingz.com (YIDING ZHOU) Date: Tue, 12 Apr 2011 12:41:57 +0800 Subject: [Lustre-devel] Lustre Windows Native Client Message-ID: Hi there, I'm looking at a Lustre client for Windows. I find a reference here at http://wiki.lustre.org/index.php/Windows_Native_Client. However, I can't find how to participate the "early access program" anywhere. Do you know where I can find binary/source code for Lustre Windows Native Client? I heard that this project is discontinued so I hope we can take this on and continue working on it. Thanks Yiding -------------- next part -------------- An HTML attachment was scrubbed... URL: From eeb at whamcloud.com Mon Apr 18 17:04:40 2011 From: eeb at whamcloud.com (Eric Barton) Date: Mon, 18 Apr 2011 18:04:40 +0100 Subject: [Lustre-devel] Lustre Windows Native Client In-Reply-To: References: Message-ID: <00c901cbfdea$b51778a0$1f4669e0$@com> Yiding, Windows Native Client is closed-source. It's owned by Oracle and they have all the rights over it. If all you are interested in is accessing Lustre from Windows, your best bet will be using Samba to export Lustre over CIFS. Cheers, Eric Eric Barton CTO Whamcloud, Inc Tel: +44 117 330 1575 Mob: +44 7920 797 273 From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of YIDING ZHOU Sent: 12 April 2011 5:42 AM To: lustre-devel at lists.lustre.org Subject: [Lustre-devel] Lustre Windows Native Client Hi there, I'm looking at a Lustre client for Windows. I find a reference here at http://wiki.lustre.org/index.php/Windows_Native_Client. However, I can't find how to participate the "early access program" anywhere. Do you know where I can find binary/source code for Lustre Windows Native Client? I heard that this project is discontinued so I hope we can take this on and continue working on it. Thanks Yiding -------------- next part -------------- An HTML attachment was scrubbed... URL: From liang at whamcloud.com Tue Apr 19 04:01:04 2011 From: liang at whamcloud.com (Liang Zhen) Date: Tue, 19 Apr 2011 12:01:04 +0800 Subject: [Lustre-devel] MDS threading model? In-Reply-To: <73AED5C780AE05478241DB067651A9210237844F@XYUS-EX22.xyus.xyratex.com> References: <2B947DB7-D404-4E77-B066-2FBCBBD90C0B@whamcloud.com> <73AED5C780AE05478241DB067651A92102000E75@XYUS-EX22.xyus.xyratex.com> <73AED5C780AE05478241DB067651A9210237844F@XYUS-EX22.xyus.xyratex.com> Message-ID: <041F3601-F6F7-4F07-A22E-1E7AD4162542@whamcloud.com> On Apr 19, 2011, at 6:41 AM, Nathan Rutman wrote: > Liang, > we're happy to share all of our Lustre architecture and feature plans, and I'm also optimistic about the way the community has really come together at LUG. > > Our plans for metadata improvements had a bit of a different tack, but it may be that with your SMP improvements there's not much more to be gained in the MDS threading model. (We're actually going to detailed investigation of your patches to see if it's worthwhile to spend more effort here.) We actually had two ideas for MDS threading improvements - one was along your lines. The other is a plan to move all the MDT ldiskfs calls from each RPC off into a single queue, instead of making the calls from separate threads. The queue would be processed by a single dedicated MDT thread. If our investigation shows there's still a lot of Personally, I think having a dedicated thread in MDT layer is not very good for a few reasons: - low layer MDD could send RPC, it's dangerous to have the single thread sending RPC - one single thread can be overloaded easily and be performance bottleneck. Actually I've tried with similar way months ago, the patch is in my branch too(lustre/mdd/mdd_sched.c), although it's a little different: - it's designed to drive any change operations to ldiskfs (handling transactions), I called it "MDD transaction handler" - just like other stack layer, each CPU partition has a requests queue, and a small threads pool to process these requests, of course threads number is much fewer then MDT service threads - "MDD transaction handler" will only process creation/removal on my branch, but it's very easy to expand to other operations. Although testing results (I don't have those data anymore, but I still can remember those graphs) shows that we gained somehow improvements of shared directory, but we got regression in other tests (i.e: many target directory tests). It's quite understandable that we see improvements of shared directory, otherwise all hundreds or thousands of MDT threads can be serialized on the single inode semaphore of the shared directory, which is very bad and kill the performance, it's also understandable that we saw regression of many target directories tests performance (i.e: uniq directory of mdtest), the patch will add one more thread context switch which increase overhead & latency, and overall more threads on runq, which is bad for system performance. So I did some changes to the patch a few months ago: only deliver high contention operations (on parent directory) to MDD transaction schedulers, by adding a small LRU cache on each CPU partition in MDD, this will help on "many target directories" case. However, I will probably drop this idea totally for some reasons: - we already got enough threads on server, more threads will overkill performance - it's hard to decide how many threads we should have in MDD (at least, I'm sure one thread is not enough) - we have to be very careful to not send RPC in MDD threads because it's unsafe - it conflicted with our other ongoing projects, and make code more complex. - the most important reason is: it benefit us too little. . for shared directory case, pdirop patch in ldiskfs can help much much more than this change (Fan Yong has already presented results on LUG) . although I've made some efforts, but still, it will add some regression to many target directories performance based on these reasons, I might remove this patch from my branch soon. > improvement to be gained beyond your patches, we will probably start working on a prototype for this to see if it helps. > > Another interesting thing that came out of LUG here besides your presentation was the Terascala work, which really highlighted for me that not all metadata ops are created equal: create rates have a heavy dependence on the OSTs, and that metadata ops performance isn't necessarily a metadata server problem! It wasn't clear to me if Oleg was saying that they had resolved the OST precreate performance limitations in Lustre 2.1 or if there is still more work to be done there. (One was a bug limiting the precreates to a small number (looks like WC LU-170 just landed for 2.1?), the other is the large growth of directory entries in the OST's 32 object dirs. (bugno?)) yes, the precreate patch has already been landed on 2.1. 32 object dirs could be issue if there is very few OSSs, however, I think it's probably fine in big cluster with tens or hundreds of OSSs, because MDS should almost always has enough precreated objects. Also, I think if we have those "CPU partition" patches in my branch on OSS, it can provide good enough performance even we only have 32 object dirs, although I agree 32 is a little low value. Regards Liang > > > > On Apr 15, 2011, at 2:21 AM, Nikita Danilov wrote: > >> Hi Nathan, Peter, >> >> I received an interesting message from Liang Zhen, which he kindly permitted me to share. The source is at http://git.whamcloud.com/?p=fs%2Flustre-dev.git;a=shortlog;h=refs%2Fheads%2Fliang%2Fb_smp. >> >> Thank you, >> Nikita. >> >> Begin forwarded message: >> >>> Hello Nikita, >>> >>> How are you doing! >>> It's great to see Xyratex sharing this white paper: >>> http://www.xyratex.com/pdfs/whitepapers/Xyratex_white_paper_Lustre_Architecture_Priorities_Overview_1-0.pdf >>> >>> it makes me feel the community is stronger than ever, :-), I'm also very interesting in the section " MDS threading model" in the white paper, because I'm working a similar thing for a while, however, I can't see too much detail from the white paper, would it be possible for you to share the idea with me? I will totally understand if you can't, thanks anyway. >>> >>> Here is what I'm doing on server side threading model now, it's probably totally different thing with what's in the white paper of Xyratex, but I just want to make sure there is no rework, also, any suggestion would be appreciated: >>> - divide server resource into several subsets at all stack level: CPUs, memory pools, threads pools, message queues..... >>> - each subset is a processing unit which can process a request standalone, unless there is conflict on target resource >>> - lifetime of request will be localized on the same processing unit as possible as we can >>> - by this way, we can dramatically reduce data migration, thread migration, lock contention between CPUs, and boost performance of metadata request processing. >>> - as you can see from above, most server side threads are binding on a few CPUs (not like before, no affinity except OST IO threads) >>> - massive number of threads will overkill performance, so limit total number of threads is another key issue, and it's important to support dynamically grow/shrink thread pool >>> - ptlrpc service will start more rational number of threads (not like now, user can have thousands threads on MDS easily) >>> - we grow number of threads only if some threads in pool are long-blocked >>> - ptlrpc service will kill some threads automatically if there are too many active threads and they are not long-blocked. >>> >>> Again, it's definitely OK if you can't share those information with me, business is business, :-) >>> >>> Thanks >>> Liang >> > > ______________________________________________________________________ > This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. > > Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. > > Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. > > The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. > ______________________________________________________________________ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eeb at whamcloud.com Tue Apr 19 13:39:57 2011 From: eeb at whamcloud.com (Eric Barton) Date: Tue, 19 Apr 2011 14:39:57 +0100 Subject: [Lustre-devel] Quota enforcement Message-ID: <011901cbfe97$46ef3630$d4cda290$@com> Hi, I'd like to take a fresh look at quota enforcement. I think the current approach of trying to implement quota purely through POSIX APIs is flawed, and I'd like to open up a debate on alternatives. If we go back to first premises, quota enforcement is about resource management - tracking and enforcing limits on consumption to ensure some measure of insulation between different users. In general, when we have 'n' resources which are all consumed independently we should also track and enforce limits on each of these independently. In conventional filesystems the relevant resources are inodes and blocks - which POSIX quota matches nicely. Although it may seem to simplify quota management to equate the POSIX quota inode count with the MDS's inode count, and the POSIX quota block count with the sum of all blocks on the OSTs, it ignores the following issues... 1. Block storage on the MDS must be sized to ensure it is not exhausted before inodes on the MDS run out. This requires assumptions about the average size of Lustre directories and utilisation of extended attributes. 2. Sufficient inodes must be reserved on the OSTs to ensure they are not exhausted before block storage. This requires assumptions about the average Lustre file size and number of stripes. 3. Imbalanced OST utilization causes allocation failures while resources are still available on other OSTs. (3) is the most glaringly obvious issue. It gives you ENOSPACE when you extend a file if one of the OSTs it's striped over is full. Very irritating if 'df' reports that plenty of space is still available and it's not something the quota system itself can help you avoid. In fact quota enforcement currently takes pains to allow quota utilisation to become imbalanced across OSTs by dynamically distributing the user's quota to where it's being used. This comes at a performance cost as quota nears exhaustion. Provided the user operates well within her quota, quota is distributed in large units with low overhead. However as she nears her limit, protocol overhead increases as quota is distributed in successively smaller units to ensure it is not all consumed prematurely on one OST. An alternative approach to (3) is to move the usage to where the resources are - i.e. implement complex/dynamic file layouts that effectively allow files to grow dynamically into available free space. This works not just for quota enforcement but for all free space. However it also comes at the cost of increasing overhead as space/quota is exhausted. It's also much harder to implement - especially for overwriting "holes" rather than simply extending files. I'd dearly like some surveys of real-world systems to discover exactly how imbalanced utilisation can really become, both for individual users and also in aggregate to provide guidance on how to proceed. I'm leaning towards static quota distribution since that matches the physical constraints, but it requires much better tools (e.g. for rebalancing files and reporting not just utilization totals but also median/min etc). Thoughts? -- Cheers, Eric Eric Barton CTO Whamcloud, Inc. Tel: +44 117 330 1575 Mob: +44 7920 797 273 From Nathan_Rutman at xyratex.com Mon Apr 18 22:41:10 2011 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Mon, 18 Apr 2011 15:41:10 -0700 Subject: [Lustre-devel] MDS threading model? In-Reply-To: <73AED5C780AE05478241DB067651A92102000E75@XYUS-EX22.xyus.xyratex.com> References: <2B947DB7-D404-4E77-B066-2FBCBBD90C0B@whamcloud.com> <73AED5C780AE05478241DB067651A92102000E75@XYUS-EX22.xyus.xyratex.com> Message-ID: <73AED5C780AE05478241DB067651A9210237844F@XYUS-EX22.xyus.xyratex.com> Liang, we're happy to share all of our Lustre architecture and feature plans, and I'm also optimistic about the way the community has really come together at LUG. Our plans for metadata improvements had a bit of a different tack, but it may be that with your SMP improvements there's not much more to be gained in the MDS threading model. (We're actually going to detailed investigation of your patches to see if it's worthwhile to spend more effort here.) We actually had two ideas for MDS threading improvements - one was along your lines. The other is a plan to move all the MDT ldiskfs calls from each RPC off into a single queue, instead of making the calls from separate threads. The queue would be processed by a single dedicated MDT thread. If our investigation shows there's still a lot of improvement to be gained beyond your patches, we will probably start working on a prototype for this to see if it helps. Another interesting thing that came out of LUG here besides your presentation was the Terascala work, which really highlighted for me that not all metadata ops are created equal: create rates have a heavy dependence on the OSTs, and that metadata ops performance isn't necessarily a metadata server problem! It wasn't clear to me if Oleg was saying that they had resolved the OST precreate performance limitations in Lustre 2.1 or if there is still more work to be done there. (One was a bug limiting the precreates to a small number (looks like WC LU-170 just landed for 2.1?), the other is the large growth of directory entries in the OST's 32 object dirs. (bugno?)) On Apr 15, 2011, at 2:21 AM, Nikita Danilov wrote: Hi Nathan, Peter, I received an interesting message from Liang Zhen, which he kindly permitted me to share. The source is at http://git.whamcloud.com/?p=fs%2Flustre-dev.git;a=shortlog;h=refs%2Fheads%2Fliang%2Fb_smp . Thank you, Nikita. Begin forwarded message: Hello Nikita, How are you doing! It's great to see Xyratex sharing this white paper: http://www.xyratex.com/pdfs/whitepapers/Xyratex_white_paper_Lustre_Architecture_Priorities_Overview_1-0.pdf it makes me feel the community is stronger than ever, :-), I'm also very interesting in the section " MDS threading model" in the white paper, because I'm working a similar thing for a while, however, I can't see too much detail from the white paper, would it be possible for you to share the idea with me? I will totally understand if you can't, thanks anyway. Here is what I'm doing on server side threading model now, it's probably totally different thing with what's in the white paper of Xyratex, but I just want to make sure there is no rework, also, any suggestion would be appreciated: - divide server resource into several subsets at all stack level: CPUs, memory pools, threads pools, message queues..... - each subset is a processing unit which can process a request standalone, unless there is conflict on target resource - lifetime of request will be localized on the same processing unit as possible as we can - by this way, we can dramatically reduce data migration, thread migration, lock contention between CPUs, and boost performance of metadata request processing. - as you can see from above, most server side threads are binding on a few CPUs (not like before, no affinity except OST IO threads) - massive number of threads will overkill performance, so limit total number of threads is another key issue, and it's important to support dynamically grow/shrink thread pool - ptlrpc service will start more rational number of threads (not like now, user can have thousands threads on MDS easily) - we grow number of threads only if some threads in pool are long-blocked - ptlrpc service will kill some threads automatically if there are too many active threads and they are not long-blocked. Again, it's definitely OK if you can't share those information with me, business is business, :-) Thanks Liang ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ben.Evans at terascala.com Tue Apr 19 13:47:18 2011 From: Ben.Evans at terascala.com (Ben Evans) Date: Tue, 19 Apr 2011 09:47:18 -0400 Subject: [Lustre-devel] MDS threading model? In-Reply-To: <041F3601-F6F7-4F07-A22E-1E7AD4162542@whamcloud.com> References: <2B947DB7-D404-4E77-B066-2FBCBBD90C0B@whamcloud.com><73AED5C780AE05478241DB067651A92102000E75@XYUS-EX22.xyus.xyratex.com><73AED5C780AE05478241DB067651A9210237844F@XYUS-EX22.xyus.xyratex.com> <041F3601-F6F7-4F07-A22E-1E7AD4162542@whamcloud.com> Message-ID: <2C7DE72B9BD00F44BAECA5B0CBB8739501135AB2@hermes.terascala.com> A couple of things regarding the Terascala work: One is that the parent-dir lock change for file creates has been taken care of in 2.1 using a much cleaner approach than I used. I'd take a look at LU-170, but whamcloud's servers seem to be down for me at the moment. I'll try again later. As to the OSTs, yes, more OSTs will solve the problem, but not every installation has the budget for hundreds or thousands of OSTs, and I'd like to see metadata performance be equal for everyone. My current understanding is that the OSTs start really hurting for performance when an OST has around 32 objects (1 million files/dir) this fits very well with ext issues (and ldiskfs) with very large numbers of files in a directory. From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Liang Zhen Sent: Tuesday, April 19, 2011 12:01 AM To: Nathan Rutman Cc: Alexander Zarochentsev; Nikita Danilov; Lustre Development Mailing List Subject: Re: [Lustre-devel] MDS threading model? On Apr 19, 2011, at 6:41 AM, Nathan Rutman wrote: Liang, we're happy to share all of our Lustre architecture and feature plans, and I'm also optimistic about the way the community has really come together at LUG. Our plans for metadata improvements had a bit of a different tack, but it may be that with your SMP improvements there's not much more to be gained in the MDS threading model. (We're actually going to detailed investigation of your patches to see if it's worthwhile to spend more effort here.) We actually had two ideas for MDS threading improvements - one was along your lines. The other is a plan to move all the MDT ldiskfs calls from each RPC off into a single queue, instead of making the calls from separate threads. The queue would be processed by a single dedicated MDT thread. If our investigation shows there's still a lot of Personally, I think having a dedicated thread in MDT layer is not very good for a few reasons: - low layer MDD could send RPC, it's dangerous to have the single thread sending RPC - one single thread can be overloaded easily and be performance bottleneck. Actually I've tried with similar way months ago, the patch is in my branch too(lustre/mdd/mdd_sched.c), although it's a little different: - it's designed to drive any change operations to ldiskfs (handling transactions), I called it "MDD transaction handler" - just like other stack layer, each CPU partition has a requests queue, and a small threads pool to process these requests, of course threads number is much fewer then MDT service threads - "MDD transaction handler" will only process creation/removal on my branch, but it's very easy to expand to other operations. Although testing results (I don't have those data anymore, but I still can remember those graphs) shows that we gained somehow improvements of shared directory, but we got regression in other tests (i.e: many target directory tests). It's quite understandable that we see improvements of shared directory, otherwise all hundreds or thousands of MDT threads can be serialized on the single inode semaphore of the shared directory, which is very bad and kill the performance, it's also understandable that we saw regression of many target directories tests performance (i.e: uniq directory of mdtest), the patch will add one more thread context switch which increase overhead & latency, and overall more threads on runq, which is bad for system performance. So I did some changes to the patch a few months ago: only deliver high contention operations (on parent directory) to MDD transaction schedulers, by adding a small LRU cache on each CPU partition in MDD, this will help on "many target directories" case. However, I will probably drop this idea totally for some reasons: - we already got enough threads on server, more threads will overkill performance - it's hard to decide how many threads we should have in MDD (at least, I'm sure one thread is not enough) - we have to be very careful to not send RPC in MDD threads because it's unsafe - it conflicted with our other ongoing projects, and make code more complex. - the most important reason is: it benefit us too little. . for shared directory case, pdirop patch in ldiskfs can help much much more than this change (Fan Yong has already presented results on LUG) . although I've made some efforts, but still, it will add some regression to many target directories performance based on these reasons, I might remove this patch from my branch soon. improvement to be gained beyond your patches, we will probably start working on a prototype for this to see if it helps. Another interesting thing that came out of LUG here besides your presentation was the Terascala work, which really highlighted for me that not all metadata ops are created equal: create rates have a heavy dependence on the OSTs, and that metadata ops performance isn't necessarily a metadata server problem! It wasn't clear to me if Oleg was saying that they had resolved the OST precreate performance limitations in Lustre 2.1 or if there is still more work to be done there. (One was a bug limiting the precreates to a small number (looks like WC LU-170 just landed for 2.1?), the other is the large growth of directory entries in the OST's 32 object dirs. (bugno?)) yes, the precreate patch has already been landed on 2.1. 32 object dirs could be issue if there is very few OSSs, however, I think it's probably fine in big cluster with tens or hundreds of OSSs, because MDS should almost always has enough precreated objects. Also, I think if we have those "CPU partition" patches in my branch on OSS, it can provide good enough performance even we only have 32 object dirs, although I agree 32 is a little low value. Regards Liang On Apr 15, 2011, at 2:21 AM, Nikita Danilov wrote: Hi Nathan, Peter, I received an interesting message from Liang Zhen, which he kindly permitted me to share. The source is at http://git.whamcloud.com/?p=fs%2Flustre-dev.git;a=shortlog;h=refs%2Fhead s%2Fliang%2Fb_smp . Thank you, Nikita. Begin forwarded message: Hello Nikita, How are you doing! It's great to see Xyratex sharing this white paper: http://www.xyratex.com/pdfs/whitepapers/Xyratex_white_paper_Lustre_Archi tecture_Priorities_Overview_1-0.pdf it makes me feel the community is stronger than ever, :-), I'm also very interesting in the section " MDS threading model" in the white paper, because I'm working a similar thing for a while, however, I can't see too much detail from the white paper, would it be possible for you to share the idea with me? I will totally understand if you can't, thanks anyway. Here is what I'm doing on server side threading model now, it's probably totally different thing with what's in the white paper of Xyratex, but I just want to make sure there is no rework, also, any suggestion would be appreciated: - divide server resource into several subsets at all stack level: CPUs, memory pools, threads pools, message queues..... - each subset is a processing unit which can process a request standalone, unless there is conflict on target resource - lifetime of request will be localized on the same processing unit as possible as we can - by this way, we can dramatically reduce data migration, thread migration, lock contention between CPUs, and boost performance of metadata request processing. - as you can see from above, most server side threads are binding on a few CPUs (not like before, no affinity except OST IO threads) - massive number of threads will overkill performance, so limit total number of threads is another key issue, and it's important to support dynamically grow/shrink thread pool - ptlrpc service will start more rational number of threads (not like now, user can have thousands threads on MDS easily) - we grow number of threads only if some threads in pool are long-blocked - ptlrpc service will kill some threads automatically if there are too many active threads and they are not long-blocked. Again, it's definitely OK if you can't share those information with me, business is business, :-) Thanks Liang ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Tue Apr 19 18:00:23 2011 From: adilger at whamcloud.com (Andreas Dilger) Date: Tue, 19 Apr 2011 12:00:23 -0600 Subject: [Lustre-devel] Quota enforcement In-Reply-To: <011901cbfe97$46ef3630$d4cda290$@com> References: <011901cbfe97$46ef3630$d4cda290$@com> Message-ID: <7A5E948E-231A-4D8C-9984-97A4D213D02F@whamcloud.com> On 2011-04-19, at 7:39 AM, "Eric Barton" wrote: > I'd like to take a fresh look at quota enforcement. I think the > current approach of trying to implement quota purely through POSIX > APIs is flawed, and I'd like to open up a debate on alternatives. > > If we go back to first premises, quota enforcement is about resource > management - tracking and enforcing limits on consumption to ensure > some measure of insulation between different users. In general, when > we have 'n' resources which are all consumed independently we should > also track and enforce limits on each of these independently. Agreed. In ldiskfs this means tracking inodes and blocks separately, because they have very different constraints. It also means that the quota for MDT and OST space needs to be tracked separately. > In conventional filesystems the relevant resources are inodes and > blocks - which POSIX quota matches nicely. Although it may seem to > simplify quota management to equate the POSIX quota inode count with > the MDS's inode count, and the POSIX quota block count with the sum of > all blocks on the OSTs, it ignores the following issues... > > 1. Block storage on the MDS must be sized to ensure it is not > exhausted before inodes on the MDS run out. This requires > assumptions about the average size of Lustre directories and > utilisation of extended attributes. Agreed. This is definitely the case with current MDTs - the amount of free space is currently always far in excess of what is needed for the number of inodes. For HDDs this is fine because MDT space is "free" when adding spindles for the IOPS. In any case, the current quota does account for space usage on the MDT. This is important for future data-on-MDT cases. The inode consumption is also a useful metric because it allows the admin to track the fixed ldiskfs inode resource, and also to compute average file size on a per user basis. > 2. Sufficient inodes must be reserved on the OSTs to ensure they are > not exhausted before block storage. This requires assumptions > about the average Lustre file size and number of stripes. That is true, but inevitable with the structure of ldiskfs. In the past we have FAR over-provisioned the inodes on ldiskfs OSTs by default because we cannot know in advance what the user is going to be doing with their filesystem. There is guidance in the manual for tuning this at setup time, but not everyone reads the manual... I consider the use of inodes/objects on OSTs to be a side-effect of the Lustre implementation, so as long as we have enough I don't think they should be accounted separately, and definitely not in the same bucket as MDT inodes. This is because the performance characteristics of a single file vary with the number of objects == OSTs, and the user may not even be controlling this mapping in the future with dynamic layouts. With newer filesystems that have dynamic inode allocation, the concept of inodes being a constrained resource on the OSTs is gone, and essentially this is just a small overhead of space on the OST, like an indirect block is. > 3. Imbalanced OST utilization causes allocation failures while > resources are still available on other OSTs. That is true whether the allocation failure is due to the OST filling up with a dynamic quota, or if the quota is running out on an OST with a static quota. Both problems would ideally be avoided by perfectly even usage, but that is not realistic. > (3) is the most glaringly obvious issue. It gives you ENOSPACE when > you extend a file if one of the OSTs it's striped over is full. Very > irritating if 'df' reports that plenty of space is still available and > it's not something the quota system itself can help you avoid. > > In fact quota enforcement currently takes pains to allow quota > utilisation to become imbalanced across OSTs by dynamically > distributing the user's quota to where it's being used. This comes at > a performance cost as quota nears exhaustion. Provided the user > operates well within her quota, quota is distributed in large units > with low overhead. However as she nears her limit, protocol overhead > increases as quota is distributed in successively smaller units to > ensure it is not all consumed prematurely on one OST. > > An alternative approach to (3) is to move the usage to where the > resources are - i.e. implement complex/dynamic file layouts that > effectively allow files to grow dynamically into available free space. > This works not just for quota enforcement but for all free space. > However it also comes at the cost of increasing overhead as > space/quota is exhausted. It's also much harder to implement - > especially for overwriting "holes" rather than simply extending files. This doesn't seem like a huge win to me, and comes at a cost in implementation complexity. I agree that dynamic layouts is something that we've discussed for a long time already, and we are moving in that direction with the layout lock from HSM, but we are still some distance away from this today, IMHO. We also have to consider that dynamic layout for large files may not always be a magic bullet. If there are many processes writing what the MDT considers "large" files, then always increasing the stripe count would only serve to increase contention on the OSTs, and not improve apace balance at all. > I'd dearly like some surveys of real-world systems to discover exactly > how imbalanced utilisation can really become, both for individual > users and also in aggregate to provide guidance on how to proceed. The common case of severe imbalance that I'm aware of is a user creating a huge tar file of their output, but not specifying a wide striping, so thousands of files currently spread over many OSTs are copied onto one or two OSTs where the tar file is located. The only other real-world case I'm aware of is applications/users specifying striping with an index == 0 instead of -1 that causes the first OST(s) to fill unevenly. I'm some sense, this is an artifact of the first problem, that users need to specify striping in the first place. With balanced round-robin allocation, the majority of "general" imbalance will be avoided, and dynamic layouts would help those extreme cases of large imbalanced files. > I'm leaning towards static quota distribution since that matches the > physical constraints, but it requires much better tools (e.g. for > rebalancing files and reporting not just utilization totals but also > median/min etc). Cheers, Andreas From adilger at whamcloud.com Thu Apr 21 18:40:15 2011 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 21 Apr 2011 12:40:15 -0600 Subject: [Lustre-devel] Research on filesystem metadata operation distribution Message-ID: I'm trying to get some data about the relative distribution of MDS operations in the wild, and I'd be grateful if some people with production filesystems that have been running for at least a week could collect some simple stats and email them to me. They can be collected by any regular user on the MDS node: lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync" It would be useful to also include "lfs df" and "lfs df -i" information, as well as a brief description of what the filesystem is used for (scratch, home, project, archive, etc). As a reminder, I'm also interested if some Lustre admins could run the "fsstats" tool from http://www.pdsi-scidac.org/fsstats/ and send me the output. Sending the output to PDSI via their submission form may also produce some positive results. http://www.pdsi-scidac.org/fsstats/files/fsstats-1.4.5.tar.gz Thanks in advance for any data. I've set replies to go only to lustre-devel, to avoid clogging the larger readership of lustre-discuss, but it may be useful for others to have this in a list archive and/or searchable via Google in the future so I don't necessarily want to keep it all to myself. Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. From Nathan_Rutman at xyratex.com Thu Apr 21 19:42:33 2011 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Thu, 21 Apr 2011 12:42:33 -0700 Subject: [Lustre-devel] 64-bit ext4 inodes In-Reply-To: <2368DAF2-4CA9-41E4-8A56-E48AEA9D4549@xyratex.com> References: <2368DAF2-4CA9-41E4-8A56-E48AEA9D4549@xyratex.com> Message-ID: <73AED5C780AE05478241DB067651A92102378476@XYUS-EX22.xyus.xyratex.com> For some reason, this doesn't seem to be getting through lustre-devel, but Andreas you probably have the pulse on this better than anyone... On Apr 20, 2011, at 11:07 AM, Nathan Rutman wrote: > We're about to start looking into 64-bit inodes on ext4 -- anybody else working on this at the moment? > ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________ From cwalker at fas.harvard.edu Thu Apr 21 18:50:28 2011 From: cwalker at fas.harvard.edu (Chris Walker) Date: Thu, 21 Apr 2011 14:50:28 -0400 Subject: [Lustre-devel] [Lustre-discuss] Research on filesystem metadata operation distribution In-Reply-To: References: Message-ID: <4DB07C74.5050206@fas.harvard.edu> /n/data (general purpose): [root at circemds1 ~]# lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync" open 27110498653 samples [reqs] close 5181068572 samples [reqs] link 19102 samples [reqs] unlink 95211743 samples [reqs] rename 11919203 samples [reqs] setattr 401208317 samples [reqs] getattr 3502169172 samples [reqs] lfs df UUID 1K-blocks Used Available Use% Mounted on circelfs-MDT0000_UUID 624910300 15503992 573693816 2% /n/data[MDT:0] circelfs-OST0000_UUID 3759898504 3576773148 183098116 95% /n/data[OST:0] circelfs-OST0001_UUID 3759898504 3567047824 192849208 94% /n/data[OST:1] circelfs-OST0002_UUID 3844954168 3634309372 210564872 94% /n/data[OST:2] circelfs-OST0003_UUID 3844954168 3642678204 202274940 94% /n/data[OST:3] circelfs-OST0004_UUID 3729601424 3539940528 189634296 94% /n/data[OST:4] circelfs-OST0005_UUID 3844954168 3658277796 186643060 95% /n/data[OST:5] circelfs-OST0006_UUID 3844954168 3650323540 194589396 94% /n/data[OST:6] circelfs-OST0007_UUID 3844954168 3648381604 196541040 94% /n/data[OST:7] circelfs-OST0008_UUID 3759898504 3589631096 170173692 95% /n/data[OST:8] circelfs-OST0009_UUID 3759898504 3579220544 180625232 95% /n/data[OST:9] circelfs-OST000a_UUID 3844954168 3678558980 166373848 95% /n/data[OST:10] circelfs-OST000b_UUID 3844954168 3675247472 169659080 95% /n/data[OST:11] circelfs-OST000c_UUID 3729601424 3529929552 199628560 94% /n/data[OST:12] circelfs-OST000d_UUID 3844954168 3616859740 228051912 94% /n/data[OST:13] circelfs-OST000e_UUID 3844954168 3625217752 219699816 94% /n/data[OST:14] circelfs-OST000f_UUID 3844954168 3644184848 200720764 94% /n/data[OST:15] circelfs-OST0010_UUID 3759898504 3576388952 183459748 95% /n/data[OST:16] circelfs-OST0011_UUID 3759898504 3560942452 198889108 94% /n/data[OST:17] circelfs-OST0012_UUID 3844954168 3664210272 180717356 95% /n/data[OST:18] circelfs-OST0013_UUID 3844954168 3657736452 187188384 95% /n/data[OST:19] circelfs-OST0014_UUID 3729601424 3559121744 170414456 95% /n/data[OST:20] circelfs-OST0015_UUID 3844954168 3619354872 225542700 94% /n/data[OST:21] circelfs-OST0016_UUID 3844954168 3640468044 204446952 94% /n/data[OST:22] circelfs-OST0017_UUID 3844954168 3674764168 170127980 95% /n/data[OST:23] circelfs-OST0018_UUID 3844954168 3627092948 217828696 94% /n/data[OST:24] circelfs-OST0019_UUID 3729601424 3535664852 193935548 94% /n/data[OST:25] circelfs-OST001a_UUID 3844954168 3634647484 210305868 94% /n/data[OST:26] circelfs-OST001b_UUID 3844954168 3636805036 208106824 94% /n/data[OST:27] circelfs-OST001c_UUID 3759898504 3556374520 203433620 94% /n/data[OST:28] circelfs-OST001d_UUID 3844954168 3662286736 182643296 95% /n/data[OST:29] circelfs-OST001e_UUID 3844954168 3638805056 206123324 94% /n/data[OST:30] circelfs-OST001f_UUID 1879950588 1782719784 97187448 94% /n/data[OST:31] circelfs-OST0020_UUID 3759898504 3581449120 178387092 95% /n/data[OST:32] circelfs-OST0021_UUID 4699875208 4416545280 283272492 93% /n/data[OST:33] circelfs-OST0022_UUID 4806194208 4539916232 266253044 94% /n/data[OST:34] circelfs-OST0023_UUID 4806194208 4506930724 299234284 93% /n/data[OST:35] circelfs-OST0024_UUID 4662002272 4391493156 270508092 94% /n/data[OST:36] circelfs-OST0025_UUID 4806194208 4575711520 230427632 95% /n/data[OST:37] circelfs-OST0026_UUID 4806194208 4556692828 249458004 94% /n/data[OST:38] circelfs-OST0027_UUID 4806194208 4516547112 289628792 93% /n/data[OST:39] filesystem summary: 157169476196 148969251344 8198648572 94% /n/data lfs df -i UUID Inodes IUsed IFree IUse% Mounted on circelfs-MDT0000_UUID 178585600 159843235 18742365 89% /n/data[MDT:0] circelfs-OST0000_UUID 49665762 3884423 45781339 7% /n/data[OST:0] circelfs-OST0001_UUID 52140996 3921567 48219429 7% /n/data[OST:1] circelfs-OST0002_UUID 56777852 4116653 52661199 7% /n/data[OST:2] circelfs-OST0003_UUID 54642729 4069383 50573346 7% /n/data[OST:3] circelfs-OST0004_UUID 51319864 3904640 47415224 7% /n/data[OST:4] circelfs-OST0005_UUID 50833017 4163924 46669093 8% /n/data[OST:5] circelfs-OST0006_UUID 52922813 4265156 48657657 8% /n/data[OST:6] circelfs-OST0007_UUID 53335983 4192842 49143141 7% /n/data[OST:7] circelfs-OST0008_UUID 46382472 3815620 42566852 8% /n/data[OST:8] circelfs-OST0009_UUID 49119318 3949828 45169490 8% /n/data[OST:9] circelfs-OST000a_UUID 45567368 3968571 41598797 8% /n/data[OST:10] circelfs-OST000b_UUID 46377787 3951113 42426674 8% /n/data[OST:11] circelfs-OST000c_UUID 53972974 4055006 49917968 7% /n/data[OST:12] circelfs-OST000d_UUID 60536895 3513288 57023607 5% /n/data[OST:13] circelfs-OST000e_UUID 58664263 3729926 54934337 6% /n/data[OST:14] circelfs-OST000f_UUID 54002048 3805109 50196939 7% /n/data[OST:15] circelfs-OST0010_UUID 50010476 4133088 45877388 8% /n/data[OST:16] circelfs-OST0011_UUID 53478928 3739762 49739166 6% /n/data[OST:17] circelfs-OST0012_UUID 48965685 3779711 45185974 7% /n/data[OST:18] circelfs-OST0013_UUID 50706841 3902412 46804429 7% /n/data[OST:19] circelfs-OST0014_UUID 46622191 4002268 42619923 8% /n/data[OST:20] circelfs-OST0015_UUID 60322395 3926565 56395830 6% /n/data[OST:21] circelfs-OST0016_UUID 55176193 4054521 51121672 7% /n/data[OST:22] circelfs-OST0017_UUID 46475197 3927689 42547508 8% /n/data[OST:23] circelfs-OST0018_UUID 58494562 4023368 54471194 6% /n/data[OST:24] circelfs-OST0019_UUID 52507373 4019644 48487729 7% /n/data[OST:25] circelfs-OST001a_UUID 56467843 3889636 52578207 6% /n/data[OST:26] circelfs-OST001b_UUID 56069947 4032664 52037283 7% /n/data[OST:27] circelfs-OST001c_UUID 55008022 4127020 50881002 7% /n/data[OST:28] circelfs-OST001d_UUID 49505092 3838234 45666858 7% /n/data[OST:29] circelfs-OST001e_UUID 55683855 4146575 51537280 7% /n/data[OST:30] circelfs-OST001f_UUID 26327143 2019442 24307701 7% /n/data[OST:31] circelfs-OST0020_UUID 47956346 3344000 44612346 6% /n/data[OST:32] circelfs-OST0021_UUID 74869419 4037050 70832369 5% /n/data[OST:33] circelfs-OST0022_UUID 70831554 4287967 66543587 6% /n/data[OST:34] circelfs-OST0023_UUID 78978347 4162475 74815872 5% /n/data[OST:35] circelfs-OST0024_UUID 71879035 4250476 67628559 5% /n/data[OST:36] circelfs-OST0025_UUID 61820346 4199521 57620825 6% /n/data[OST:37] circelfs-OST0026_UUID 66674379 4299034 62375345 6% /n/data[OST:38] circelfs-OST0027_UUID 76668558 4256622 72411936 5% /n/data[OST:39] filesystem summary: 178585600 159843235 18742365 89% /n/data /n/panlfs (general purpose): [root at panmds ~]# lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync" open 55439389107 samples [reqs] close 5873727879 samples [reqs] link 1855475 samples [reqs] unlink 127524184 samples [reqs] rename 8877227 samples [reqs] getxattr 26584744 samples [reqs] setattr 171354158 samples [reqs] getattr 3188738225 samples [reqs] [root at panmds ~]# lfs df UUID 1K-blocks Used Available Use% Mounted on panlfs-MDT0000_UUID 1181496752 33912532 1080064732 2% /n/panlfs[MDT:0] panlfs-OST0000_UUID 5765436120 4974923264 790512780 86% /n/panlfs[OST:0] panlfs-OST0001_UUID 5765436120 4967568572 797754164 86% /n/panlfs[OST:1] panlfs-OST0002_UUID 5765436120 4965892684 799366576 86% /n/panlfs[OST:2] panlfs-OST0003_UUID 5765436120 4965261380 800015864 86% /n/panlfs[OST:3] panlfs-OST0004_UUID 5765436120 4963104608 802331336 86% /n/panlfs[OST:4] panlfs-OST0005_UUID 5765436120 5014836692 750577924 86% /n/panlfs[OST:5] panlfs-OST0006_UUID 5765436120 4980876888 784558272 86% /n/panlfs[OST:6] panlfs-OST0007_UUID 5765436120 4991355684 774080308 86% /n/panlfs[OST:7] panlfs-OST0008_UUID 5765436120 4959617256 805687812 86% /n/panlfs[OST:8] panlfs-OST0009_UUID 5765436120 4961260628 804175148 86% /n/panlfs[OST:9] panlfs-OST000a_UUID 5765436120 4964872652 800558348 86% /n/panlfs[OST:10] panlfs-OST000b_UUID 5765436120 4961539444 803894612 86% /n/panlfs[OST:11] panlfs-OST000c_UUID 5765436120 4968309644 796943316 86% /n/panlfs[OST:12] panlfs-OST000d_UUID 5765436120 4966318752 799117364 86% /n/panlfs[OST:13] panlfs-OST000e_UUID 5765436120 4971024728 794309804 86% /n/panlfs[OST:14] panlfs-OST000f_UUID 5765436120 4964518012 800767592 86% /n/panlfs[OST:15] panlfs-OST0010_UUID 5765436120 4971079772 794356052 86% /n/panlfs[OST:16] panlfs-OST0011_UUID 5765436120 4947691048 817743996 85% /n/panlfs[OST:17] panlfs-OST0012_UUID 5765436120 4782610928 982801640 82% /n/panlfs[OST:18] panlfs-OST0013_UUID 5765436120 4791320796 974109108 83% /n/panlfs[OST:19] panlfs-OST0014_UUID 5765436120 4788726168 976685372 83% /n/panlfs[OST:20] panlfs-OST0015_UUID 5765436120 4780325788 985098044 82% /n/panlfs[OST:21] panlfs-OST0016_UUID 5765436120 4831302484 934133556 83% /n/panlfs[OST:22] panlfs-OST0017_UUID 5765436120 4789102248 976324656 83% /n/panlfs[OST:23] panlfs-OST0018_UUID 5765436120 4784434788 980982900 82% /n/panlfs[OST:24] panlfs-OST0019_UUID 5765436120 4786139740 979295356 83% /n/panlfs[OST:25] panlfs-OST001a_UUID 5765436120 4784068940 981247400 82% /n/panlfs[OST:26] panlfs-OST001b_UUID 5765436120 4781602996 983714956 82% /n/panlfs[OST:27] panlfs-OST001c_UUID 5765436120 4781088948 984338980 82% /n/panlfs[OST:28] panlfs-OST001d_UUID 5765436120 4780381632 985054484 82% /n/panlfs[OST:29] panlfs-OST001e_UUID 5766984312 4069861960 1696963576 70% /n/panlfs[OST:30] panlfs-OST001f_UUID 5766984312 3908778724 1858056516 67% /n/panlfs[OST:31] panlfs-OST0020_UUID 5766984312 3916492512 1850483608 67% /n/panlfs[OST:32] panlfs-OST0021_UUID 5766984312 4015278136 1751682736 69% /n/panlfs[OST:33] panlfs-OST0022_UUID 5766984312 4039640892 1727343380 70% /n/panlfs[OST:34] panlfs-OST0023_UUID 5766984312 4032875720 1734085616 69% /n/panlfs[OST:35] panlfs-OST0024_UUID 7689310636 5379578792 2309710912 69% /n/panlfs[OST:36] panlfs-OST0025_UUID 7689310636 5368547620 2320760968 69% /n/panlfs[OST:37] panlfs-OST0026_UUID 5766984312 4031962692 1735021096 69% /n/panlfs[OST:38] panlfs-OST0027_UUID 5766984312 4064421580 1702550384 70% /n/panlfs[OST:39] panlfs-OST0028_UUID 5766984312 4017407756 1749576536 69% /n/panlfs[OST:40] panlfs-OST0029_UUID 5766984312 4088980072 1678004236 70% /n/panlfs[OST:41] panlfs-OST002a_UUID 5766984312 4037071336 1729901712 70% /n/panlfs[OST:42] panlfs-OST002b_UUID 5766984312 4031950200 1735034096 69% /n/panlfs[OST:43] panlfs-OST002c_UUID 7689310636 5430528320 2258779760 70% /n/panlfs[OST:44] panlfs-OST002d_UUID 7689310636 5311913940 2377396612 69% /n/panlfs[OST:45] panlfs-OST002e_UUID 5766984312 4025550116 1741432148 69% /n/panlfs[OST:46] panlfs-OST002f_UUID 5766984312 3953045972 1813937256 68% /n/panlfs[OST:47] panlfs-OST0030_UUID 5766984312 4208388240 1558493060 72% /n/panlfs[OST:48] panlfs-OST0031_UUID 5766984312 4011293516 1755680556 69% /n/panlfs[OST:49] panlfs-OST0032_UUID 5766984312 3926746328 1840231656 68% /n/panlfs[OST:50] panlfs-OST0033_UUID 5766984312 4027338528 1739639084 69% /n/panlfs[OST:51] panlfs-OST0034_UUID 7689310636 5377549996 2311737664 69% /n/panlfs[OST:52] panlfs-OST0035_UUID 7689310636 5372207348 2317103220 69% /n/panlfs[OST:53] panlfs-OST0036_UUID 5766984312 3927872888 1839107328 68% /n/panlfs[OST:54] panlfs-OST0037_UUID 5766984312 4023713564 1743270716 69% /n/panlfs[OST:55] panlfs-OST0038_UUID 5766984312 4021830276 1745148900 69% /n/panlfs[OST:56] panlfs-OST0039_UUID 5766984312 3934077132 1832886248 68% /n/panlfs[OST:57] panlfs-OST003a_UUID 5766984312 4033701168 1733152684 69% /n/panlfs[OST:58] panlfs-OST003b_UUID 5766984312 4023597696 1743383540 69% /n/panlfs[OST:59] panlfs-OST003c_UUID 7689310636 5376225432 2313034004 69% /n/panlfs[OST:60] panlfs-OST003d_UUID 7689310636 5382523640 2306769476 70% /n/panlfs[OST:61] filesystem summary: 372885192176 286292109256 86590897004 76% /n/panlfs lfs df -i UUID Inodes IUsed IFree IUse% Mounted on panlfs-MDT0000_UUID 337608704 100870063 236738641 29% /n/panlfs[MDT:0] panlfs-OST0000_UUID 199061357 1428948 197632409 0% /n/panlfs[OST:0] panlfs-OST0001_UUID 200841380 1433128 199408252 0% /n/panlfs[OST:1] panlfs-OST0002_UUID 201279904 1435696 199844208 0% /n/panlfs[OST:2] panlfs-OST0003_UUID 201426659 1442402 199984257 0% /n/panlfs[OST:3] panlfs-OST0004_UUID 202009708 1439610 200570098 0% /n/panlfs[OST:4] panlfs-OST0005_UUID 189068210 1433302 187634908 0% /n/panlfs[OST:5] panlfs-OST0006_UUID 197500278 1430549 196069729 0% /n/panlfs[OST:6] panlfs-OST0007_UUID 194897600 1412501 193485099 0% /n/panlfs[OST:7] panlfs-OST0008_UUID 202853017 1444880 201408137 0% /n/panlfs[OST:8] panlfs-OST0009_UUID 202450065 1442617 201007448 0% /n/panlfs[OST:9] panlfs-OST000a_UUID 201533261 1441066 200092195 0% /n/panlfs[OST:10] panlfs-OST000b_UUID 202347095 1441975 200905120 0% /n/panlfs[OST:11] panlfs-OST000c_UUID 200661982 1429801 199232181 0% /n/panlfs[OST:12] panlfs-OST000d_UUID 201207975 1424647 199783328 0% /n/panlfs[OST:13] panlfs-OST000e_UUID 200013486 1435930 198577556 0% /n/panlfs[OST:14] panlfs-OST000f_UUID 201655294 1437041 200218253 0% /n/panlfs[OST:15] panlfs-OST0010_UUID 199977755 1437687 198540068 0% /n/panlfs[OST:16] panlfs-OST0011_UUID 205792546 1441571 204350975 0% /n/panlfs[OST:17] panlfs-OST0012_UUID 247183952 1510162 245673790 0% /n/panlfs[OST:18] panlfs-OST0013_UUID 244975158 1502607 243472551 0% /n/panlfs[OST:19] panlfs-OST0014_UUID 245626362 1517839 244108523 0% /n/panlfs[OST:20] panlfs-OST0015_UUID 247752267 1514148 246238119 0% /n/panlfs[OST:21] panlfs-OST0016_UUID 235012597 1514167 233498430 0% /n/panlfs[OST:22] panlfs-OST0017_UUID 245566479 1523072 244043407 0% /n/panlfs[OST:23] panlfs-OST0018_UUID 246728144 1511148 245216996 0% /n/panlfs[OST:24] panlfs-OST0019_UUID 246287058 1508750 244778308 0% /n/panlfs[OST:25] panlfs-OST001a_UUID 246828538 1519283 245309255 0% /n/panlfs[OST:26] panlfs-OST001b_UUID 247425369 1517824 245907545 0% /n/panlfs[OST:27] panlfs-OST001c_UUID 247570951 1520608 246050343 0% /n/panlfs[OST:28] panlfs-OST001d_UUID 247744143 1521256 246222887 0% /n/panlfs[OST:29] panlfs-OST001e_UUID 366182400 1402921 364779479 0% /n/panlfs[OST:30] panlfs-OST001f_UUID 366182400 1413243 364769157 0% /n/panlfs[OST:31] panlfs-OST0020_UUID 366182400 1407899 364774501 0% /n/panlfs[OST:32] panlfs-OST0021_UUID 366182400 1449305 364733095 0% /n/panlfs[OST:33] panlfs-OST0022_UUID 366182400 1435498 364746902 0% /n/panlfs[OST:34] panlfs-OST0023_UUID 366182400 1446112 364736288 0% /n/panlfs[OST:35] panlfs-OST0024_UUID 488243200 1924003 486319197 0% /n/panlfs[OST:36] panlfs-OST0025_UUID 488243200 1932387 486310813 0% /n/panlfs[OST:37] panlfs-OST0026_UUID 366182400 1440911 364741489 0% /n/panlfs[OST:38] panlfs-OST0027_UUID 366182400 1444227 364738173 0% /n/panlfs[OST:39] panlfs-OST0028_UUID 366182400 1446877 364735523 0% /n/panlfs[OST:40] panlfs-OST0029_UUID 366182400 1433436 364748964 0% /n/panlfs[OST:41] panlfs-OST002a_UUID 366182400 1446075 364736325 0% /n/panlfs[OST:42] panlfs-OST002b_UUID 366182400 1444201 364738199 0% /n/panlfs[OST:43] panlfs-OST002c_UUID 488243200 1928893 486314307 0% /n/panlfs[OST:44] panlfs-OST002d_UUID 488243200 1884929 486358271 0% /n/panlfs[OST:45] panlfs-OST002e_UUID 366182400 1445724 364736676 0% /n/panlfs[OST:46] panlfs-OST002f_UUID 366182400 1405979 364776421 0% /n/panlfs[OST:47] panlfs-OST0030_UUID 366182400 1307695 364874705 0% /n/panlfs[OST:48] panlfs-OST0031_UUID 366182400 1449966 364732434 0% /n/panlfs[OST:49] panlfs-OST0032_UUID 366182400 1409123 364773277 0% /n/panlfs[OST:50] panlfs-OST0033_UUID 366182400 1447543 364734857 0% /n/panlfs[OST:51] panlfs-OST0034_UUID 488243200 1924754 486318446 0% /n/panlfs[OST:52] panlfs-OST0035_UUID 488243200 1924491 486318709 0% /n/panlfs[OST:53] panlfs-OST0036_UUID 366182400 1402421 364779979 0% /n/panlfs[OST:54] panlfs-OST0037_UUID 366182400 1445228 364737172 0% /n/panlfs[OST:55] panlfs-OST0038_UUID 366182400 1445448 364736952 0% /n/panlfs[OST:56] panlfs-OST0039_UUID 366182400 1409186 364773214 0% /n/panlfs[OST:57] panlfs-OST003a_UUID 366182400 1443444 364738956 0% /n/panlfs[OST:58] panlfs-OST003b_UUID 366182400 1445485 364736915 0% /n/panlfs[OST:59] panlfs-OST003c_UUID 488243200 1925336 486317864 0% /n/panlfs[OST:60] panlfs-OST003d_UUID 488243200 1920467 486322733 0% /n/panlfs[OST:61] filesystem summary: 337608704 100870063 236738641 29% /n/panlfs On 4/21/11 2:40 PM, Andreas Dilger wrote: > I'm trying to get some data about the relative distribution of MDS operations in the wild, and I'd be grateful if some people with production filesystems that have been running for at least a week could collect some simple stats and email them to me. They can be collected by any regular user on the MDS node: > > lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync" > > It would be useful to also include "lfs df" and "lfs df -i" information, as well as a brief description of what the filesystem is used for (scratch, home, project, archive, etc). > > > > As a reminder, I'm also interested if some Lustre admins could run the "fsstats" tool from http://www.pdsi-scidac.org/fsstats/ and send me the output. Sending the output to PDSI via their submission form may also produce some positive results. > > http://www.pdsi-scidac.org/fsstats/files/fsstats-1.4.5.tar.gz > > > Thanks in advance for any data. I've set replies to go only to lustre-devel, to avoid clogging the larger readership of lustre-discuss, but it may be useful for others to have this in a list archive and/or searchable via Google in the future so I don't necessarily want to keep it all to myself. > > Cheers, Andreas > -- > Andreas Dilger > Principal Engineer > Whamcloud, Inc. > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss From mark at msi.umn.edu Thu Apr 21 21:47:22 2011 From: mark at msi.umn.edu (Mark Nelson) Date: Thu, 21 Apr 2011 16:47:22 -0500 Subject: [Lustre-devel] [Lustre-discuss] Research on filesystem metadata operation distribution In-Reply-To: References: Message-ID: <4DB0A5EA.7090100@msi.umn.edu> Hi Andreas, Here are the statistics for our current lustre storage as requested. This filesystem is used for scratch on a ~8700 core cluster. > [root at mds ~]# lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync" > open 52365106 samples [reqs] > close 26309366 samples [reqs] > link 164 samples [reqs] > unlink 4199061 samples [reqs] > rename 605499 samples [reqs] > getattr 204467294 samples [reqs] > setattr 6947314 samples [reqs] > getxattr 1211618173 samples [reqs] > sync 2399 samples [reqs] > mark at node1083:~> lfs df > UUID 1K-blocks Used Available Use% Mounted on > lustre-MDT0000_UUID 2446151592 7290552 2299069836 0% /scratch1[MDT:0] > lustre-OST0000_UUID 2850686364 2260170480 445708680 79% /scratch1[OST:0] > lustre-OST0001_UUID 2850686364 2360436928 345415236 82% /scratch1[OST:1] > lustre-OST0002_UUID 2850686364 2301852200 404026800 80% /scratch1[OST:2] > lustre-OST0003_UUID 2850686364 2297681284 408197928 80% /scratch1[OST:3] > lustre-OST0004_UUID 2850686364 2268630672 437220700 79% /scratch1[OST:4] > lustre-OST0005_UUID 2850686364 2352652832 353226340 82% /scratch1[OST:5] > lustre-OST0006_UUID 2850686364 2256628780 449250944 79% /scratch1[OST:6] > lustre-OST0007_UUID 2850686364 2239905176 465895280 78% /scratch1[OST:7] > lustre-OST0008_UUID 2850686364 2256707044 449145120 79% /scratch1[OST:8] > lustre-OST0009_UUID 2850686364 2318356564 387467160 81% /scratch1[OST:9] > lustre-OST000a_UUID 2850686364 2246170292 459654224 78% /scratch1[OST:10] > lustre-OST000b_UUID 2850686364 2289467904 416357844 80% /scratch1[OST:11] > lustre-OST000c_UUID 2850686364 2267773568 438077736 79% /scratch1[OST:12] > lustre-OST000d_UUID 2850686364 2370072892 335806104 83% /scratch1[OST:13] > lustre-OST000e_UUID 2850686364 2373615788 332262332 83% /scratch1[OST:14] > lustre-OST000f_UUID 2850686364 2268974064 436784684 79% /scratch1[OST:15] > lustre-OST0010_UUID 2850686364 2254083524 451795496 79% /scratch1[OST:16] > lustre-OST0011_UUID 2850686364 2305220652 400636828 80% /scratch1[OST:17] > lustre-OST0012_UUID 2850686364 2289688148 416189404 80% /scratch1[OST:18] > lustre-OST0013_UUID 2850686364 2248032400 457846596 78% /scratch1[OST:19] > lustre-OST0014_UUID 2850686364 2348553416 357297956 82% /scratch1[OST:20] > lustre-OST0015_UUID 2850686364 2336745344 369133676 81% /scratch1[OST:21] > lustre-OST0016_UUID 2850686364 2286710092 419095976 80% /scratch1[OST:22] > lustre-OST0017_UUID 2850686364 2258930636 446948364 79% /scratch1[OST:23] > lustre-OST0018_UUID 2850686364 2394833760 311045260 84% /scratch1[OST:24] > lustre-OST0019_UUID 2850686364 2315306792 390571460 81% /scratch1[OST:25] > lustre-OST001a_UUID 2850686364 2300589812 405289220 80% /scratch1[OST:26] > lustre-OST001b_UUID 2850686364 2296248192 409542792 80% /scratch1[OST:27] > lustre-OST001c_UUID 2850686364 2306921556 398957656 80% /scratch1[OST:28] > lustre-OST001d_UUID 2850686364 2277275716 428541804 79% /scratch1[OST:29] > lustre-OST001e_UUID 2850686364 2290738376 415141384 80% /scratch1[OST:30] > lustre-OST001f_UUID 2850686364 2358853432 347025588 82% /scratch1[OST:31] > lustre-OST0020_UUID 2850686364 2164837628 541041392 75% /scratch1[OST:32] > lustre-OST0021_UUID 2850686364 2250094264 455757108 78% /scratch1[OST:33] > lustre-OST0022_UUID 2850686364 2318377076 387474296 81% /scratch1[OST:34] > lustre-OST0023_UUID 2850686364 2287996640 417827084 80% /scratch1[OST:35] > lustre-OST0024_UUID 2850686364 2266655604 439213268 79% /scratch1[OST:36] > lustre-OST0025_UUID 2850686364 2196015760 509804940 77% /scratch1[OST:37] > lustre-OST0026_UUID 2850686364 2326710900 379141252 81% /scratch1[OST:38] > lustre-OST0027_UUID 2850686364 2263182244 442696788 79% /scratch1[OST:39] > lustre-OST0028_UUID 2850686364 2281411520 424466108 80% /scratch1[OST:40] > lustre-OST0029_UUID 2850686364 2253024788 452854156 79% /scratch1[OST:41] > lustre-OST002a_UUID 2850686364 2265263264 440614844 79% /scratch1[OST:42] > lustre-OST002b_UUID 2850686364 2358109672 347769300 82% /scratch1[OST:43] > lustre-OST002c_UUID 2850686364 2246612120 459253300 78% /scratch1[OST:44] > lustre-OST002d_UUID 2850686364 2315081464 390671376 81% /scratch1[OST:45] > lustre-OST002e_UUID 2850686364 2151456488 554423308 75% /scratch1[OST:46] > lustre-OST002f_UUID 2850686364 2276389044 429401272 79% /scratch1[OST:47] > > filesystem summary: 136832945472 109819046792 20061966364 80% /scratch1 > mark at node1083:~> lfs df -i > UUID Inodes IUsed IFree IUse% Mounted on > lustre-MDT0000_UUID 623065492 13350254 609715238 2% /scratch1[MDT:0] > lustre-OST0000_UUID 147951373 356246 147595127 0% /scratch1[OST:0] > lustre-OST0001_UUID 122874520 348299 122526221 0% /scratch1[OST:1] > lustre-OST0002_UUID 137525462 310049 137215413 0% /scratch1[OST:2] > lustre-OST0003_UUID 138550809 340738 138210071 0% /scratch1[OST:3] > lustre-OST0004_UUID 145867558 363872 145503686 0% /scratch1[OST:4] > lustre-OST0005_UUID 124846336 353476 124492860 0% /scratch1[OST:5] > lustre-OST0006_UUID 148769303 334806 148434497 0% /scratch1[OST:6] > lustre-OST0007_UUID 153043111 338572 152704539 0% /scratch1[OST:7] > lustre-OST0008_UUID 148836688 343526 148493162 0% /scratch1[OST:8] > lustre-OST0009_UUID 133406354 332875 133073479 0% /scratch1[OST:9] > lustre-OST000a_UUID 151428889 318410 151110479 0% /scratch1[OST:10] > lustre-OST000b_UUID 140643612 370513 140273099 0% /scratch1[OST:11] > lustre-OST000c_UUID 146024187 317395 145706792 0% /scratch1[OST:12] > lustre-OST000d_UUID 120508125 334975 120173150 0% /scratch1[OST:13] > lustre-OST000e_UUID 119591403 335426 119255977 0% /scratch1[OST:14] > lustre-OST000f_UUID 145755615 354062 145401553 0% /scratch1[OST:15] > lustre-OST0010_UUID 149476374 334318 149142056 0% /scratch1[OST:16] > lustre-OST0011_UUID 136596942 304167 136292775 0% /scratch1[OST:17] > lustre-OST0012_UUID 140548883 326959 140221924 0% /scratch1[OST:18] > lustre-OST0013_UUID 150984278 329164 150655114 0% /scratch1[OST:19] > lustre-OST0014_UUID 125850214 335954 125514260 0% /scratch1[OST:20] > lustre-OST0015_UUID 128803314 340503 128462811 0% /scratch1[OST:21] > lustre-OST0016_UUID 141332100 356012 140976088 0% /scratch1[OST:22] > lustre-OST0017_UUID 148282098 332665 147949433 0% /scratch1[OST:23] > lustre-OST0018_UUID 114305646 341206 113964440 0% /scratch1[OST:24] > lustre-OST0019_UUID 134162367 320108 133842259 0% /scratch1[OST:25] > lustre-OST001a_UUID 137846405 346482 137499923 0% /scratch1[OST:26] > lustre-OST001b_UUID 138954376 354556 138599820 0% /scratch1[OST:27] > lustre-OST001c_UUID 136261269 314506 135946763 0% /scratch1[OST:28] > lustre-OST001d_UUID 143665519 346163 143319356 0% /scratch1[OST:29] > lustre-OST001e_UUID 140292861 310260 139982601 0% /scratch1[OST:30] > lustre-OST001f_UUID 123343915 374877 122969038 0% /scratch1[OST:31] > lustre-OST0020_UUID 171817157 349887 171467270 0% /scratch1[OST:32] > lustre-OST0021_UUID 150499898 352977 150146921 0% /scratch1[OST:33] > lustre-OST0022_UUID 133410450 335784 133074666 0% /scratch1[OST:34] > lustre-OST0023_UUID 140965613 325669 140639944 0% /scratch1[OST:35] > lustre-OST0024_UUID 146302421 339776 145962645 0% /scratch1[OST:36] > lustre-OST0025_UUID 163987206 343710 163643496 0% /scratch1[OST:37] > lustre-OST0026_UUID 131322985 361117 130961868 0% /scratch1[OST:38] > lustre-OST0027_UUID 147155706 296609 146859097 0% /scratch1[OST:39] > lustre-OST0028_UUID 142674349 344009 142330340 0% /scratch1[OST:40] > lustre-OST0029_UUID 149723181 332111 149391070 0% /scratch1[OST:41] > lustre-OST002a_UUID 146671256 320469 146350787 0% /scratch1[OST:42] > lustre-OST002b_UUID 123435148 350587 123084561 0% /scratch1[OST:43] > lustre-OST002c_UUID 151317772 343224 150974548 0% /scratch1[OST:44] > lustre-OST002d_UUID 134212577 353558 133859019 0% /scratch1[OST:45] > lustre-OST002e_UUID 175115293 326239 174789054 0% /scratch1[OST:46] > lustre-OST002f_UUID 143864174 326506 143537668 0% /scratch1[OST:47] > > filesystem summary: 623065492 13350254 609715238 2% /scratch1 -- Mark Nelson, HPC Systems Administrator Minnesota Supercomputing Institute Phone: (612)626-4479 Email: mark at msi.umn.edu From adilger at whamcloud.com Thu Apr 21 22:56:27 2011 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 21 Apr 2011 16:56:27 -0600 Subject: [Lustre-devel] 64-bit ext4 inodes In-Reply-To: <73AED5C780AE05478241DB067651A92102378476@XYUS-EX22.xyus.xyratex.com> References: <2368DAF2-4CA9-41E4-8A56-E48AEA9D4549@xyratex.com> <73AED5C780AE05478241DB067651A92102378476@XYUS-EX22.xyus.xyratex.com> Message-ID: <50A7CDE4-73B3-4BEB-9BF4-BE0D077A3A00@whamcloud.com> On Apr 20, 2011, at 11:07 AM, Nathan Rutman wrote: > We're about to start looking into 64-bit inodes on ext4 -- anybody else working on this at the moment? I don't know of anyone other than Fujitsu that are looking at this. I haven't had any kind of technical discussion with them about it, nor looked at their code, but just implied that this is what they did based on their slides. One important consideration to note with 64-bit inode numbers is that they cannot be used safely on 1.8 MDS filesystems. That is because the IGIF FID namespace only has room for 2^32 inode numbers (mapped into the 2.x FID SEQ field, see ), so upgrading a 1.8 MDS with 64-bit inode numbers to 2.x would cause a huge world of hurt. If this is limited to 2.x filesystems that only identify MDS inodes via FIDs to the clients then this is not a concern. If you are concerned with being able to upgrade 32-bit inode filesystems into 64-bit inode filesystems, you should look at the data-in-dirent patch that is currently in the 2.x ldiskfs patchset. It was developed to allow storing the 128-bit Lustre FID in the directory entry in a compatible manner, but was also designed to allow storing the high 32 bits of a 64-bit inode number into the directory entry. That allows compatibility for upgrades by avoiding the need to atomically upgrade a whole directory full of dirents to have 64-bit inode number fields. It also avoids the need to store 64-bit inode numbers when most of them are 32-bit values. I'm not against the idea of exploring this approach, but I am concerned that e2fsck time will continue to grow with the number of inodes in a single filesystem, and scalability of a single MDT will increasingly be an issue. AFAIK, e2fsck times are currently on the order of 1h/100M inodes, and no Lustre filesystems that I know of are above 500M inodes today just due to the average file size being so large and the long e2fsck times. With improvements in ext4 to reduce e2fsck time like flex_bg, and SSDs, it may be possible to reduce the e2fsck time/inode ratio noticeably, but I think it would take more effort on the e2fsck side than the ext4 side to make many billions of inodes in a single MDT a practical approach. Things like metadata prefetching that Dave Dillow was doing for e2scan with event-driven completion handlers that process the blocks whenever they arrive from the disk, and multi-threading some of the passes of e2fsck with an understanding of the underlying disk layout (using the s_raid_stride and s_raid_stripe_width). Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. From Nathan_Rutman at xyratex.com Thu Apr 21 23:36:11 2011 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Thu, 21 Apr 2011 16:36:11 -0700 Subject: [Lustre-devel] 64-bit ext4 inodes In-Reply-To: <50A7CDE4-73B3-4BEB-9BF4-BE0D077A3A00@whamcloud.com> References: <2368DAF2-4CA9-41E4-8A56-E48AEA9D4549@xyratex.com> <73AED5C780AE05478241DB067651A92102378476@XYUS-EX22.xyus.xyratex.com> <50A7CDE4-73B3-4BEB-9BF4-BE0D077A3A00@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102378479@XYUS-EX22.xyus.xyratex.com> On Apr 21, 2011, at 3:56 PM, Andreas Dilger wrote: > On Apr 20, 2011, at 11:07 AM, Nathan Rutman wrote: >> We're about to start looking into 64-bit inodes on ext4 -- anybody else working on this at the moment? > > I don't know of anyone other than Fujitsu that are looking at this. I haven't had any kind of technical discussion with them about it, nor looked at their code, but just implied that this is what they did based on their slides. Ah right, thanks. > > One important consideration to note with 64-bit inode numbers is that they cannot be used safely on 1.8 MDS filesystems. That is because the IGIF FID namespace only has room for 2^32 inode numbers (mapped into the 2.x FID SEQ field, see ), so upgrading a 1.8 MDS with 64-bit inode numbers to 2.x would cause a huge world of hurt. If this is limited to 2.x filesystems that only identify MDS inodes via FIDs to the clients then this is not a concern. This would be for new systems; upgrading is not a concern. > > If you are concerned with being able to upgrade 32-bit inode filesystems into 64-bit inode filesystems, you should look at the data-in-dirent patch that is currently in the 2.x ldiskfs patchset. It was developed to allow storing the 128-bit Lustre FID in the directory entry in a compatible manner, but was also designed to allow storing the high 32 bits of a 64-bit inode number into the directory entry. That allows compatibility for upgrades by avoiding the need to atomically upgrade a whole directory full of dirents to have 64-bit inode number fields. It also avoids the need to store 64-bit inode numbers when most of them are 32-bit values. > > > I'm not against the idea of exploring this approach, but I am concerned that e2fsck time will continue to grow with the number of inodes in a single filesystem, and scalability of a single MDT will increasingly be an issue. > > AFAIK, e2fsck times are currently on the order of 1h/100M inodes, and no Lustre filesystems that I know of are above 500M inodes today just due to the average file size being so large and the long e2fsck times. With improvements in ext4 to reduce e2fsck time like flex_bg, and SSDs, it may be possible to reduce the e2fsck time/inode ratio noticeably, but I think it would take more effort on the e2fsck side than the ext4 side to make many billions of inodes in a single MDT a practical approach. With an on-line, continuous filesystem check (or other solutions) this should no longer be limiting :) In any case, fsck time is a problem that needs to be solved, for 32-bit or 64-bit inodes. > > Things like metadata prefetching that Dave Dillow was doing for e2scan with event-driven completion handlers that process the blocks whenever they arrive from the disk, and multi-threading some of the passes of e2fsck with an understanding of the underlying disk layout (using the s_raid_stride and s_raid_stripe_width). > > Cheers, Andreas > -- > Andreas Dilger > Principal Engineer > Whamcloud, Inc. > > > ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________ From Evan.Felix at pnl.gov Fri Apr 22 14:55:44 2011 From: Evan.Felix at pnl.gov (Felix, Evan J) Date: Fri, 22 Apr 2011 07:55:44 -0700 Subject: [Lustre-devel] Research on filesystem metadata operation distribution In-Reply-To: References: Message-ID: <3ED20A6EAD4D954BB360447C97BF811B01036254506B@EMAIL05.pnl.gov> Info on our current large cluster: /mscf on Chinook: 20TB - mostly home directories and application binaries/libraries. open 4554527798 samples [reqs] close 473872913 samples [reqs] link 681239 samples [reqs] unlink 24285832 samples [reqs] rename 3369767 samples [reqs] getxattr 3006968011 samples [reqs] setxattr 995589 samples [reqs] setattr 30371813 samples [reqs] getattr 2067961374 samples [reqs] /dtemp on Chinook: 270TB - user data for processing and output. open 946342778 samples [reqs] close 136642017 samples [reqs] link 1601 samples [reqs] unlink 18105075 samples [reqs] rename 6232018 samples [reqs] getxattr 81475 samples [reqs] setxattr 131891 samples [reqs] setattr 34112680 samples [reqs] getattr 1095110611 samples [reqs] df info: UUID 1K-blocks Used Available Use% Mounted on sfs-MDT0000_UUID 1834869856 4327988 1725684272 0% /dtemp[MDT:0] sfs-OST0000_UUID 2064245920 1068818228 984870244 51% /dtemp[OST:0] sfs-OST0001_UUID 2064245920 1147446156 906245256 55% /dtemp[OST:1] sfs-OST0002_UUID 2064245920 1113427560 940268856 53% /dtemp[OST:2] sfs-OST0003_UUID 2064245920 1159010028 894677244 56% /dtemp[OST:3] sfs-OST0004_UUID 2064245920 1147221340 906467384 55% /dtemp[OST:4] sfs-OST0005_UUID 2064245920 1095167780 958527096 53% /dtemp[OST:5] sfs-OST0006_UUID 2064245920 1063356308 990322272 51% /dtemp[OST:6] sfs-OST0007_UUID 2064245920 1064588100 989099968 51% /dtemp[OST:7] sfs-OST0008_UUID 2064245920 1190708224 862988936 57% /dtemp[OST:8] sfs-OST0009_UUID 2064245920 1112665488 941024380 53% /dtemp[OST:9] sfs-OST000a_UUID 2064245920 945953764 1107727176 45% /dtemp[OST:10] sfs-OST000b_UUID 2064245920 1224351324 829342584 59% /dtemp[OST:11] sfs-OST000c_UUID 2064245920 1080636532 973049116 52% /dtemp[OST:12] sfs-OST000d_UUID 2064245920 1017028208 1036655292 49% /dtemp[OST:13] sfs-OST000e_UUID 2064245920 1028559680 1025142148 49% /dtemp[OST:14] sfs-OST000f_UUID 2064245920 1022769156 1030897212 49% /dtemp[OST:15] sfs-OST0010_UUID 2064245920 1100019724 953687264 53% /dtemp[OST:16] sfs-OST0011_UUID 2064245920 1052428920 1001264028 50% /dtemp[OST:17] sfs-OST0012_UUID 2064245920 1013339316 1040330552 49% /dtemp[OST:18] sfs-OST0013_UUID 2064245920 1080871180 972806700 52% /dtemp[OST:19] sfs-OST0014_UUID 2064245920 987840500 1065852152 47% /dtemp[OST:20] sfs-OST0015_UUID 2064245920 1103825556 949866500 53% /dtemp[OST:21] sfs-OST0016_UUID 2064245920 1092751864 960926824 52% /dtemp[OST:22] sfs-OST0017_UUID 2064245920 1076291320 977379232 52% /dtemp[OST:23] sfs-OST0018_UUID 2064245920 1138079040 915597952 55% /dtemp[OST:24] sfs-OST0019_UUID 2064245920 1135952920 917739652 55% /dtemp[OST:25] sfs-OST001a_UUID 2064245920 1205589392 848111684 58% /dtemp[OST:26] sfs-OST001b_UUID 2064245920 1073418548 980271440 52% /dtemp[OST:27] sfs-OST001c_UUID 2064245920 1011666464 1042017224 49% /dtemp[OST:28] sfs-OST001d_UUID 2064245920 952649808 1101040760 46% /dtemp[OST:29] sfs-OST001e_UUID 2064245920 976172548 1077586588 47% /dtemp[OST:30] sfs-OST001f_UUID 2064245920 986284680 1067396376 47% /dtemp[OST:31] sfs-OST0020_UUID 2064245920 1154685332 899011516 55% /dtemp[OST:32] sfs-OST0021_UUID 2064245920 1145045268 908638940 55% /dtemp[OST:33] sfs-OST0022_UUID 2064245920 1002669208 1051017476 48% /dtemp[OST:34] sfs-OST0023_UUID 2064245920 1196066908 857608280 57% /dtemp[OST:35] sfs-OST0024_UUID 2064245920 1099025172 954671316 53% /dtemp[OST:36] sfs-OST0025_UUID 2064245920 1056398932 997293372 51% /dtemp[OST:37] sfs-OST0026_UUID 2064245920 1178277020 875422164 57% /dtemp[OST:38] sfs-OST0027_UUID 2064245920 1204857620 848839936 58% /dtemp[OST:39] sfs-OST0028_UUID 2064245920 1040356188 1013403316 50% /dtemp[OST:40] sfs-OST0029_UUID 2064245920 1217397640 836297196 58% /dtemp[OST:41] sfs-OST002a_UUID 2064245920 1126382584 927322856 54% /dtemp[OST:42] sfs-OST002b_UUID 2064245920 1156904268 896788008 56% /dtemp[OST:43] sfs-OST002c_UUID 2064245920 1175057720 878646580 56% /dtemp[OST:44] sfs-OST002d_UUID 2064245920 1203375028 850316144 58% /dtemp[OST:45] sfs-OST002e_UUID 2064245920 1112189444 941503504 53% /dtemp[OST:46] sfs-OST002f_UUID 2064245920 1087356908 966328428 52% /dtemp[OST:47] sfs-OST0030_UUID 2064245920 1211991352 841713660 58% /dtemp[OST:48] sfs-OST0031_UUID 2064245920 990200212 969120668 47% /dtemp[OST:49] sfs-OST0032_UUID 2064245920 1069921844 889408180 51% /dtemp[OST:50] sfs-OST0033_UUID 2064245920 1009058432 950276708 48% /dtemp[OST:51] sfs-OST0034_UUID 2064245920 1165364648 793969240 56% /dtemp[OST:52] sfs-OST0035_UUID 2064245920 1205991652 847704532 58% /dtemp[OST:53] sfs-OST0036_UUID 2064245920 1013856480 1039833748 49% /dtemp[OST:54] sfs-OST0037_UUID 2064245920 1071568160 982125316 51% /dtemp[OST:55] sfs-OST0038_UUID 2064245920 1067675792 986000280 51% /dtemp[OST:56] sfs-OST0039_UUID 2064245920 1053890700 999813836 51% /dtemp[OST:57] sfs-OST003a_UUID 2064245920 1135658592 918042728 55% /dtemp[OST:58] sfs-OST003b_UUID 2064245920 1218140260 835564952 59% /dtemp[OST:59] sfs-OST003c_UUID 2064245920 1035919232 1017757280 50% /dtemp[OST:60] sfs-OST003d_UUID 2064245920 1016449632 1037243904 49% /dtemp[OST:61] sfs-OST003e_UUID 2064245920 1128825020 924881152 54% /dtemp[OST:62] sfs-OST003f_UUID 2064245920 1107880660 945794748 53% /dtemp[OST:63] sfs-OST0040_UUID 2064245920 1185860764 867836796 57% /dtemp[OST:64] sfs-OST0041_UUID 2064245920 1080292868 973412016 52% /dtemp[OST:65] sfs-OST0042_UUID 2064245920 1112667788 941016688 53% /dtemp[OST:66] sfs-OST0043_UUID 2064245920 1104678368 949023060 53% /dtemp[OST:67] sfs-OST0044_UUID 2064245920 1106512800 947174584 53% /dtemp[OST:68] sfs-OST0045_UUID 2064245920 1075918920 977765432 52% /dtemp[OST:69] sfs-OST0046_UUID 2064245920 1180045252 873641736 57% /dtemp[OST:70] sfs-OST0047_UUID 2064245920 1116326384 937355352 54% /dtemp[OST:71] sfs-OST0048_UUID 2064245920 1206908764 846776016 58% /dtemp[OST:72] sfs-OST0049_UUID 2064245920 1080584572 973118472 52% /dtemp[OST:73] sfs-OST004a_UUID 2064245920 1134900624 918780276 54% /dtemp[OST:74] sfs-OST004b_UUID 2064245920 1051051796 1002653240 50% /dtemp[OST:75] sfs-OST004c_UUID 2064245920 999055640 1054628328 48% /dtemp[OST:76] sfs-OST004d_UUID 2064245920 1039090536 1014603492 50% /dtemp[OST:77] sfs-OST004e_UUID 2064245920 1096989904 956687368 53% /dtemp[OST:78] sfs-OST004f_UUID 2064245920 1283380656 770315848 62% /dtemp[OST:79] sfs-OST0050_UUID 2064245920 1076040124 977642024 52% /dtemp[OST:80] sfs-OST0051_UUID 2064245920 984345000 1069336180 47% /dtemp[OST:81] sfs-OST0052_UUID 2064245920 1013599600 1040091552 49% /dtemp[OST:82] sfs-OST0053_UUID 2064245920 1160257412 893434280 56% /dtemp[OST:83] sfs-OST0054_UUID 2064245920 1155219336 898480408 55% /dtemp[OST:84] sfs-OST0055_UUID 2064245920 1123087368 930604308 54% /dtemp[OST:85] sfs-OST0056_UUID 2064245920 1183220488 870473776 57% /dtemp[OST:86] sfs-OST0057_UUID 2064245920 1017526048 1036152256 49% /dtemp[OST:87] sfs-OST0058_UUID 2064245920 1154269816 899422772 55% /dtemp[OST:88] sfs-OST0059_UUID 2064245920 1195696256 858001948 57% /dtemp[OST:89] sfs-OST005a_UUID 2064245920 1193575944 860118532 57% /dtemp[OST:90] sfs-OST005b_UUID 2064245920 1085868548 967818624 52% /dtemp[OST:91] sfs-OST005c_UUID 2064245920 1044960288 1008718892 50% /dtemp[OST:92] sfs-OST005d_UUID 2064245920 1082544472 971135872 52% /dtemp[OST:93] sfs-OST005e_UUID 2064245920 1075745924 977952724 52% /dtemp[OST:94] sfs-OST005f_UUID 2064245920 1054877620 998818084 51% /dtemp[OST:95] sfs-OST0060_UUID 2064245920 1118777916 934912992 54% /dtemp[OST:96] sfs-OST0061_UUID 2064245920 1067545756 986139120 51% /dtemp[OST:97] sfs-OST0062_UUID 2064245920 1023269860 1030489276 49% /dtemp[OST:98] sfs-OST0063_UUID 2064245920 1280644316 773067836 62% /dtemp[OST:99] sfs-OST0064_UUID 2064245920 1097805864 955887200 53% /dtemp[OST:100] sfs-OST0065_UUID 2064245920 1021619552 1032058604 49% /dtemp[OST:101] sfs-OST0066_UUID 2064245920 1057169164 996534968 51% /dtemp[OST:102] sfs-OST0067_UUID 2064245920 1146992012 906681344 55% /dtemp[OST:103] sfs-OST0068_UUID 2064245920 1026243604 1027447080 49% /dtemp[OST:104] sfs-OST0069_UUID 2064245920 1004975972 1048712764 48% /dtemp[OST:105] sfs-OST006a_UUID 2064245920 1084540164 969157304 52% /dtemp[OST:106] sfs-OST006b_UUID 2064245920 1196168044 857520712 57% /dtemp[OST:107] sfs-OST006c_UUID 2064245920 1061778752 991903068 51% /dtemp[OST:108] sfs-OST006d_UUID 2064245920 1027008356 1026677916 49% /dtemp[OST:109] sfs-OST006e_UUID 2064245920 1039766492 1013915812 50% /dtemp[OST:110] sfs-OST006f_UUID 2064245920 1145531072 908160924 55% /dtemp[OST:111] sfs-OST0070_UUID 2064245920 1073432604 980256548 52% /dtemp[OST:112] sfs-OST0071_UUID 2064245920 1040646388 1013052688 50% /dtemp[OST:113] sfs-OST0072_UUID 2064245920 1080501192 973182680 52% /dtemp[OST:114] sfs-OST0073_UUID 2064245920 1130126556 923578552 54% /dtemp[OST:115] sfs-OST0074_UUID 2064245920 1219314512 834380732 59% /dtemp[OST:116] sfs-OST0075_UUID 2064245920 1099077204 954603912 53% /dtemp[OST:117] sfs-OST0076_UUID 2064245920 1172247528 881433476 56% /dtemp[OST:118] sfs-OST0077_UUID 2064245920 1029514300 1024132536 49% /dtemp[OST:119] sfs-OST0078_UUID 2064245920 1091720628 961961084 52% /dtemp[OST:120] sfs-OST0079_UUID 2064245920 1108579792 945112504 53% /dtemp[OST:121] sfs-OST007a_UUID 2064245920 996134068 1057564556 48% /dtemp[OST:122] sfs-OST007b_UUID 2064245920 1077044384 976639468 52% /dtemp[OST:123] sfs-OST007c_UUID 2064245920 1145309944 908385504 55% /dtemp[OST:124] sfs-OST007d_UUID 2064245920 1073612880 980068604 52% /dtemp[OST:125] sfs-OST007e_UUID 2064245920 1075483592 978201364 52% /dtemp[OST:126] sfs-OST007f_UUID 2064245920 1120388952 933310856 54% /dtemp[OST:127] sfs-OST0080_UUID 2064245920 1156587688 897128456 56% /dtemp[OST:128] sfs-OST0081_UUID 2064245920 1030957312 1022709824 49% /dtemp[OST:129] sfs-OST0082_UUID 2064245920 999881344 1053790940 48% /dtemp[OST:130] sfs-OST0083_UUID 2064245920 1095521408 958174240 53% /dtemp[OST:131] sfs-OST0084_UUID 2064245920 1221129792 832576716 59% /dtemp[OST:132] sfs-OST0085_UUID 2064245920 994100256 1059584780 48% /dtemp[OST:133] sfs-OST0086_UUID 2064245920 1160580632 893117576 56% /dtemp[OST:134] sfs-OST0087_UUID 2064245920 1192673900 861027648 57% /dtemp[OST:135] sfs-OST0088_UUID 2064245920 1061730172 991947440 51% /dtemp[OST:136] sfs-OST0089_UUID 2064245920 1163157924 890531400 56% /dtemp[OST:137] sfs-OST008a_UUID 2064245920 1018606752 1035074608 49% /dtemp[OST:138] sfs-OST008b_UUID 2064245920 1135650388 918033196 55% /dtemp[OST:139] sfs-OST008c_UUID 2064245920 1064232464 989443304 51% /dtemp[OST:140] sfs-OST008d_UUID 2064245920 1157718560 895964732 56% /dtemp[OST:141] sfs-OST008e_UUID 2064245920 1099315156 954369152 53% /dtemp[OST:142] sfs-OST008f_UUID 2064245920 1021270580 1032433420 49% /dtemp[OST:143] filesystem summary: 297251412480 158088513192 137265570408 53% /dtemp UUID 1K-blocks Used Available Use% Mounted on vader-mds1_UUID 1834869856 13282676 1716729584 0% /mscf[MDT:0] vader-ost1_UUID 1334535272 1068848892 252068648 80% /mscf[OST:0] vader-ost2_UUID 1334535272 1103266920 217654380 82% /mscf[OST:1] vader-ost3_UUID 1334535272 1042379244 278463360 78% /mscf[OST:2] vader-ost4_UUID 1334535272 1043839320 277043180 78% /mscf[OST:3] vader-ost5_UUID 1334535272 1069106456 251805632 80% /mscf[OST:4] vader-ost6_UUID 1334535272 1063143776 257762956 79% /mscf[OST:5] vader-ost7_UUID 1334535272 1076778096 244094912 80% /mscf[OST:6] vader-ost8_UUID 1334535272 1032947836 287954436 77% /mscf[OST:7] vader-ost9_UUID 1334535272 1087367008 233610108 81% /mscf[OST:8] vader-ost10_UUID 1334535272 1095883032 225037608 82% /mscf[OST:9] vader-ost11_UUID 1334535272 1038984844 281938072 77% /mscf[OST:10] vader-ost12_UUID 1334535272 1080016528 240874192 80% /mscf[OST:11] vader-ost13_UUID 1334535272 1023102820 297807716 76% /mscf[OST:12] vader-ost14_UUID 1334535272 979408332 341486352 73% /mscf[OST:13] vader-ost15_UUID 1334535272 1099037168 221864212 82% /mscf[OST:14] vader-ost16_UUID 1334535272 1045873940 275045200 78% /mscf[OST:15] filesystem summary: 21352564352 16949984212 4184510964 79% /mscf UUID Inodes IUsed IFree IUse% Mounted on sfs-MDT0000_UUID 474870531 17235068 457635463 3% /dtemp[MDT:0] sfs-OST0000_UUID 131072000 933107 130138893 0% /dtemp[OST:0] sfs-OST0001_UUID 131072000 909959 130162041 0% /dtemp[OST:1] sfs-OST0002_UUID 131072000 863249 130208751 0% /dtemp[OST:2] sfs-OST0003_UUID 131072000 802817 130269183 0% /dtemp[OST:3] sfs-OST0004_UUID 131072000 1005695 130066305 0% /dtemp[OST:4] sfs-OST0005_UUID 131072000 816126 130255874 0% /dtemp[OST:5] sfs-OST0006_UUID 131072000 820701 130251299 0% /dtemp[OST:6] sfs-OST0007_UUID 131072000 945117 130126883 0% /dtemp[OST:7] sfs-OST0008_UUID 131072000 988375 130083625 0% /dtemp[OST:8] sfs-OST0009_UUID 131072000 962663 130109337 0% /dtemp[OST:9] sfs-OST000a_UUID 131072000 1004796 130067204 0% /dtemp[OST:10] sfs-OST000b_UUID 131072000 908056 130163944 0% /dtemp[OST:11] sfs-OST000c_UUID 131072000 977148 130094852 0% /dtemp[OST:12] sfs-OST000d_UUID 131072000 963137 130108863 0% /dtemp[OST:13] sfs-OST000e_UUID 131072000 788005 130283995 0% /dtemp[OST:14] sfs-OST000f_UUID 131072000 716657 130355343 0% /dtemp[OST:15] sfs-OST0010_UUID 131072000 997047 130074953 0% /dtemp[OST:16] sfs-OST0011_UUID 131072000 996429 130075571 0% /dtemp[OST:17] sfs-OST0012_UUID 131072000 925896 130146104 0% /dtemp[OST:18] sfs-OST0013_UUID 131072000 974211 130097789 0% /dtemp[OST:19] sfs-OST0014_UUID 131072000 983833 130088167 0% /dtemp[OST:20] sfs-OST0015_UUID 131072000 921903 130150097 0% /dtemp[OST:21] sfs-OST0016_UUID 131072000 938759 130133241 0% /dtemp[OST:22] sfs-OST0017_UUID 131072000 778331 130293669 0% /dtemp[OST:23] sfs-OST0018_UUID 131072000 940254 130131746 0% /dtemp[OST:24] sfs-OST0019_UUID 131072000 791892 130280108 0% /dtemp[OST:25] sfs-OST001a_UUID 131072000 981569 130090431 0% /dtemp[OST:26] sfs-OST001b_UUID 131072000 953077 130118923 0% /dtemp[OST:27] sfs-OST001c_UUID 131072000 891157 130180843 0% /dtemp[OST:28] sfs-OST001d_UUID 131072000 976900 130095100 0% /dtemp[OST:29] sfs-OST001e_UUID 131072000 1040702 130031298 0% /dtemp[OST:30] sfs-OST001f_UUID 131072000 976216 130095784 0% /dtemp[OST:31] sfs-OST0020_UUID 131072000 912421 130159579 0% /dtemp[OST:32] sfs-OST0021_UUID 131072000 927803 130144197 0% /dtemp[OST:33] sfs-OST0022_UUID 131072000 945489 130126511 0% /dtemp[OST:34] sfs-OST0023_UUID 131072000 929574 130142426 0% /dtemp[OST:35] sfs-OST0024_UUID 131072000 904851 130167149 0% /dtemp[OST:36] sfs-OST0025_UUID 131072000 947878 130124122 0% /dtemp[OST:37] sfs-OST0026_UUID 131072000 954019 130117981 0% /dtemp[OST:38] sfs-OST0027_UUID 131072000 777785 130294215 0% /dtemp[OST:39] sfs-OST0028_UUID 131072000 924282 130147718 0% /dtemp[OST:40] sfs-OST0029_UUID 131072000 921669 130150331 0% /dtemp[OST:41] sfs-OST002a_UUID 131072000 956530 130115470 0% /dtemp[OST:42] sfs-OST002b_UUID 131072000 1000389 130071611 0% /dtemp[OST:43] sfs-OST002c_UUID 131072000 797964 130274036 0% /dtemp[OST:44] sfs-OST002d_UUID 131072000 986655 130085345 0% /dtemp[OST:45] sfs-OST002e_UUID 131072000 887212 130184788 0% /dtemp[OST:46] sfs-OST002f_UUID 131072000 997138 130074862 0% /dtemp[OST:47] sfs-OST0030_UUID 131072000 976089 130095911 0% /dtemp[OST:48] sfs-OST0031_UUID 131072000 898961 130173039 0% /dtemp[OST:49] sfs-OST0032_UUID 131072000 930610 130141390 0% /dtemp[OST:50] sfs-OST0033_UUID 131072000 977095 130094905 0% /dtemp[OST:51] sfs-OST0034_UUID 131072000 993845 130078155 0% /dtemp[OST:52] sfs-OST0035_UUID 131072000 911942 130160058 0% /dtemp[OST:53] sfs-OST0036_UUID 131072000 924638 130147362 0% /dtemp[OST:54] sfs-OST0037_UUID 131072000 958901 130113099 0% /dtemp[OST:55] sfs-OST0038_UUID 131072000 979253 130092747 0% /dtemp[OST:56] sfs-OST0039_UUID 131072000 980803 130091197 0% /dtemp[OST:57] sfs-OST003a_UUID 131072000 791658 130280342 0% /dtemp[OST:58] sfs-OST003b_UUID 131072000 911553 130160447 0% /dtemp[OST:59] sfs-OST003c_UUID 131072000 854410 130217590 0% /dtemp[OST:60] sfs-OST003d_UUID 131072000 966392 130105608 0% /dtemp[OST:61] sfs-OST003e_UUID 131072000 972723 130099277 0% /dtemp[OST:62] sfs-OST003f_UUID 131072000 973009 130098991 0% /dtemp[OST:63] sfs-OST0040_UUID 131072000 900419 130171581 0% /dtemp[OST:64] sfs-OST0041_UUID 131072000 899906 130172094 0% /dtemp[OST:65] sfs-OST0042_UUID 131072000 949489 130122511 0% /dtemp[OST:66] sfs-OST0043_UUID 131072000 992597 130079403 0% /dtemp[OST:67] sfs-OST0044_UUID 131072000 1001642 130070358 0% /dtemp[OST:68] sfs-OST0045_UUID 131072000 951350 130120650 0% /dtemp[OST:69] sfs-OST0046_UUID 131072000 956321 130115679 0% /dtemp[OST:70] sfs-OST0047_UUID 131072000 732851 130339149 0% /dtemp[OST:71] sfs-OST0048_UUID 131072000 927208 130144792 0% /dtemp[OST:72] sfs-OST0049_UUID 131072000 914598 130157402 0% /dtemp[OST:73] sfs-OST004a_UUID 131072000 903386 130168614 0% /dtemp[OST:74] sfs-OST004b_UUID 131072000 938516 130133484 0% /dtemp[OST:75] sfs-OST004c_UUID 131072000 985172 130086828 0% /dtemp[OST:76] sfs-OST004d_UUID 131072000 956051 130115949 0% /dtemp[OST:77] sfs-OST004e_UUID 131072000 946741 130125259 0% /dtemp[OST:78] sfs-OST004f_UUID 131072000 897018 130174982 0% /dtemp[OST:79] sfs-OST0050_UUID 131072000 854290 130217710 0% /dtemp[OST:80] sfs-OST0051_UUID 131072000 958563 130113437 0% /dtemp[OST:81] sfs-OST0052_UUID 131072000 888145 130183855 0% /dtemp[OST:82] sfs-OST0053_UUID 131072000 798914 130273086 0% /dtemp[OST:83] sfs-OST0054_UUID 131072000 994307 130077693 0% /dtemp[OST:84] sfs-OST0055_UUID 131072000 876102 130195898 0% /dtemp[OST:85] sfs-OST0056_UUID 131072000 992537 130079463 0% /dtemp[OST:86] sfs-OST0057_UUID 131072000 1025381 130046619 0% /dtemp[OST:87] sfs-OST0058_UUID 131072000 883171 130188829 0% /dtemp[OST:88] sfs-OST0059_UUID 131072000 963811 130108189 0% /dtemp[OST:89] sfs-OST005a_UUID 131072000 956580 130115420 0% /dtemp[OST:90] sfs-OST005b_UUID 131072000 1018335 130053665 0% /dtemp[OST:91] sfs-OST005c_UUID 131072000 921394 130150606 0% /dtemp[OST:92] sfs-OST005d_UUID 131072000 997103 130074897 0% /dtemp[OST:93] sfs-OST005e_UUID 131072000 959395 130112605 0% /dtemp[OST:94] sfs-OST005f_UUID 131072000 957806 130114194 0% /dtemp[OST:95] sfs-OST0060_UUID 131072000 961372 130110628 0% /dtemp[OST:96] sfs-OST0061_UUID 131072000 876284 130195716 0% /dtemp[OST:97] sfs-OST0062_UUID 131072000 976349 130095651 0% /dtemp[OST:98] sfs-OST0063_UUID 131072000 927210 130144790 0% /dtemp[OST:99] sfs-OST0064_UUID 131072000 942963 130129037 0% /dtemp[OST:100] sfs-OST0065_UUID 131072000 927662 130144338 0% /dtemp[OST:101] sfs-OST0066_UUID 131072000 993961 130078039 0% /dtemp[OST:102] sfs-OST0067_UUID 131072000 952107 130119893 0% /dtemp[OST:103] sfs-OST0068_UUID 131072000 960114 130111886 0% /dtemp[OST:104] sfs-OST0069_UUID 131072000 1019083 130052917 0% /dtemp[OST:105] sfs-OST006a_UUID 131072000 1004608 130067392 0% /dtemp[OST:106] sfs-OST006b_UUID 131072000 891039 130180961 0% /dtemp[OST:107] sfs-OST006c_UUID 131072000 969055 130102945 0% /dtemp[OST:108] sfs-OST006d_UUID 131072000 963317 130108683 0% /dtemp[OST:109] sfs-OST006e_UUID 131072000 994995 130077005 0% /dtemp[OST:110] sfs-OST006f_UUID 131072000 887923 130184077 0% /dtemp[OST:111] sfs-OST0070_UUID 131072000 968551 130103449 0% /dtemp[OST:112] sfs-OST0071_UUID 131072000 947606 130124394 0% /dtemp[OST:113] sfs-OST0072_UUID 131072000 941221 130130779 0% /dtemp[OST:114] sfs-OST0073_UUID 131072000 912978 130159022 0% /dtemp[OST:115] sfs-OST0074_UUID 131072000 866475 130205525 0% /dtemp[OST:116] sfs-OST0075_UUID 131072000 943833 130128167 0% /dtemp[OST:117] sfs-OST0076_UUID 131072000 966207 130105793 0% /dtemp[OST:118] sfs-OST0077_UUID 131072000 925939 130146061 0% /dtemp[OST:119] sfs-OST0078_UUID 131072000 878610 130193390 0% /dtemp[OST:120] sfs-OST0079_UUID 131072000 972294 130099706 0% /dtemp[OST:121] sfs-OST007a_UUID 131072000 983109 130088891 0% /dtemp[OST:122] sfs-OST007b_UUID 131072000 992429 130079571 0% /dtemp[OST:123] sfs-OST007c_UUID 131072000 884171 130187829 0% /dtemp[OST:124] sfs-OST007d_UUID 131072000 984967 130087033 0% /dtemp[OST:125] sfs-OST007e_UUID 131072000 974043 130097957 0% /dtemp[OST:126] sfs-OST007f_UUID 131072000 904518 130167482 0% /dtemp[OST:127] sfs-OST0080_UUID 131072000 803345 130268655 0% /dtemp[OST:128] sfs-OST0081_UUID 131072000 988062 130083938 0% /dtemp[OST:129] sfs-OST0082_UUID 131072000 952675 130119325 0% /dtemp[OST:130] sfs-OST0083_UUID 131072000 910066 130161934 0% /dtemp[OST:131] sfs-OST0084_UUID 131072000 895140 130176860 0% /dtemp[OST:132] sfs-OST0085_UUID 131072000 1036804 130035196 0% /dtemp[OST:133] sfs-OST0086_UUID 131072000 892136 130179864 0% /dtemp[OST:134] sfs-OST0087_UUID 131072000 809896 130262104 0% /dtemp[OST:135] sfs-OST0088_UUID 131072000 786857 130285143 0% /dtemp[OST:136] sfs-OST0089_UUID 131072000 789836 130282164 0% /dtemp[OST:137] sfs-OST008a_UUID 131072000 1034379 130037621 0% /dtemp[OST:138] sfs-OST008b_UUID 131072000 887163 130184837 0% /dtemp[OST:139] sfs-OST008c_UUID 131072000 975873 130096127 0% /dtemp[OST:140] sfs-OST008d_UUID 131072000 921621 130150379 0% /dtemp[OST:141] sfs-OST008e_UUID 131072000 992654 130079346 0% /dtemp[OST:142] sfs-OST008f_UUID 131072000 936173 130135827 0% /dtemp[OST:143] filesystem summary: 474870531 17235068 457635463 3% /dtemp UUID Inodes IUsed IFree IUse% Mounted on vader-mds1_UUID 486297503 30900708 455396795 6% /mscf[MDT:0] vader-ost1_UUID 68167938 1746343 66421595 2% /mscf[OST:0] vader-ost2_UUID 59548367 1730686 57817681 2% /mscf[OST:1] vader-ost3_UUID 74872494 1833651 73038843 2% /mscf[OST:2] vader-ost4_UUID 74466464 1792689 72673775 2% /mscf[OST:3] vader-ost5_UUID 68137249 1780726 66356523 2% /mscf[OST:4] vader-ost6_UUID 69541748 1694569 67847179 2% /mscf[OST:5] vader-ost7_UUID 66083245 1644111 64439134 2% /mscf[OST:6] vader-ost8_UUID 77159492 1762644 75396848 2% /mscf[OST:7] vader-ost9_UUID 63563009 1770994 61792015 2% /mscf[OST:8] vader-ost10_UUID 61284780 1621740 59663040 2% /mscf[OST:9] vader-ost11_UUID 75684308 1796701 73887607 2% /mscf[OST:10] vader-ost12_UUID 65304005 1674597 63629408 2% /mscf[OST:11] vader-ost13_UUID 79549121 1690897 77858224 2% /mscf[OST:12] vader-ost14_UUID 84738048 1577666 83160382 1% /mscf[OST:13] vader-ost15_UUID 60569734 1695209 58874525 2% /mscf[OST:14] vader-ost16_UUID 74029395 1864062 72165333 2% /mscf[OST:15] filesystem summary: 486297503 30900708 455396795 6% /mscf -----Original Message----- From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Andreas Dilger Sent: Thursday, April 21, 2011 11:40 AM To: lustre-discuss at lists.lustre.org discuss; lustre-devel at lists.lustre.org Mailing List Subject: [Lustre-devel] Research on filesystem metadata operation distribution I'm trying to get some data about the relative distribution of MDS operations in the wild, and I'd be grateful if some people with production filesystems that have been running for at least a week could collect some simple stats and email them to me. They can be collected by any regular user on the MDS node: lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync" It would be useful to also include "lfs df" and "lfs df -i" information, as well as a brief description of what the filesystem is used for (scratch, home, project, archive, etc). As a reminder, I'm also interested if some Lustre admins could run the "fsstats" tool from http://www.pdsi-scidac.org/fsstats/ and send me the output. Sending the output to PDSI via their submission form may also produce some positive results. http://www.pdsi-scidac.org/fsstats/files/fsstats-1.4.5.tar.gz Thanks in advance for any data. I've set replies to go only to lustre-devel, to avoid clogging the larger readership of lustre-discuss, but it may be useful for others to have this in a list archive and/or searchable via Google in the future so I don't necessarily want to keep it all to myself. Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. _______________________________________________ Lustre-devel mailing list Lustre-devel at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-devel From marc at llnl.gov Fri Apr 22 15:13:37 2011 From: marc at llnl.gov (D. Marc Stearman) Date: Fri, 22 Apr 2011 08:13:37 -0700 Subject: [Lustre-devel] 64-bit ext4 inodes In-Reply-To: <50A7CDE4-73B3-4BEB-9BF4-BE0D077A3A00@whamcloud.com> References: <2368DAF2-4CA9-41E4-8A56-E48AEA9D4549@xyratex.com> <73AED5C780AE05478241DB067651A92102378476@XYUS-EX22.xyus.xyratex.com> <50A7CDE4-73B3-4BEB-9BF4-BE0D077A3A00@whamcloud.com> Message-ID: <5E5695A6-CE4E-4253-B97C-85FAD3C96FFF@llnl.gov> Andreas, LLNL often has file systems with more than 500M inodes in use. We create them with 1B inodes just for that reason. And yes, e2fsck times are horrible. Very painful. -Marc ---- D. Marc Stearman Lustre Operations Lead marc at llnl.gov 925.423.9670 Pager: 1.888.203.0641 On Apr 21, 2011, at 3:56 PM, Andreas Dilger wrote: > On Apr 20, 2011, at 11:07 AM, Nathan Rutman wrote: >> We're about to start looking into 64-bit inodes on ext4 -- anybody else working on this at the moment? > > I don't know of anyone other than Fujitsu that are looking at this. I haven't had any kind of technical discussion with them about it, nor looked at their code, but just implied that this is what they did based on their slides. > > One important consideration to note with 64-bit inode numbers is that they cannot be used safely on 1.8 MDS filesystems. That is because the IGIF FID namespace only has room for 2^32 inode numbers (mapped into the 2.x FID SEQ field, see ), so upgrading a 1.8 MDS with 64-bit inode numbers to 2.x would cause a huge world of hurt. If this is limited to 2.x filesystems that only identify MDS inodes via FIDs to the clients then this is not a concern. > > If you are concerned with being able to upgrade 32-bit inode filesystems into 64-bit inode filesystems, you should look at the data-in-dirent patch that is currently in the 2.x ldiskfs patchset. It was developed to allow storing the 128-bit Lustre FID in the directory entry in a compatible manner, but was also designed to allow storing the high 32 bits of a 64-bit inode number into the directory entry. That allows compatibility for upgrades by avoiding the need to atomically upgrade a whole directory full of dirents to have 64-bit inode number fields. It also avoids the need to store 64-bit inode numbers when most of them are 32-bit values. > > > I'm not against the idea of exploring this approach, but I am concerned that e2fsck time will continue to grow with the number of inodes in a single filesystem, and scalability of a single MDT will increasingly be an issue. > > AFAIK, e2fsck times are currently on the order of 1h/100M inodes, and no Lustre filesystems that I know of are above 500M inodes today just due to the average file size being so large and the long e2fsck times. With improvements in ext4 to reduce e2fsck time like flex_bg, and SSDs, it may be possible to reduce the e2fsck time/inode ratio noticeably, but I think it would take more effort on the e2fsck side than the ext4 side to make many billions of inodes in a single MDT a practical approach. > > Things like metadata prefetching that Dave Dillow was doing for e2scan with event-driven completion handlers that process the blocks whenever they arrive from the disk, and multi-threading some of the passes of e2fsck with an understanding of the underlying disk layout (using the s_raid_stride and s_raid_stripe_width). > > Cheers, Andreas > -- > Andreas Dilger > Principal Engineer > Whamcloud, Inc. > > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From alexey_lyashkov at xyratex.com Fri Apr 22 15:33:25 2011 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Fri, 22 Apr 2011 19:33:25 +0400 Subject: [Lustre-devel] 64-bit ext4 inodes In-Reply-To: <5E5695A6-CE4E-4253-B97C-85FAD3C96FFF@llnl.gov> References: <2368DAF2-4CA9-41E4-8A56-E48AEA9D4549@xyratex.com> <73AED5C780AE05478241DB067651A92102378476@XYUS-EX22.xyus.xyratex.com> <50A7CDE4-73B3-4BEB-9BF4-BE0D077A3A00@whamcloud.com> <5E5695A6-CE4E-4253-B97C-85FAD3C96FFF@llnl.gov> Message-ID: <28DD036A-3277-4D65-87A5-ADB9ECF7B51C@xyratex.com> Andreas, is possible to check different inodes groups on different cores in parallel? It's should be dramatically increase time of checking on systems with 8-16 CPU cores. On Apr 22, 2011, at 19:13, D. Marc Stearman wrote: > Andreas, LLNL often has file systems with more than 500M inodes in use. We create them with 1B inodes just for that reason. And yes, e2fsck times are horrible. Very painful. > > -Marc > > ---- > D. Marc Stearman > Lustre Operations Lead > marc at llnl.gov > 925.423.9670 > Pager: 1.888.203.0641 > > > > > On Apr 21, 2011, at 3:56 PM, Andreas Dilger wrote: > >> On Apr 20, 2011, at 11:07 AM, Nathan Rutman wrote: >>> We're about to start looking into 64-bit inodes on ext4 -- anybody else working on this at the moment? >> >> I don't know of anyone other than Fujitsu that are looking at this. I haven't had any kind of technical discussion with them about it, nor looked at their code, but just implied that this is what they did based on their slides. >> >> One important consideration to note with 64-bit inode numbers is that they cannot be used safely on 1.8 MDS filesystems. That is because the IGIF FID namespace only has room for 2^32 inode numbers (mapped into the 2.x FID SEQ field, see ), so upgrading a 1.8 MDS with 64-bit inode numbers to 2.x would cause a huge world of hurt. If this is limited to 2.x filesystems that only identify MDS inodes via FIDs to the clients then this is not a concern. >> >> If you are concerned with being able to upgrade 32-bit inode filesystems into 64-bit inode filesystems, you should look at the data-in-dirent patch that is currently in the 2.x ldiskfs patchset. It was developed to allow storing the 128-bit Lustre FID in the directory entry in a compatible manner, but was also designed to allow storing the high 32 bits of a 64-bit inode number into the directory entry. That allows compatibility for upgrades by avoiding the need to atomically upgrade a whole directory full of dirents to have 64-bit inode number fields. It also avoids the need to store 64-bit inode numbers when most of them are 32-bit values. >> >> >> I'm not against the idea of exploring this approach, but I am concerned that e2fsck time will continue to grow with the number of inodes in a single filesystem, and scalability of a single MDT will increasingly be an issue. >> >> AFAIK, e2fsck times are currently on the order of 1h/100M inodes, and no Lustre filesystems that I know of are above 500M inodes today just due to the average file size being so large and the long e2fsck times. With improvements in ext4 to reduce e2fsck time like flex_bg, and SSDs, it may be possible to reduce the e2fsck time/inode ratio noticeably, but I think it would take more effort on the e2fsck side than the ext4 side to make many billions of inodes in a single MDT a practical approach. >> >> Things like metadata prefetching that Dave Dillow was doing for e2scan with event-driven completion handlers that process the blocks whenever they arrive from the disk, and multi-threading some of the passes of e2fsck with an understanding of the underlying disk layout (using the s_raid_stride and s_raid_stripe_width). >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Principal Engineer >> Whamcloud, Inc. >> >> >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel -------------------------------------------- Alexey Lyashkov alexey_lyashkov at xyratex.com ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________ From nirmal at fnal.gov Fri Apr 22 16:08:50 2011 From: nirmal at fnal.gov (Nirmal Seenu) Date: Fri, 22 Apr 2011 11:08:50 -0500 Subject: [Lustre-devel] [Lustre-discuss] Research on filesystem metadata operation distribution Message-ID: <4DB1A812.4080504@fnal.gov> Hi Andreas, Attached are the fsstats and other statistics that you requested from the 2 lustre file systems that I administer. Thanks Nirmal -------------- next part -------------- We use this filesystem for project and code storage without any backup. Users need to write these files manually to the tape if they need it a backup uptime 11:04:04 up 10 days, 45 min, 1 user, load average: 0.11, 0.22, 0.25 open 21842492 samples [reqs] close 7783529 samples [reqs] unlink 89046 samples [reqs] rename 10803 samples [reqs] getattr 136445347 samples [reqs] setattr 97527 samples [reqs] sync 43379 samples [reqs] 'lfs df' output: UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 110092260 3350556 106741704 3% /lustre[MDT:0] lustre-OST0000_UUID 2884114628 2499472756 384641652 86% /lustre[OST:0] lustre-OST0001_UUID 2884114628 2531717148 352397396 87% /lustre[OST:1] lustre-OST0002_UUID 2884114628 2471885580 412215628 85% /lustre[OST:2] lustre-OST0003_UUID 2884114628 2494999084 389115444 86% /lustre[OST:3] lustre-OST0004_UUID 2884114628 2499011196 385103332 86% /lustre[OST:4] lustre-OST0005_UUID 2884114628 2486785532 397329008 86% /lustre[OST:5] lustre-OST0006_UUID 2884114628 2520312396 363801188 87% /lustre[OST:6] lustre-OST0007_UUID 2884114628 2453422812 430691748 85% /lustre[OST:7] lustre-OST0008_UUID 2884114628 2570931872 313182360 89% /lustre[OST:8] lustre-OST0009_UUID 2884085444 2383729096 500356272 82% /lustre[OST:9] lustre-OST000a_UUID 2884110020 2512100708 372009308 87% /lustre[OST:10] lustre-OST000b_UUID 2884110020 2460095924 424014092 85% /lustre[OST:11] lustre-OST000c_UUID 2884114628 2494064804 390049740 86% /lustre[OST:12] lustre-OST000d_UUID 2884114628 2378430176 505684448 82% /lustre[OST:13] lustre-OST000e_UUID 2884114628 2510388140 373726484 87% /lustre[OST:14] lustre-OST000f_UUID 2884114628 2483525540 400572704 86% /lustre[OST:15] lustre-OST0010_UUID 2884114628 2496379872 387734664 86% /lustre[OST:16] lustre-OST0011_UUID 2884114628 2460190636 423923896 85% /lustre[OST:17] lustre-OST0012_UUID 2884114628 2451608360 432506172 85% /lustre[OST:18] lustre-OST0013_UUID 2884114628 2413632956 470481516 83% /lustre[OST:19] lustre-OST0014_UUID 2884114628 2427340220 456774284 84% /lustre[OST:20] lustre-OST0015_UUID 2884110020 2386463304 497646628 82% /lustre[OST:21] lustre-OST0016_UUID 2884110020 2469800676 414309340 85% /lustre[OST:22] lustre-OST0017_UUID 2884110020 2508144364 375965512 86% /lustre[OST:23] lustre-OST0018_UUID 2884114628 2491842856 392264528 86% /lustre[OST:24] lustre-OST0019_UUID 2884114628 2544022572 340091964 88% /lustre[OST:25] lustre-OST001a_UUID 2884114628 2384050288 500064256 82% /lustre[OST:26] lustre-OST001b_UUID 2884114628 2541539344 342575220 88% /lustre[OST:27] lustre-OST001c_UUID 2884114628 2420779448 463335108 83% /lustre[OST:28] lustre-OST001d_UUID 2884114628 2505354096 378760528 86% /lustre[OST:29] lustre-OST001e_UUID 2884114628 2357578536 526536016 81% /lustre[OST:30] lustre-OST001f_UUID 2884114628 2540526796 343587744 88% /lustre[OST:31] lustre-OST0020_UUID 2884110020 2476502220 407607596 85% /lustre[OST:32] lustre-OST0021_UUID 2884110020 2477269720 406838184 85% /lustre[OST:33] lustre-OST0022_UUID 2884110020 2500748300 383361636 86% /lustre[OST:34] lustre-OST0023_UUID 2884114628 2462201160 421913380 85% /lustre[OST:35] lustre-OST0024_UUID 2883950796 2440698836 443251796 84% /lustre[OST:36] lustre-OST0025_UUID 2883950796 2482524832 401425792 86% /lustre[OST:37] lustre-OST0026_UUID 2883950796 2416383064 467567580 83% /lustre[OST:38] lustre-OST0027_UUID 2883950796 2455693800 428251840 85% /lustre[OST:39] lustre-OST0028_UUID 2883950796 2445335428 438614560 84% /lustre[OST:40] lustre-OST0029_UUID 2883950796 2443137624 440813060 84% /lustre[OST:41] lustre-OST002a_UUID 2883950796 2461243568 422707132 85% /lustre[OST:42] lustre-OST002b_UUID 2883950796 2439715980 444234808 84% /lustre[OST:43] lustre-OST002c_UUID 2883950796 2499767284 384183356 86% /lustre[OST:44] lustre-OST002d_UUID 2883950796 2365586028 518364668 82% /lustre[OST:45] lustre-OST002e_UUID 2883950796 2546567452 337382720 88% /lustre[OST:46] lustre-OST002f_UUID 2883950796 2432341816 451608888 84% /lustre[OST:47] filesystem summary: 138435470112 118495844200 19939575176 85% /lustre 'lfs df -i' output: UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 31457280 21292654 10164626 67% /lustre[MDT:0] lustre-OST0000_UUID 96533255 372787 96160468 0% /lustre[OST:0] lustre-OST0001_UUID 88448539 349169 88099370 0% /lustre[OST:1] lustre-OST0002_UUID 103403114 345853 103057261 0% /lustre[OST:2] lustre-OST0003_UUID 97609328 330442 97278886 0% /lustre[OST:3] lustre-OST0004_UUID 96605816 329958 96275858 0% /lustre[OST:4] lustre-OST0005_UUID 99659060 326786 99332274 0% /lustre[OST:5] lustre-OST0006_UUID 91319290 368732 90950558 0% /lustre[OST:6] lustre-OST0007_UUID 108048101 375147 107672954 0% /lustre[OST:7] lustre-OST0008_UUID 78569804 293663 78276141 0% /lustre[OST:8] lustre-OST0009_UUID 125402379 313292 125089087 0% /lustre[OST:9] lustre-OST000a_UUID 93318430 316102 93002328 0% /lustre[OST:10] lustre-OST000b_UUID 106322365 318846 106003519 0% /lustre[OST:11] lustre-OST000c_UUID 97758361 245905 97512456 0% /lustre[OST:12] lustre-OST000d_UUID 126748774 327661 126421113 0% /lustre[OST:13] lustre-OST000e_UUID 93786667 355045 93431622 0% /lustre[OST:14] lustre-OST000f_UUID 100467098 322388 100144710 0% /lustre[OST:15] lustre-OST0010_UUID 97267658 333970 96933688 0% /lustre[OST:16] lustre-OST0011_UUID 106334239 353241 105980998 0% /lustre[OST:17] lustre-OST0012_UUID 108411835 285270 108126565 0% /lustre[OST:18] lustre-OST0013_UUID 117952887 333641 117619246 0% /lustre[OST:19] lustre-OST0014_UUID 114526216 332614 114193602 0% /lustre[OST:20] lustre-OST0015_UUID 124742257 330578 124411679 0% /lustre[OST:21] lustre-OST0016_UUID 103917055 339719 103577336 0% /lustre[OST:22] lustre-OST0017_UUID 94337542 350736 93986806 0% /lustre[OST:23] lustre-OST0018_UUID 98509166 441223 98067943 0% /lustre[OST:24] lustre-OST0019_UUID 85452177 432591 85019586 0% /lustre[OST:25] lustre-OST001a_UUID 125453573 437488 125016085 0% /lustre[OST:26] lustre-OST001b_UUID 86097048 453227 85643821 0% /lustre[OST:27] lustre-OST001c_UUID 116282388 448593 115833795 0% /lustre[OST:28] lustre-OST001d_UUID 95133208 443076 94690132 0% /lustre[OST:29] lustre-OST001e_UUID 132088113 454090 131634023 0% /lustre[OST:30] lustre-OST001f_UUID 86329718 432760 85896958 0% /lustre[OST:31] lustre-OST0020_UUID 102316620 414670 101901950 0% /lustre[OST:32] lustre-OST0021_UUID 102164199 454124 101710075 0% /lustre[OST:33] lustre-OST0022_UUID 96297776 457346 95840430 0% /lustre[OST:34] lustre-OST0023_UUID 105905193 426827 105478366 0% /lustre[OST:35] lustre-OST0024_UUID 111478595 666776 110811819 0% /lustre[OST:36] lustre-OST0025_UUID 101013469 658184 100355285 0% /lustre[OST:37] lustre-OST0026_UUID 117527698 636971 116890727 0% /lustre[OST:38] lustre-OST0027_UUID 107707258 643009 107064249 0% /lustre[OST:39] lustre-OST0028_UUID 110287346 633506 109653840 0% /lustre[OST:40] lustre-OST0029_UUID 110859994 656701 110203293 0% /lustre[OST:41] lustre-OST002a_UUID 106307620 630813 105676807 0% /lustre[OST:42] lustre-OST002b_UUID 111705160 647482 111057678 0% /lustre[OST:43] lustre-OST002c_UUID 96698009 653175 96044834 0% /lustre[OST:44] lustre-OST002d_UUID 130237227 646036 129591191 0% /lustre[OST:45] lustre-OST002e_UUID 84928910 583928 84344982 0% /lustre[OST:46] lustre-OST002f_UUID 113564717 662474 112902243 0% /lustre[OST:47] filesystem summary: 31457280 21292654 10164626 67% /lustre -------------- next part -------------- We use this filesystem for project and code storage without any backup. Users need to write these files manually to the tape if they need it a backup uptime 11:03:33 up 28 days, 21:03, 2 users, load average: 0.24, 0.33, 0.27 open 584823418 samples [reqs] close 287320453 samples [reqs] link 150 samples [reqs] unlink 11514436 samples [reqs] rename 2401131 samples [reqs] getattr 2520767021 samples [reqs] setattr 131905450 samples [reqs] getxattr 13030 samples [reqs] sync 7225 samples [reqs] 'lfs df' output: UUID 1K-blocks Used Available Use% Mounted on lqcdproj-MDT0000_UUID 293579372 7857624 285721748 2% /lqcdproj[MDT:0] lqcdproj-OST0000_UUID 2884110020 2590181764 293921204 89% /lqcdproj[OST:0] lqcdproj-OST0001_UUID 2884114628 2578555520 305519296 89% /lqcdproj[OST:1] lqcdproj-OST0002_UUID 2884114628 2637417688 246680044 91% /lqcdproj[OST:2] lqcdproj-OST0003_UUID 2884114628 2582852532 301249252 89% /lqcdproj[OST:3] lqcdproj-OST0004_UUID 2884114628 2620080388 264025788 90% /lqcdproj[OST:4] lqcdproj-OST0005_UUID 2884114628 2604582988 279522872 90% /lqcdproj[OST:5] lqcdproj-OST0006_UUID 2884114628 2586588840 297511516 89% /lqcdproj[OST:6] lqcdproj-OST0007_UUID 2884114628 2569621824 314467596 89% /lqcdproj[OST:7] lqcdproj-OST0008_UUID 2884114628 2659842076 224236404 92% /lqcdproj[OST:8] lqcdproj-OST0009_UUID 2884114628 2724758644 159302636 94% /lqcdproj[OST:9] lqcdproj-OST000a_UUID 2884110020 2561081440 323017532 88% /lqcdproj[OST:10] lqcdproj-OST000b_UUID 2884110020 2544083880 340016388 88% /lqcdproj[OST:11] lqcdproj-OST000c_UUID 2884110020 2642758032 241341396 91% /lqcdproj[OST:12] lqcdproj-OST000d_UUID 2884114628 2583504688 300604652 89% /lqcdproj[OST:13] lqcdproj-OST000e_UUID 2884114628 2578575996 305531808 89% /lqcdproj[OST:14] lqcdproj-OST000f_UUID 2884114628 2595276180 288824004 89% /lqcdproj[OST:15] lqcdproj-OST0010_UUID 2884114628 2620524300 263582496 90% /lqcdproj[OST:16] lqcdproj-OST0011_UUID 2884114628 2611184664 272924232 90% /lqcdproj[OST:17] lqcdproj-OST0012_UUID 2884114628 2606211324 277901256 90% /lqcdproj[OST:18] lqcdproj-OST0013_UUID 2884114628 2636079432 248031000 91% /lqcdproj[OST:19] lqcdproj-OST0014_UUID 2884114628 2599592996 284515536 90% /lqcdproj[OST:20] lqcdproj-OST0015_UUID 2884114628 2592473340 291635996 89% /lqcdproj[OST:21] lqcdproj-OST0016_UUID 2884110020 2604694892 279414104 90% /lqcdproj[OST:22] lqcdproj-OST0017_UUID 2884110020 2648410128 235695700 91% /lqcdproj[OST:23] lqcdproj-OST0018_UUID 2884114628 2686381976 197732648 93% /lqcdproj[OST:24] lqcdproj-OST0019_UUID 2884114628 2533603660 350496404 87% /lqcdproj[OST:25] lqcdproj-OST001a_UUID 2884114628 2626796040 257311016 91% /lqcdproj[OST:26] lqcdproj-OST001b_UUID 2884114628 2555034880 329070616 88% /lqcdproj[OST:27] lqcdproj-OST001c_UUID 2884114628 2621915896 262190744 90% /lqcdproj[OST:28] lqcdproj-OST001d_UUID 2884114628 2606788840 277320264 90% /lqcdproj[OST:29] lqcdproj-OST001e_UUID 2884114628 2616370992 267738128 90% /lqcdproj[OST:30] lqcdproj-OST001f_UUID 2884114628 2543756572 340349332 88% /lqcdproj[OST:31] lqcdproj-OST0020_UUID 2884114628 2646802380 237301584 91% /lqcdproj[OST:32] lqcdproj-OST0021_UUID 2884114628 2564478480 319625340 88% /lqcdproj[OST:33] lqcdproj-OST0022_UUID 2884114628 2517282212 366822488 87% /lqcdproj[OST:34] lqcdproj-OST0023_UUID 2884114628 2501465080 382640824 86% /lqcdproj[OST:35] lqcdproj-OST0024_UUID 2884114628 2510496668 373608216 87% /lqcdproj[OST:36] lqcdproj-OST0025_UUID 2884110020 2504784112 379316448 86% /lqcdproj[OST:37] lqcdproj-OST0026_UUID 2884114628 2530597412 353508568 87% /lqcdproj[OST:38] lqcdproj-OST0027_UUID 2884110020 2620957092 263146608 90% /lqcdproj[OST:39] lqcdproj-OST0028_UUID 2884114628 2549798152 334303804 88% /lqcdproj[OST:40] lqcdproj-OST0029_UUID 2884110020 2504016944 380081608 86% /lqcdproj[OST:41] lqcdproj-OST002a_UUID 2884110020 2598782176 285319260 90% /lqcdproj[OST:42] lqcdproj-OST002b_UUID 2884114628 2552118656 331990968 88% /lqcdproj[OST:43] lqcdproj-OST002c_UUID 2884114628 2596699732 287408360 90% /lqcdproj[OST:44] lqcdproj-OST002d_UUID 2884114628 2602101740 282009660 90% /lqcdproj[OST:45] lqcdproj-OST002e_UUID 2884110020 2519556192 364534932 87% /lqcdproj[OST:46] lqcdproj-OST002f_UUID 2884110020 2521895376 362207776 87% /lqcdproj[OST:47] lqcdproj-OST0030_UUID 2884059332 2609648960 274405616 90% /lqcdproj[OST:48] lqcdproj-OST0031_UUID 2884059332 2483629124 400402048 86% /lqcdproj[OST:49] lqcdproj-OST0032_UUID 2884271796 2541670644 342593520 88% /lqcdproj[OST:50] lqcdproj-OST0033_UUID 2884059332 2525701256 358338688 87% /lqcdproj[OST:51] lqcdproj-OST0034_UUID 2884059332 2541320880 342688588 88% /lqcdproj[OST:52] lqcdproj-OST0035_UUID 2884059332 2542146756 341893700 88% /lqcdproj[OST:53] lqcdproj-OST0036_UUID 2884059332 2608223652 275822700 90% /lqcdproj[OST:54] lqcdproj-OST0037_UUID 2884059332 2550321108 333718140 88% /lqcdproj[OST:55] lqcdproj-OST0038_UUID 2884271796 2532312320 351922216 87% /lqcdproj[OST:56] lqcdproj-OST0039_UUID 2884059332 2600728976 283324084 90% /lqcdproj[OST:57] lqcdproj-OST003a_UUID 2884271796 2527692476 356570160 87% /lqcdproj[OST:58] lqcdproj-OST003b_UUID 2884059332 2522126108 361913352 87% /lqcdproj[OST:59] lqcdproj-OST003c_UUID 2883950796 2539851408 344088720 88% /lqcdproj[OST:60] lqcdproj-OST003d_UUID 2883950796 2522631612 361319180 87% /lqcdproj[OST:61] lqcdproj-OST003e_UUID 2883950796 2519383116 364557456 87% /lqcdproj[OST:62] lqcdproj-OST003f_UUID 2883950796 2517934588 366009188 87% /lqcdproj[OST:63] lqcdproj-OST0040_UUID 2883950796 2567027044 316917440 89% /lqcdproj[OST:64] lqcdproj-OST0041_UUID 2883950796 2551930848 332019944 88% /lqcdproj[OST:65] lqcdproj-OST0042_UUID 2883950796 2594902772 289039436 89% /lqcdproj[OST:66] lqcdproj-OST0043_UUID 2883950796 2527737320 356207972 87% /lqcdproj[OST:67] lqcdproj-OST0044_UUID 2883950796 2612826076 271117460 90% /lqcdproj[OST:68] lqcdproj-OST0045_UUID 2883950796 2538947204 344990568 88% /lqcdproj[OST:69] lqcdproj-OST0046_UUID 2883950796 2504534560 379408952 86% /lqcdproj[OST:70] lqcdproj-OST0047_UUID 2883950796 2480640640 403299636 86% /lqcdproj[OST:71] lqcdproj-OST0048_UUID 5768390140 5063888540 704486076 87% /lqcdproj[OST:72] lqcdproj-OST0049_UUID 5768390140 5037949292 730419100 87% /lqcdproj[OST:73] lqcdproj-OST004a_UUID 5768390140 5016546960 751818916 86% /lqcdproj[OST:74] lqcdproj-OST004b_UUID 5768390140 5291495028 476887856 91% /lqcdproj[OST:75] lqcdproj-OST004c_UUID 5768390140 5252542904 515836124 91% /lqcdproj[OST:76] lqcdproj-OST004d_UUID 5768390140 5340007080 428377284 92% /lqcdproj[OST:77] lqcdproj-OST004e_UUID 5768390140 5266269356 502114756 91% /lqcdproj[OST:78] lqcdproj-OST004f_UUID 5768390140 5299866296 468502640 91% /lqcdproj[OST:79] lqcdproj-OST0050_UUID 5768390140 5245145984 523230832 90% /lqcdproj[OST:80] lqcdproj-OST0051_UUID 5768390140 5328721480 439651896 92% /lqcdproj[OST:81] lqcdproj-OST0052_UUID 5768390140 5254759116 513546288 91% /lqcdproj[OST:82] lqcdproj-OST0053_UUID 5768390140 5300003640 468372496 91% /lqcdproj[OST:83] lqcdproj-OST0054_UUID 5768227844 5041973956 726238088 87% /lqcdproj[OST:84] lqcdproj-OST0055_UUID 5768227844 4860381548 907845424 84% /lqcdproj[OST:85] lqcdproj-OST0056_UUID 5768226308 4895829896 872370424 84% /lqcdproj[OST:86] lqcdproj-OST0057_UUID 5768227844 4979154952 789069676 86% /lqcdproj[OST:87] lqcdproj-OST0058_UUID 5768227844 4938993612 829219672 85% /lqcdproj[OST:88] lqcdproj-OST0059_UUID 5768227844 4816827608 951400104 83% /lqcdproj[OST:89] lqcdproj-OST005a_UUID 5768227844 5390550920 377632628 93% /lqcdproj[OST:90] lqcdproj-OST005b_UUID 5768226308 5102879020 665346192 88% /lqcdproj[OST:91] lqcdproj-OST005c_UUID 5768226308 5289334572 478883020 91% /lqcdproj[OST:92] lqcdproj-OST005d_UUID 5768226308 5170952732 597271096 89% /lqcdproj[OST:93] lqcdproj-OST005e_UUID 5768226308 5311539252 456615240 92% /lqcdproj[OST:94] lqcdproj-OST005f_UUID 5768226308 5180524124 587701028 89% /lqcdproj[OST:95] lqcdproj-OST0060_UUID 5768390140 5340309376 428066340 92% /lqcdproj[OST:96] lqcdproj-OST0061_UUID 5768390140 5504060088 264318804 95% /lqcdproj[OST:97] lqcdproj-OST0062_UUID 5768390140 5332176216 436204732 92% /lqcdproj[OST:98] lqcdproj-OST0063_UUID 5768390140 5325875212 442501768 92% /lqcdproj[OST:99] filesystem summary: 369167172928 330453843024 38712005568 89% /lqcdproj 'lfs df -i' output: UUID Inodes IUsed IFree IUse% Mounted on lqcdproj-MDT0000_UUID 83886080 60183877 23702203 71% /lqcdproj[MDT:0] lqcdproj-OST0000_UUID 73880027 397957 73482070 0% /lqcdproj[OST:0] lqcdproj-OST0001_UUID 76716189 326412 76389777 0% /lqcdproj[OST:1] lqcdproj-OST0002_UUID 62039485 365250 61674235 0% /lqcdproj[OST:2] lqcdproj-OST0003_UUID 75704592 389068 75315524 0% /lqcdproj[OST:3] lqcdproj-OST0004_UUID 66395655 387095 66008560 0% /lqcdproj[OST:4] lqcdproj-OST0005_UUID 70270553 387645 69882908 0% /lqcdproj[OST:5] lqcdproj-OST0006_UUID 74783738 402291 74381447 0% /lqcdproj[OST:6] lqcdproj-OST0007_UUID 78977890 354689 78623201 0% /lqcdproj[OST:7] lqcdproj-OST0008_UUID 56447971 381369 56066602 0% /lqcdproj[OST:8] lqcdproj-OST0009_UUID 40225332 386336 39838996 0% /lqcdproj[OST:9] lqcdproj-OST000a_UUID 81093144 335999 80757145 0% /lqcdproj[OST:10] lqcdproj-OST000b_UUID 85379427 372892 85006535 0% /lqcdproj[OST:11] lqcdproj-OST000c_UUID 60706689 368692 60337997 0% /lqcdproj[OST:12] lqcdproj-OST000d_UUID 75570483 417998 75152485 0% /lqcdproj[OST:13] lqcdproj-OST000e_UUID 76758073 373415 76384658 0% /lqcdproj[OST:14] lqcdproj-OST000f_UUID 72628594 418982 72209612 0% /lqcdproj[OST:15] lqcdproj-OST0010_UUID 66315271 417689 65897582 0% /lqcdproj[OST:16] lqcdproj-OST0011_UUID 68628234 395743 68232491 0% /lqcdproj[OST:17] lqcdproj-OST0012_UUID 69896803 422513 69474290 0% /lqcdproj[OST:18] lqcdproj-OST0013_UUID 62414995 406196 62008799 0% /lqcdproj[OST:19] lqcdproj-OST0014_UUID 71551266 420858 71130408 0% /lqcdproj[OST:20] lqcdproj-OST0015_UUID 73383915 473593 72910322 0% /lqcdproj[OST:21] lqcdproj-OST0016_UUID 70203077 350831 69852246 0% /lqcdproj[OST:22] lqcdproj-OST0017_UUID 59279796 354823 58924973 0% /lqcdproj[OST:23] lqcdproj-OST0018_UUID 49830967 397804 49433163 0% /lqcdproj[OST:24] lqcdproj-OST0019_UUID 88106064 478322 87627742 0% /lqcdproj[OST:25] lqcdproj-OST001a_UUID 64805023 475376 64329647 0% /lqcdproj[OST:26] lqcdproj-OST001b_UUID 82699043 429106 82269937 0% /lqcdproj[OST:27] lqcdproj-OST001c_UUID 66038893 489210 65549683 0% /lqcdproj[OST:28] lqcdproj-OST001d_UUID 69823950 492503 69331447 0% /lqcdproj[OST:29] lqcdproj-OST001e_UUID 67416308 480399 66935909 0% /lqcdproj[OST:30] lqcdproj-OST001f_UUID 85552039 462525 85089514 0% /lqcdproj[OST:31] lqcdproj-OST0020_UUID 59764579 436517 59328062 0% /lqcdproj[OST:32] lqcdproj-OST0021_UUID 80352833 443796 79909037 0% /lqcdproj[OST:33] lqcdproj-OST0022_UUID 92194297 486193 91708104 0% /lqcdproj[OST:34] lqcdproj-OST0023_UUID 96100782 438395 95662387 0% /lqcdproj[OST:35] lqcdproj-OST0024_UUID 93849871 445381 93404490 0% /lqcdproj[OST:36] lqcdproj-OST0025_UUID 95270131 438654 94831477 0% /lqcdproj[OST:37] lqcdproj-OST0026_UUID 88838023 458719 88379304 0% /lqcdproj[OST:38] lqcdproj-OST0027_UUID 66243226 454994 65788232 0% /lqcdproj[OST:39] lqcdproj-OST0028_UUID 83983949 404830 83579119 0% /lqcdproj[OST:40] lqcdproj-OST0029_UUID 95530677 507408 95023269 0% /lqcdproj[OST:41] lqcdproj-OST002a_UUID 71825296 493335 71331961 0% /lqcdproj[OST:42] lqcdproj-OST002b_UUID 83462030 463037 82998993 0% /lqcdproj[OST:43] lqcdproj-OST002c_UUID 72272617 418893 71853724 0% /lqcdproj[OST:44] lqcdproj-OST002d_UUID 70962743 459521 70503222 0% /lqcdproj[OST:45] lqcdproj-OST002e_UUID 91614329 475872 91138457 0% /lqcdproj[OST:46] lqcdproj-OST002f_UUID 91024844 471181 90553663 0% /lqcdproj[OST:47] lqcdproj-OST0030_UUID 69055273 452680 68602593 0% /lqcdproj[OST:48] lqcdproj-OST0031_UUID 100580517 472965 100107552 0% /lqcdproj[OST:49] lqcdproj-OST0032_UUID 86092421 442133 85650288 0% /lqcdproj[OST:50] lqcdproj-OST0033_UUID 90112864 523345 89589519 0% /lqcdproj[OST:51] lqcdproj-OST0034_UUID 86119923 435310 85684613 0% /lqcdproj[OST:52] lqcdproj-OST0035_UUID 85966328 488184 85478144 0% /lqcdproj[OST:53] lqcdproj-OST0036_UUID 69415217 456297 68958920 0% /lqcdproj[OST:54] lqcdproj-OST0037_UUID 83978786 544230 83434556 0% /lqcdproj[OST:55] lqcdproj-OST0038_UUID 88442388 452519 87989869 0% /lqcdproj[OST:56] lqcdproj-OST0039_UUID 71330410 497821 70832589 0% /lqcdproj[OST:57] lqcdproj-OST003a_UUID 89635608 490778 89144830 0% /lqcdproj[OST:58] lqcdproj-OST003b_UUID 90907311 424005 90483306 0% /lqcdproj[OST:59] lqcdproj-OST003c_UUID 86566472 541625 86024847 0% /lqcdproj[OST:60] lqcdproj-OST003d_UUID 90887507 557711 90329796 0% /lqcdproj[OST:61] lqcdproj-OST003e_UUID 91640006 498086 91141920 0% /lqcdproj[OST:62] lqcdproj-OST003f_UUID 92065253 561201 91504052 0% /lqcdproj[OST:63] lqcdproj-OST0040_UUID 79696800 465862 79230938 0% /lqcdproj[OST:64] lqcdproj-OST0041_UUID 83514851 509864 83004987 0% /lqcdproj[OST:65] lqcdproj-OST0042_UUID 72815810 553804 72262006 0% /lqcdproj[OST:66] lqcdproj-OST0043_UUID 89538470 485101 89053369 0% /lqcdproj[OST:67] lqcdproj-OST0044_UUID 68371742 590562 67781180 0% /lqcdproj[OST:68] lqcdproj-OST0045_UUID 86781407 530509 86250898 0% /lqcdproj[OST:69] lqcdproj-OST0046_UUID 95374442 520383 94854059 0% /lqcdproj[OST:70] lqcdproj-OST0047_UUID 101368448 540909 100827539 0% /lqcdproj[OST:71] lqcdproj-OST0048_UUID 176450916 325516 176125400 0% /lqcdproj[OST:72] lqcdproj-OST0049_UUID 182867396 257184 182610212 0% /lqcdproj[OST:73] lqcdproj-OST004a_UUID 188295902 335107 187960795 0% /lqcdproj[OST:74] lqcdproj-OST004b_UUID 119589439 365661 119223778 0% /lqcdproj[OST:75] lqcdproj-OST004c_UUID 129335184 373375 128961809 0% /lqcdproj[OST:76] lqcdproj-OST004d_UUID 107420240 324475 107095765 0% /lqcdproj[OST:77] lqcdproj-OST004e_UUID 125869668 339472 125530196 0% /lqcdproj[OST:78] lqcdproj-OST004f_UUID 117440370 309409 117130961 0% /lqcdproj[OST:79] lqcdproj-OST0050_UUID 131179818 368779 130811039 0% /lqcdproj[OST:80] lqcdproj-OST0051_UUID 110236040 318875 109917165 0% /lqcdproj[OST:81] lqcdproj-OST0052_UUID 128692536 365676 128326860 0% /lqcdproj[OST:82] lqcdproj-OST0053_UUID 117800270 703645 117096625 0% /lqcdproj[OST:83] lqcdproj-OST0054_UUID 182316097 752625 181563472 0% /lqcdproj[OST:84] lqcdproj-OST0055_UUID 227694176 732602 226961574 0% /lqcdproj[OST:85] lqcdproj-OST0056_UUID 218861790 762690 218099100 0% /lqcdproj[OST:86] lqcdproj-OST0057_UUID 197967520 699297 197268223 0% /lqcdproj[OST:87] lqcdproj-OST0058_UUID 208030253 721695 207308558 0% /lqcdproj[OST:88] lqcdproj-OST0059_UUID 238574357 724298 237850059 0% /lqcdproj[OST:89] lqcdproj-OST005a_UUID 95653147 1233916 94419231 1% /lqcdproj[OST:90] lqcdproj-OST005b_UUID 167677190 1340368 166336822 0% /lqcdproj[OST:91] lqcdproj-OST005c_UUID 121032709 1309775 119722934 1% /lqcdproj[OST:92] lqcdproj-OST005d_UUID 150574698 1256304 149318394 0% /lqcdproj[OST:93] lqcdproj-OST005e_UUID 116437508 2265744 114171764 1% /lqcdproj[OST:94] lqcdproj-OST005f_UUID 149201722 2276944 146924778 1% /lqcdproj[OST:95] lqcdproj-OST0060_UUID 109313154 2292963 107020191 2% /lqcdproj[OST:96] lqcdproj-OST0061_UUID 68387346 2304833 66082513 3% /lqcdproj[OST:97] lqcdproj-OST0062_UUID 111369029 2315527 109053502 2% /lqcdproj[OST:98] lqcdproj-OST0063_UUID 112945226 2316494 110628732 2% /lqcdproj[OST:99] filesystem summary: 83886080 60183877 23702203 71% /lqcdproj -------------- next part -------------- fsstats v1.4.5 (03/22/2008) Copyright (c) 2005, 2007 Panasas, Inc. fsstats: checkpointing every 10 minutes to /tmp/fsstats.Jit5u ----- CUT HERE ----- REPORT FOLLOWS ----- CUT HERE ----- Generated by fsstats v1.4.5 (03/22/2008) complete : processed 160 files in 0 secs #Generated by fsstats v1.4.5 (03/22/2008) #Comment: This is a comment line that can be modified or repeated before #uploading to record voluntarily added information. skipped special files,0 skipped duplicate hardlinks,0 skipped snapshot dirs,0 total capacity used,10.87 MB total user data,10.57 MB percent overhead,0.027253 histogram,file size count,108,items average,98.339066,KB min,0,KB max,5821,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,67,0.620370,0.620370,23.616211,0.002224,0.002224 2,4,6,0.055556,0.675926,12.378906,0.001166,0.003389 4,8,8,0.074074,0.750000,49.458984,0.004657,0.008046 8,16,4,0.037037,0.787037,45.958984,0.004327,0.012373 16,32,5,0.046296,0.833333,89.917969,0.008466,0.020840 32,64,8,0.074074,0.907407,373.796875,0.035195,0.056035 64,128,3,0.027778,0.935185,268.972656,0.025326,0.081361 128,256,3,0.027778,0.962963,466.640625,0.043937,0.125298 512,1024,2,0.018519,0.981481,1962.119141,0.184746,0.310044 1024,2048,1,0.009259,0.990741,1506.072266,0.141806,0.451851 4096,8192,1,0.009259,1.000000,5821.686523,0.548149,1.000000 histogram,capacity used count,108,items average,101.148148,KB min,0,KB max,5836,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,15,0.138889,0.138889,0.000000,0.000000,0.000000 4,8,60,0.555556,0.694444,240.000000,0.021970,0.021970 8,16,9,0.083333,0.777778,84.000000,0.007689,0.029659 16,32,6,0.055556,0.833333,112.000000,0.010253,0.039912 32,64,7,0.064815,0.898148,336.000000,0.030758,0.070670 64,128,4,0.037037,0.935185,352.000000,0.032223,0.102893 128,256,3,0.027778,0.962963,480.000000,0.043940,0.146833 512,1024,2,0.018519,0.981481,1972.000000,0.180520,0.327353 1024,2048,1,0.009259,0.990741,1512.000000,0.138411,0.465763 4096,8192,1,0.009259,1.000000,5836.000000,0.534237,1.000000 histogram,positive overhead count,160,items average,1.896130,KB min,0,KB max,14,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,67,0.418750,0.418750,184.383789,0.607763,0.607763 2,4,6,0.037500,0.456250,11.621094,0.038305,0.646069 4,8,60,0.375000,0.831250,6.541016,0.021560,0.667629 8,16,4,0.025000,0.856250,6.041016,0.019912,0.687541 16,32,5,0.031250,0.887500,6.082031,0.020048,0.707589 32,64,8,0.050000,0.937500,30.203125,0.099555,0.807144 64,128,3,0.018750,0.956250,15.027344,0.049533,0.856677 128,256,3,0.018750,0.975000,13.359375,0.044035,0.900712 512,1024,2,0.012500,0.987500,9.880859,0.032569,0.933281 1024,2048,1,0.006250,0.993750,5.927734,0.019539,0.952820 4096,8192,1,0.006250,1.000000,14.313477,0.047180,1.000000 histogram,negative overhead count,0,items average,0.000000,KB min,0,KB max,0,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct histogram,directory size (entries) count,52,items average,3.057692,ents min,0,ents max,57,ents bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,1,29,0.557692,0.557692,18.000000,0.113208,0.113208 2,3,17,0.326923,0.884615,36.000000,0.226415,0.339623 4,7,4,0.076923,0.961538,16.000000,0.100629,0.440252 32,63,2,0.038462,1.000000,89.000000,0.559748,1.000000 histogram,directory size count,52,items average,4.000000,KB min,4,KB max,4,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 4,8,52,1.000000,1.000000,208.000000,1.000000,1.000000 histogram,filename length count,160,items average,11.625000,chars min,1,chars max,38,chars bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,7,37,0.231250,0.231250,211.000000,0.113441,0.113441 8,15,94,0.587500,0.818750,1034.000000,0.555914,0.669355 16,23,23,0.143750,0.962500,432.000000,0.232258,0.901613 24,31,4,0.025000,0.987500,113.000000,0.060753,0.962366 32,39,2,0.012500,1.000000,70.000000,0.037634,1.000000 histogram,link count count,108,items average,1.000000,links min,1,links max,1,links bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,1,108,1.000000,1.000000,108.000000,1.000000,1.000000 histogram,symlink target length count,0,items aver ----- BEGIN CSV ----- ----- END CSV ----- ----- BEGIN NORMAL ----- total 10.87 MB used to store 10.57 MB user data, overhead 2.73% file size (Regular file's last addressable byte): Range, Count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of files, total size of files in range, total size of files in range as a % of total size of files, total size of files in this range or smaller as a % of total size of files. count=108 avg=98.34 KB min=0.00 KB max=5821.69 KB [ 0- 2 KB): 67 (62.04%) ( 62.04% cumulative) 23.62 KB ( 0.22%) ( 0.22% cumulative) [ 2- 4 KB): 6 ( 5.56%) ( 67.59% cumulative) 12.38 KB ( 0.12%) ( 0.34% cumulative) [ 4- 8 KB): 8 ( 7.41%) ( 75.00% cumulative) 49.46 KB ( 0.47%) ( 0.80% cumulative) [ 8- 16 KB): 4 ( 3.70%) ( 78.70% cumulative) 45.96 KB ( 0.43%) ( 1.24% cumulative) [ 16- 32 KB): 5 ( 4.63%) ( 83.33% cumulative) 89.92 KB ( 0.85%) ( 2.08% cumulative) [ 32- 64 KB): 8 ( 7.41%) ( 90.74% cumulative) 373.80 KB ( 3.52%) ( 5.60% cumulative) [ 64- 128 KB): 3 ( 2.78%) ( 93.52% cumulative) 268.97 KB ( 2.53%) ( 8.14% cumulative) [ 128- 256 KB): 3 ( 2.78%) ( 96.30% cumulative) 466.64 KB ( 4.39%) ( 12.53% cumulative) [ 512-1024 KB): 2 ( 1.85%) ( 98.15% cumulative) 1962.12 KB (18.47%) ( 31.00% cumulative) [1024-2048 KB): 1 ( 0.93%) ( 99.07% cumulative) 1506.07 KB (14.18%) ( 45.19% cumulative) [4096-8192 KB): 1 ( 0.93%) (100.00% cumulative) 5821.69 KB (54.81%) (100.00% cumulative) capacity used (Regualr file's space used on disk): Range, Count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of file, total capacity used in range, total capacity used in range as a % of total capacity used, total capacity used in this range or smaller as a % of total capacity. count=108 avg=101.15 KB min=0.00 KB max=5836.00 KB [ 0- 2 KB): 15 (13.89%) ( 13.89% cumulative) 0.00 KB ( 0.00%) ( 0.00% cumulative) [ 4- 8 KB): 60 (55.56%) ( 69.44% cumulative) 240.00 KB ( 2.20%) ( 2.20% cumulative) [ 8- 16 KB): 9 ( 8.33%) ( 77.78% cumulative) 84.00 KB ( 0.77%) ( 2.97% cumulative) [ 16- 32 KB): 6 ( 5.56%) ( 83.33% cumulative) 112.00 KB ( 1.03%) ( 3.99% cumulative) [ 32- 64 KB): 7 ( 6.48%) ( 89.81% cumulative) 336.00 KB ( 3.08%) ( 7.07% cumulative) [ 64- 128 KB): 4 ( 3.70%) ( 93.52% cumulative) 352.00 KB ( 3.22%) ( 10.29% cumulative) [ 128- 256 KB): 3 ( 2.78%) ( 96.30% cumulative) 480.00 KB ( 4.39%) ( 14.68% cumulative) [ 512-1024 KB): 2 ( 1.85%) ( 98.15% cumulative) 1972.00 KB (18.05%) ( 32.74% cumulative) [1024-2048 KB): 1 ( 0.93%) ( 99.07% cumulative) 1512.00 KB (13.84%) ( 46.58% cumulative) [4096-8192 KB): 1 ( 0.93%) (100.00% cumulative) 5836.00 KB (53.42%) (100.00% cumulative) positive overhead (Files whose capacity used is larger than or equal to its size. Includes directories, symlinks and regular files): Range of file sizes, count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of files, total bytes that overflowed in range, total bytes that overflowed in this range as a % of total bytes that overflowed, total bytes that overflowed in this range or smaller as a % of total bytes that overflowed. count=160 avg=1.90 KB min=0.00 KB max=14.31 KB [ 0- 2 KB): 67 (41.88%) ( 41.88% cumulative) 184.38 KB (60.78%) ( 60.78% cumulative) [ 2- 4 KB): 6 ( 3.75%) ( 45.62% cumulative) 11.62 KB ( 3.83%) ( 64.61% cumulative) [ 4- 8 KB): 60 (37.50%) ( 83.12% cumulative) 6.54 KB ( 2.16%) ( 66.76% cumulative) [ 8- 16 KB): 4 ( 2.50%) ( 85.62% cumulative) 6.04 KB ( 1.99%) ( 68.75% cumulative) [ 16- 32 KB): 5 ( 3.12%) ( 88.75% cumulative) 6.08 KB ( 2.00%) ( 70.76% cumulative) [ 32- 64 KB): 8 ( 5.00%) ( 93.75% cumulative) 30.20 KB ( 9.96%) ( 80.71% cumulative) [ 64- 128 KB): 3 ( 1.88%) ( 95.62% cumulative) 15.03 KB ( 4.95%) ( 85.67% cumulative) [ 128- 256 KB): 3 ( 1.88%) ( 97.50% cumulative) 13.36 KB ( 4.40%) ( 90.07% cumulative) [ 512-1024 KB): 2 ( 1.25%) ( 98.75% cumulative) 9.88 KB ( 3.26%) ( 93.33% cumulative) [1024-2048 KB): 1 ( 0.62%) ( 99.38% cumulative) 5.93 KB ( 1.95%) ( 95.28% cumulative) [4096-8192 KB): 1 ( 0.62%) (100.00% cumulative) 14.31 KB ( 4.72%) (100.00% cumulative) negative overhead (Files whose size is larger than capacity used. Includes directories, symlinks and regular files): Range in file size, count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of files, total bytes that underflowed in range, total bytes that underflowed in range as a % of total bytes that underflowed, total bytes that underflowed in this range or smaller as a % total bytes that underflowed. count=0 avg=0.00 KB min=0.00 KB max=0.00 KB directory size (entries): Range of entries, count of directories in range, number of dirs in range as a % of total num of dirs, number of dirs in this range or smaller as a % total number of dirs, total entries in range, number of entries in range as a % of total number of entries, number of entries in this range or smaller as a % of total number of entries. count=52 avg=3.06 ents min=0.00 ents max=57.00 ents [ 0- 1 ents]: 29 (55.77%) ( 55.77% cumulative) 18.00 ents (11.32%) ( 11.32% cumulative) [ 2- 3 ents]: 17 (32.69%) ( 88.46% cumulative) 36.00 ents (22.64%) ( 33.96% cumulative) [ 4- 7 ents]: 4 ( 7.69%) ( 96.15% cumulative) 16.00 ents (10.06%) ( 44.03% cumulative) [32-63 ents]: 2 ( 3.85%) (100.00% cumulative) 89.00 ents (55.97%) (100.00% cumulative) directory size (KB's): Range of dir sizes, count of directories in range, number of dirs in range as a % of total num of dirs, number of dirs in this range or smaller as a % total number of dirs, total size of dirs in range, total size of dirs in range as a % of total size of dirs, total size of dirs in this range or smaller as a % of total size of dirs. count=52 avg=4.00 KB min=4.00 KB max=4.00 KB [4-8 KB): 52 (100.00%) (100.00% cumulative) 208.00 KB (100.00%) (100.00% cumulative) filename length (Includes regular files, dirs, symlinks and special files): Range of filename lengths, count of files in range, count of files in range as a % of total number of files, total bytes of filenames in range, total bytes of filenames in range as a % of total number of bytes of filenames, total number of bytes of filenames in this range or smaller as a % of total number bytes of filenames. count=160 avg=11.62 chars min=1.00 chars max=38.00 chars [ 0- 7 chars]: 37 (23.12%) ( 23.12% cumulative) 211.00 chars (11.34%) ( 11.34% cumulative) [ 8-15 chars]: 94 (58.75%) ( 81.88% cumulative) 1034.00 chars (55.59%) ( 66.94% cumulative) [16-23 chars]: 23 (14.37%) ( 96.25% cumulative) 432.00 chars (23.23%) ( 90.16% cumulative) [24-31 chars]: 4 ( 2.50%) ( 98.75% cumulative) 113.00 chars ( 6.08%) ( 96.24% cumulative) [32-39 chars]: 2 ( 1.25%) (100.00% cumulative) 70.00 chars ( 3.76%) (100.00% cumulative) link count(of regular files): Range of link counts, count of files in range, number of files in range as a % of total files, number of files in this range or smaller as a % of total files, total links in range, total links in range as a % of total links, total links in this range or smaller as a % of total links. count=108 avg=1.00 links min=1.00 links max=1.00 links [0-1 links]: 108 (100.00%) (100.00% cumulative) 108.00 links (100.00%) (100.00% cumulative) symlink target length: Range of symlink file size, count of files in range, number of files in range as a % of total files, number of files in this range or smaller as a % of total files, total size of targets in range, total size of targets in range as a % of total size of targets, total size of targets in this range or smaller as a % of total size of targets. count=0 avg=0.00 chars min=0.00 chars max=0.00 chars symlink targets: 0.00% relative, 0.00% absolute time since last modification (Reguar files):Range in days, count of files in range, number of files in range as a % of total number of files, number of files in this range or smaller as a % of total number of files, total of age in range, total age in range as a % of total age of all files, total age in this range or smaller as a % of total age of all files. count=108 avg=269.49 days min=0.00 days max=1476.77 days [ 0- 2 days): 3 ( 2.78%) ( 2.78% cumulative) 0.13 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 2 ( 1.85%) ( 4.63% cumulative) 22.60 days ( 0.08%) ( 0.08% cumulative) [ 64- 128 days): 2 ( 1.85%) ( 6.48% cumulative) 241.54 days ( 0.83%) ( 0.91% cumulative) [ 128- 256 days): 93 (86.11%) ( 92.59% cumulative) 17601.39 days (60.48%) ( 61.38% cumulative) [1024-2048 days): 8 ( 7.41%) (100.00% cumulative) 11239.17 days (38.62%) (100.00% cumulative) time since last modification (size of regular files by age): Range in days, count of bytes whose age is in range, count of bytes in range as a % total number of bytes, count of bytes in this range or smaller as a % of total number of bytes, total days in range (each weighted by its size), today weighted days in range as a % of total weighted days, total days in this range or smaller as a % of total weighted days. count=10620 avg=162.97 days min=0.00 days max=1476.77 days [ 0- 2 days): 125 ( 1.18%) ( 1.18% cumulative) 14.15 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 35 ( 0.33%) ( 1.52% cumulative) 324.91 days ( 0.02%) ( 0.02% cumulative) [ 64- 128 days): 0 ( 0.00%) ( 1.52% cumulative) 58.97 days ( 0.00%) ( 0.02% cumulative) [ 128- 256 days): 10394 (97.87%) ( 99.39% cumulative) 1652205.22 days (95.46%) ( 95.48% cumulative) [1024-2048 days): 64 ( 0.61%) (100.00% cumulative) 78258.43 days ( 4.52%) (100.00% cumulative) time since creation (Reguar files):Range in days, count of files in range, number of files in range as a % of total number of files, number of files in this range or smaller as a % of total number of files, total of age in range, total age in range as a % of total age of all files, total age in this range or smaller as a % of total age of all files. count=108 avg=123.38 days min=0.00 days max=133.80 days [ 0- 2 days): 6 ( 5.56%) ( 5.56% cumulative) 0.14 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 2 ( 1.85%) ( 7.41% cumulative) 22.60 days ( 0.17%) ( 0.17% cumulative) [ 64-128 days): 2 ( 1.85%) ( 9.26% cumulative) 241.54 days ( 1.81%) ( 1.98% cumulative) [128-256 days): 98 (90.74%) (100.00% cumulative) 13060.63 days (98.02%) (100.00% cumulative) time since creation (size of regular files by age): Range in days, count of bytes whose age is in range, count of bytes in range as a % total number of bytes, count of bytes in this range or smaller as a % of total number of bytes, total days in range (each weighted by its size), today weighted days in range as a % of total weighted days, total days in this range or smaller as a % of total weighted days. count=10620 avg=128.20 days min=0.00 days max=133.80 days [ 0- 2 days): 189 ( 1.78%) ( 1.78% cumulative) 14.17 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 35 ( 0.33%) ( 2.12% cumulative) 324.91 days ( 0.02%) ( 0.02% cumulative) [ 64-128 days): 0 ( 0.00%) ( 2.12% cumulative) 58.97 days ( 0.00%) ( 0.03% cumulative) [128-256 days): 10395 (97.88%) (100.00% cumulative) 1361186.59 days (99.97%) (100.00% cumulative) time since last access (Reguar files):Range in days, count of files in range, number of files in range as a % of total number of files, number of files in this range or smaller as a % of total number of files, total of age in range, total age in range as a % of total age of all files, total age in this range or smaller as a age,0.000000,chars min,0,chars max,0,chars bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct relative symlink target pct,0.000000 absolute symlink target pct,0.000000 histogram,mtime (files) count,108,items average,269.489178,days min,0,days max,1476,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,3,0.027778,0.027778,0.134664,0.000005,0.000005 8,16,2,0.018519,0.046296,22.597153,0.000776,0.000781 64,128,2,0.018519,0.064815,241.538322,0.008299,0.009080 128,256,93,0.861111,0.925926,17601.394433,0.604759,0.613838 1024,2048,8,0.074074,1.000000,11239.166667,0.386162,1.000000 histogram,mtime (KB) count,10620,items average,162.971823,days min,0,days max,1476,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,125,0.011825,0.011825,14.145070,0.000008,0.000008 8,16,35,0.003346,0.015171,324.905894,0.000188,0.000196 64,128,0,0.000046,0.015217,58.969313,0.000034,0.000230 128,256,10394,0.978717,0.993935,1652205.215533,0.954556,0.954786 1024,2048,64,0.006065,1.000000,78258.428154,0.045214,1.000000 histogram,ctime (files) count,108,items average,123.378713,days min,0,days max,133,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,6,0.055556,0.055556,0.135637,0.000010,0.000010 8,16,2,0.018519,0.074074,22.597153,0.001696,0.001706 64,128,2,0.018519,0.092593,241.538322,0.018127,0.019833 128,256,98,0.907407,1.000000,13060.629861,0.980167,1.000000 histogram,ctime (KB) count,10620,items average,128.202001,days min,0,days max,133,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,189,0.017833,0.017833,14.165750,0.000010,0.000010 8,16,35,0.003346,0.021180,324.905894,0.000239,0.000249 64,128,0,0.000046,0.021226,58.969313,0.000043,0.000292 128,256,10395,0.978774,1.000000,1361186.585965,0.999708,1.000000 histogram,atime (files) count,108,items average,112.781134,days min,0,days max,133,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,11,0.101852,0.101852,0.252789,0.000021,0.000021 2,4,1,0.009259,0.111111,3.150567,0.000259,0.000279 8,16,2,0.018519,0.129630,22.597326,0.001855,0.002135 64,128,8,0.074074,0.203704,667.235093,0.054780,0.056914 128,256,86,0.796296,1.000000,11487.126725,0.943086,1.000000 histogram,atime (KB) count,10620,items average,129.314495,days min,0,days max,133,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,125,0.011798,0.011798,0.212883,0.000000,0.000000 2,4,0,0.000082,0.011880,2.756746,0.000002,0.000002 8,16,2,0.000207,0.012087,30.055238,0.000022,0.000024 64,128,5,0.000496,0.012583,398.196445,0.000290,0.000314 128,256,10486,0.987417,1.000000,1372968.781739,0.999686,1.000000 % of total age of all files. count=108 avg=112.78 days min=0.00 days max=133.81 days [ 0- 2 days): 11 (10.19%) ( 10.19% cumulative) 0.25 days ( 0.00%) ( 0.00% cumulative) [ 2- 4 days): 1 ( 0.93%) ( 11.11% cumulative) 3.15 days ( 0.03%) ( 0.03% cumulative) [ 8- 16 days): 2 ( 1.85%) ( 12.96% cumulative) 22.60 days ( 0.19%) ( 0.21% cumulative) [ 64-128 days): 8 ( 7.41%) ( 20.37% cumulative) 667.24 days ( 5.48%) ( 5.69% cumulative) [128-256 days): 86 (79.63%) (100.00% cumulative) 11487.13 days (94.31%) (100.00% cumulative) time since access (size of regular files by age): Range in days, count of bytes whose age is in range, count of bytes in range as a % total number of bytes, count of bytes in this range or smaller as a % of total number of bytes, total days in range (each weighted by its size), today weighted days in range as a % of total weighted days, total days in this range or smaller as a % of total weighted days. count=10620 avg=129.31 days min=0.00 days max=133.81 days [ 0- 2 days): 125 ( 1.18%) ( 1.18% cumulative) 0.21 days ( 0.00%) ( 0.00% cumulative) [ 2- 4 days): 0 ( 0.01%) ( 1.19% cumulative) 2.76 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 2 ( 0.02%) ( 1.21% cumulative) 30.06 days ( 0.00%) ( 0.00% cumulative) [ 64-128 days): 5 ( 0.05%) ( 1.26% cumulative) 398.20 days ( 0.03%) ( 0.03% cumulative) [128-256 days): 10486 (98.74%) (100.00% cumulative) 1372968.78 days (99.97%) (100.00% cumulative) ----- END NORMAL ----- -------------- next part -------------- fsstats v1.4.5 (03/22/2008) Copyright (c) 2005, 2007 Panasas, Inc. fsstats: checkpointing every 10 minutes to /tmp/fsstats.XNMqZ ----- CUT HERE ----- REPORT FOLLOWS ----- CUT HERE ----- Generated by fsstats v1.4.5 (03/22/2008) complete : processed 177 files in 0 secs #Generated by fsstats v1.4.5 (03/22/2008) #Comment: This is a comment line that can be modified or repeated before #uploading to record voluntarily added information. skipped special files,0 skipped duplicate hardlinks,0 skipped snapshot dirs,0 total capacity used,4.13 GB total user data,4.13 GB percent overhead,0.001079 histogram,file size count,125,items average,34636.909227,KB min,0,KB max,1121089,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,52,0.416000,0.416000,16.776367,0.000004,0.000004 2,4,10,0.080000,0.496000,23.101562,0.000005,0.000009 4,8,9,0.072000,0.568000,50.756836,0.000012,0.000021 8,16,2,0.016000,0.584000,31.125977,0.000007,0.000028 16,32,5,0.040000,0.624000,89.585938,0.000021,0.000049 32,64,12,0.096000,0.720000,598.371094,0.000138,0.000187 64,128,12,0.096000,0.816000,1088.425781,0.000251,0.000438 128,256,10,0.080000,0.896000,1682.211914,0.000389,0.000827 256,512,2,0.016000,0.912000,705.442383,0.000163,0.000990 512,1024,1,0.008000,0.920000,720.912109,0.000167,0.001156 1024,2048,3,0.024000,0.944000,4329.050781,0.001000,0.002156 8192,16384,1,0.008000,0.952000,10100.000000,0.002333,0.004489 131072,262144,1,0.008000,0.960000,210727.285156,0.048671,0.053160 262144,524288,2,0.016000,0.976000,737342.406250,0.170302,0.223462 1048576,2097152,3,0.024000,1.000000,3362108.201172,0.776538,1.000000 histogram,capacity used count,125,items average,34674.336000,KB min,0,KB max,1122192,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,11,0.088000,0.088000,0.000000,0.000000,0.000000 4,8,52,0.416000,0.504000,208.000000,0.000048,0.000048 8,16,8,0.064000,0.568000,64.000000,0.000015,0.000063 16,32,7,0.056000,0.624000,128.000000,0.000030,0.000092 32,64,8,0.064000,0.688000,384.000000,0.000089,0.000181 64,128,14,0.112000,0.800000,1168.000000,0.000269,0.000450 128,256,12,0.096000,0.896000,1996.000000,0.000461,0.000911 256,512,2,0.016000,0.912000,720.000000,0.000166,0.001077 512,1024,1,0.008000,0.920000,728.000000,0.000168,0.001245 1024,2048,3,0.024000,0.944000,4348.000000,0.001003,0.002248 8192,16384,1,0.008000,0.952000,10116.000000,0.002334,0.004582 131072,262144,1,0.008000,0.960000,210940.000000,0.048668,0.053250 262144,524288,2,0.016000,0.976000,738076.000000,0.170288,0.223537 1048576,2097152,3,0.024000,1.000000,3365416.000000,0.776463,1.000000 histogram,positive overhead count,177,items average,26.431337,KB min,0,KB max,1102,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,52,0.293785,0.293785,147.223633,0.031469,0.031469 2,4,10,0.056497,0.350282,16.898438,0.003612,0.035081 4,8,61,0.344633,0.694915,17.243164,0.003686,0.038767 8,16,2,0.011299,0.706215,0.874023,0.000187,0.038954 16,32,5,0.028249,0.734463,6.414062,0.001371,0.040325 32,64,12,0.067797,0.802260,53.628906,0.011463,0.051788 64,128,12,0.067797,0.870056,71.574219,0.015299,0.067087 128,256,10,0.056497,0.926554,53.788086,0.011497,0.078584 256,512,2,0.011299,0.937853,14.557617,0.003112,0.081696 512,1024,1,0.005650,0.943503,7.087891,0.001515,0.083211 1024,2048,3,0.016949,0.960452,18.949219,0.004050,0.087261 8192,16384,1,0.005650,0.966102,16.000000,0.003420,0.090681 131072,262144,1,0.005650,0.971751,212.714844,0.045468,0.136149 262144,524288,2,0.011299,0.983051,733.593750,0.156806,0.292956 1048576,2097152,3,0.016949,1.000000,3307.798828,0.707044,1.000000 histogram,negative overhead count,0,items average,0.000000,KB min,0,KB max,0,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct histogram,directory size (entries) count,52,items average,3.384615,ents min,0,ents max,53,ents bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,1,29,0.557692,0.557692,19.000000,0.107955,0.107955 2,3,17,0.326923,0.884615,35.000000,0.198864,0.306818 4,7,3,0.057692,0.942308,13.000000,0.073864,0.380682 16,31,1,0.019231,0.961538,24.000000,0.136364,0.517045 32,63,2,0.038462,1.000000,85.000000,0.482955,1.000000 histogram,directory size count,52,items average,4.000000,KB min,4,KB max,4,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 4,8,52,1.000000,1.000000,208.000000,1.000000,1.000000 histogram,filename length count,177,items average,11.621469,chars min,1,chars max,38,chars bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,7,38,0.214689,0.214689,222.000000,0.107924,0.107924 8,15,108,0.610169,0.824859,1188.000000,0.577540,0.685464 16,23,25,0.141243,0.966102,466.000000,0.226544,0.912008 24,31,4,0.022599,0.988701,111.000000,0.053962,0.965970 32,39,2,0.011299,1.000000,70.000000,0.034030,1.000000 histogram,link count count,125,items average,1.000000,links min,1,links max,1,links bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,1,125,1.000000,1.000000,125.000000,1.000000,1.000000 histogram,symlink target length count,0,items average,0.000000,chars min,0,chars max,0,chars bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct relative symlink target pct,0.000000 absolute symlink target pct,0.000000 histogram,mtime (files) count,125,items average,587.425578,days min,0,days max,1476,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,3,0.024000,0.024000,0.086632,0.000001,0.000001 8,16,1,0.008000,0.032000,8.779931,0.000120,0.000121 32,64,1,0.008000,0.040000,56.723912,0.000773,0.000893 64,128,2,0.016000,0.056000,197.997905,0.002696,0.003590 128,256,6,0.048000,0.104000,1360.101748,0.018523,0.022113 256,512,7,0.056000,0.160000,2188.198958,0.029801,0.051913 512,1024,97,0.776000,0.936000,58377.148576,0.795024,0.846937 1024,2048,8,0.064000,1.000000,11239.159630,0.153063,1.000000 histogram,mtime (KB) count,4329613,items average,318.663291,days min,0,days max,1476,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,416,0.000096,0.000096,33.727150,0.000000,0.000000 8,16,15,0.000004,0.000100,138.481112,0.000000,0.000000 32,64,0,0.000000,0.000100,15.621234,0.000000,0.000000 64,128,6,0.000001,0.000101,621.224514,0.000000,0.000001 128,256,105,0.000024,0.000126,23875.082325,0.000017,0.000018 256,512,4100706,0.947130,0.947256,1249295168.890066,0.905490,0.905508 512,1024,228298,0.052729,0.999985,130290824.096532,0.094435,0.999943 1024,2048,64,0.000015,1.000000,78258.371490,0.000057,1.000000 histogram,ctime (files) count,125,items average,494.162416,days min,0,days max,568,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,6,0.048000,0.048000,0.089792,0.000001,0.000001 8,16,1,0.008000,0.056000,8.779931,0.000142,0.000144 32,64,1,0.008000,0.064000,56.723912,0.000918,0.001062 64,128,2,0.016000,0.080000,197.997905,0.003205,0.004267 128,256,6,0.048000,0.128000,1360.101748,0.022019,0.026286 256,512,7,0.056000,0.184000,2188.198958,0.035425,0.061711 512,1024,102,0.816000,1.000000,57958.409815,0.938289,1.000000 histogram,ctime (KB) count,4329613,items average,318.500276,days min,0,days max,568,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,480,0.000111,0.000111,33.794360,0.000000,0.000000 8,16,15,0.000004,0.000115,138.481112,0.000000,0.000000 32,64,0,0.000000,0.000115,15.621234,0.000000,0.000000 64,128,6,0.000001,0.000116,621.224514,0.000000,0.000001 128,256,105,0.000024,0.000140,23875.082325,0.000017,0.000018 256,512,4100706,0.947130,0.947270,1249295168.890066,0.905954,0.905972 512,1024,228298,0.052730,1.000000,129663288.883868,0.094028,1.000000 histogram,atime (files) count,125,items average,466.049218,days min,0,days max,568,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,11,0.088000,0.088000,0.204861,0.000004,0.000004 8,16,2,0.016000,0.104000,18.801389,0.000323,0.000326 32,64,1,0.008000,0.112000,56.724884,0.000974,0.001300 64,128,3,0.024000,0.136000,313.714387,0.005385,0.006685 128,256,5,0.040000,0.176000,1129.614120,0.019390,0.026076 256,512,7,0.056000,0.232000,2186.884942,0.037539,0.063615 512,1024,96,0.768000,1.000000,54550.207639,0.936385,1.000000 histogram,atime (KB) count,4329613,items ----- BEGIN CSV ----- ----- END CSV ----- ----- BEGIN NORMAL ----- total 4.13 GB used to store 4.13 GB user data, overhead 0.11% file size (Regular file's last addressable byte): Range, Count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of files, total size of files in range, total size of files in range as a % of total size of files, total size of files in this range or smaller as a % of total size of files. count=125 avg=34636.91 KB min=0.00 KB max=1121089.39 KB [ 0- 2 KB): 52 (41.60%) ( 41.60% cumulative) 16.78 KB ( 0.00%) ( 0.00% cumulative) [ 2- 4 KB): 10 ( 8.00%) ( 49.60% cumulative) 23.10 KB ( 0.00%) ( 0.00% cumulative) [ 4- 8 KB): 9 ( 7.20%) ( 56.80% cumulative) 50.76 KB ( 0.00%) ( 0.00% cumulative) [ 8- 16 KB): 2 ( 1.60%) ( 58.40% cumulative) 31.13 KB ( 0.00%) ( 0.00% cumulative) [ 16- 32 KB): 5 ( 4.00%) ( 62.40% cumulative) 89.59 KB ( 0.00%) ( 0.00% cumulative) [ 32- 64 KB): 12 ( 9.60%) ( 72.00% cumulative) 598.37 KB ( 0.01%) ( 0.02% cumulative) [ 64- 128 KB): 12 ( 9.60%) ( 81.60% cumulative) 1088.43 KB ( 0.03%) ( 0.04% cumulative) [ 128- 256 KB): 10 ( 8.00%) ( 89.60% cumulative) 1682.21 KB ( 0.04%) ( 0.08% cumulative) [ 256- 512 KB): 2 ( 1.60%) ( 91.20% cumulative) 705.44 KB ( 0.02%) ( 0.10% cumulative) [ 512- 1024 KB): 1 ( 0.80%) ( 92.00% cumulative) 720.91 KB ( 0.02%) ( 0.12% cumulative) [ 1024- 2048 KB): 3 ( 2.40%) ( 94.40% cumulative) 4329.05 KB ( 0.10%) ( 0.22% cumulative) [ 8192- 16384 KB): 1 ( 0.80%) ( 95.20% cumulative) 10100.00 KB ( 0.23%) ( 0.45% cumulative) [ 131072- 262144 KB): 1 ( 0.80%) ( 96.00% cumulative) 210727.29 KB ( 4.87%) ( 5.32% cumulative) [ 262144- 524288 KB): 2 ( 1.60%) ( 97.60% cumulative) 737342.41 KB (17.03%) ( 22.35% cumulative) [1048576-2097152 KB): 3 ( 2.40%) (100.00% cumulative) 3362108.20 KB (77.65%) (100.00% cumulative) capacity used (Regualr file's space used on disk): Range, Count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of file, total capacity used in range, total capacity used in range as a % of total capacity used, total capacity used in this range or smaller as a % of total capacity. count=125 avg=34674.34 KB min=0.00 KB max=1122192.00 KB [ 0- 2 KB): 11 ( 8.80%) ( 8.80% cumulative) 0.00 KB ( 0.00%) ( 0.00% cumulative) [ 4- 8 KB): 52 (41.60%) ( 50.40% cumulative) 208.00 KB ( 0.00%) ( 0.00% cumulative) [ 8- 16 KB): 8 ( 6.40%) ( 56.80% cumulative) 64.00 KB ( 0.00%) ( 0.01% cumulative) [ 16- 32 KB): 7 ( 5.60%) ( 62.40% cumulative) 128.00 KB ( 0.00%) ( 0.01% cumulative) [ 32- 64 KB): 8 ( 6.40%) ( 68.80% cumulative) 384.00 KB ( 0.01%) ( 0.02% cumulative) [ 64- 128 KB): 14 (11.20%) ( 80.00% cumulative) 1168.00 KB ( 0.03%) ( 0.05% cumulative) [ 128- 256 KB): 12 ( 9.60%) ( 89.60% cumulative) 1996.00 KB ( 0.05%) ( 0.09% cumulative) [ 256- 512 KB): 2 ( 1.60%) ( 91.20% cumulative) 720.00 KB ( 0.02%) ( 0.11% cumulative) [ 512- 1024 KB): 1 ( 0.80%) ( 92.00% cumulative) 728.00 KB ( 0.02%) ( 0.12% cumulative) [ 1024- 2048 KB): 3 ( 2.40%) ( 94.40% cumulative) 4348.00 KB ( 0.10%) ( 0.22% cumulative) [ 8192- 16384 KB): 1 ( 0.80%) ( 95.20% cumulative) 10116.00 KB ( 0.23%) ( 0.46% cumulative) [ 131072- 262144 KB): 1 ( 0.80%) ( 96.00% cumulative) 210940.00 KB ( 4.87%) ( 5.32% cumulative) [ 262144- 524288 KB): 2 ( 1.60%) ( 97.60% cumulative) 738076.00 KB (17.03%) ( 22.35% cumulative) [1048576-2097152 KB): 3 ( 2.40%) (100.00% cumulative) 3365416.00 KB (77.65%) (100.00% cumulative) positive overhead (Files whose capacity used is larger than or equal to its size. Includes directories, symlinks and regular files): Range of file sizes, count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of files, total bytes that overflowed in range, total bytes that overflowed in this range as a % of total bytes that overflowed, total bytes that overflowed in this range or smaller as a % of total bytes that overflowed. count=177 avg=26.43 KB min=0.00 KB max=1102.61 KB [ 0- 2 KB): 52 (29.38%) ( 29.38% cumulative) 147.22 KB ( 3.15%) ( 3.15% cumulative) [ 2- 4 KB): 10 ( 5.65%) ( 35.03% cumulative) 16.90 KB ( 0.36%) ( 3.51% cumulative) [ 4- 8 KB): 61 (34.46%) ( 69.49% cumulative) 17.24 KB ( 0.37%) ( 3.88% cumulative) [ 8- 16 KB): 2 ( 1.13%) ( 70.62% cumulative) 0.87 KB ( 0.02%) ( 3.90% cumulative) [ 16- 32 KB): 5 ( 2.82%) ( 73.45% cumulative) 6.41 KB ( 0.14%) ( 4.03% cumulative) [ 32- 64 KB): 12 ( 6.78%) ( 80.23% cumulative) 53.63 KB ( 1.15%) ( 5.18% cumulative) [ 64- 128 KB): 12 ( 6.78%) ( 87.01% cumulative) 71.57 KB ( 1.53%) ( 6.71% cumulative) [ 128- 256 KB): 10 ( 5.65%) ( 92.66% cumulative) 53.79 KB ( 1.15%) ( 7.86% cumulative) [ 256- 512 KB): 2 ( 1.13%) ( 93.79% cumulative) 14.56 KB ( 0.31%) ( 8.17% cumulative) [ 512- 1024 KB): 1 ( 0.56%) ( 94.35% cumulative) 7.09 KB ( 0.15%) ( 8.32% cumulative) [ 1024- 2048 KB): 3 ( 1.69%) ( 96.05% cumulative) 18.95 KB ( 0.41%) ( 8.73% cumulative) [ 8192- 16384 KB): 1 ( 0.56%) ( 96.61% cumulative) 16.00 KB ( 0.34%) ( 9.07% cumulative) [ 131072- 262144 KB): 1 ( 0.56%) ( 97.18% cumulative) 212.71 KB ( 4.55%) ( 13.61% cumulative) [ 262144- 524288 KB): 2 ( 1.13%) ( 98.31% cumulative) 733.59 KB (15.68%) ( 29.30% cumulative) [1048576-2097152 KB): 3 ( 1.69%) (100.00% cumulative) 3307.80 KB (70.70%) (100.00% cumulative) negative overhead (Files whose size is larger than capacity used. Includes directories, symlinks and regular files): Range in file size, count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of files, total bytes that underflowed in range, total bytes that underflowed in range as a % of total bytes that underflowed, total bytes that underflowed in this range or smaller as a % total bytes that underflowed. count=0 avg=0.00 KB min=0.00 KB max=0.00 KB directory size (entries): Range of entries, count of directories in range, number of dirs in range as a % of total num of dirs, number of dirs in this range or smaller as a % total number of dirs, total entries in range, number of entries in range as a % of total number of entries, number of entries in this range or smaller as a % of total number of entries. count=52 avg=3.38 ents min=0.00 ents max=53.00 ents [ 0- 1 ents]: 29 (55.77%) ( 55.77% cumulative) 19.00 ents (10.80%) ( 10.80% cumulative) [ 2- 3 ents]: 17 (32.69%) ( 88.46% cumulative) 35.00 ents (19.89%) ( 30.68% cumulative) [ 4- 7 ents]: 3 ( 5.77%) ( 94.23% cumulative) 13.00 ents ( 7.39%) ( 38.07% cumulative) [16-31 ents]: 1 ( 1.92%) ( 96.15% cumulative) 24.00 ents (13.64%) ( 51.70% cumulative) [32-63 ents]: 2 ( 3.85%) (100.00% cumulative) 85.00 ents (48.30%) (100.00% cumulative) directory size (KB's): Range of dir sizes, count of directories in range, number of dirs in range as a % of total num of dirs, number of dirs in this range or smaller as a % total number of dirs, total size of dirs in range, total size of dirs in range as a % of total size of dirs, total size of dirs in this range or smaller as a % of total size of dirs. count=52 avg=4.00 KB min=4.00 KB max=4.00 KB [4-8 KB): 52 (100.00%) (100.00% cumulative) 208.00 KB (100.00%) (100.00% cumulative) filename length (Includes regular files, dirs, symlinks and special files): Range of filename lengths, count of files in range, count of files in range as a % of total number of files, total bytes of filenames in range, total bytes of filenames in range as a % of total number of bytes of filenames, total number of bytes of filenames in this range or smaller as a % of total number bytes of filenames. count=177 avg=11.62 chars min=1.00 chars max=38.00 chars [ 0- 7 chars]: 38 (21.47%) ( 21.47% cumulative) 222.00 chars (10.79%) ( 10.79% cumulative) [ 8-15 chars]: 108 (61.02%) ( 82.49% cumulative) 1188.00 chars (57.75%) ( 68.55% cumulative) [16-23 chars]: 25 (14.12%) ( 96.61% cumulative) 466.00 chars (22.65%) ( 91.20% cumulative) [24-31 chars]: 4 ( 2.26%) ( 98.87% cumulative) 111.00 chars ( 5.40%) ( 96.60% cumulative) [32-39 chars]: 2 ( 1.13%) (100.00% cumulative) 70.00 chars ( 3.40%) (100.00% cumulative) link count(of regular files): Range of link counts, count of files in range, number of files in range as a % of total files, number of files in this range or smaller as a % of total files, total links in range, total links in range as a % of total links, total links in this range or smaller as a % of total links. count=125 avg=1.00 links min=1.00 links max=1.00 links [0-1 links]: 125 (100.00%) (100.00% cumulative) 125.00 links (100.00%) (100.00% cumulative) symlink target length: Range of symlink file size, count of files in range, number of files in range as a % of total files, number of files in this range or smaller as a % of total files, total size of targets in range, total size of targets in range as a % of total size of targets, total size of targets in this range or smaller as a % of total size of targets. count=0 avg=0.00 chars min=0.00 chars max=0.00 chars symlink targets: 0.00% relative, 0.00% absolute time since last modification (Reguar files):Range in days, count of files in range, number of files in range as a % of total number of files, number of files in this range or smaller as a % of total number of files, total of age in range, total age in range as a % of total age of all files, total age in this range or smaller as a % of total age of all files. count=125 avg=587.43 days min=0.00 days max=1476.77 days [ 0- 2 days): 3 ( 2.40%) ( 2.40% cumulative) 0.09 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 1 ( 0.80%) ( 3.20% cumulative) 8.78 days ( 0.01%) ( 0.01% cumulative) [ 32- 64 days): 1 ( 0.80%) ( 4.00% cumulative) 56.72 days ( 0.08%) ( 0.09% cumulative) [ 64- 128 days): 2 ( 1.60%) ( 5.60% cumulative) 198.00 days ( 0.27%) ( 0.36% cumulative) [ 128- 256 days): 6 ( 4.80%) ( 10.40% cumulative) 1360.10 days ( 1.85%) ( 2.21% cumulative) [ 256- 512 days): 7 ( 5.60%) ( 16.00% cumulative) 2188.20 days ( 2.98%) ( 5.19% cumulative) [ 512-1024 days): 97 (77.60%) ( 93.60% cumulative) 58377.15 days (79.50%) ( 84.69% cumulative) [1024-2048 days): 8 ( 6.40%) (100.00% cumulative) 11239.16 days (15.31%) (100.00% cumulative) time since last modification (size of regular files by age): Range in days, count of bytes whose age is in range, count of bytes in range as a % total number of bytes, count of bytes in this range or smaller as a % of total number of bytes, total days in range (each weighted by its size), today weighted days in range as a % of total weighted days, total days in this range or smaller as a % of total weighted days. count=4329613 avg=318.66 days min=0.00 days max=1476.77 days [ 0- 2 days): 416 ( 0.01%) ( 0.01% cumulative) 33.73 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 15 ( 0.00%) ( 0.01% cumulative) 138.48 days ( 0.00%) ( 0.00% cumulative) [ 32- 64 days): 0 ( 0.00%) ( 0.01% cumulative) 15.62 days ( 0.00%) ( 0.00% cumulative) [ 64- 128 days): 6 ( 0.00%) ( 0.01% cumulative) 621.22 days ( 0.00%) ( 0.00% cumulative) [ 128- 256 days): 105 ( 0.00%) ( 0.01% cumulative) 23875.08 days ( 0.00%) ( 0.00% cumulative) [ 256- 512 days): 4100706 (94.71%) ( 94.73% cumulative) 1249295168.89 days (90.55%) ( 90.55% cumulative) [ 512-1024 days): 228298 ( 5.27%) (100.00% cumulative) 130290824.10 days ( 9.44%) ( 99.99% cumulative) [1024-2048 days): 64 ( 0.00%) (100.00% cumulative) 78258.37 days ( 0.01%) (100.00% cumulative) time since creation (Reguar files):Range in days, count of files in range, number of files in range as a % of total number of files, number of files in this range or smaller as a % of total number of files, total of age in range, total age in range as a % of total age of all files, total age in this range or smaller as a % of total age of all files. count=125 avg=494.16 days min=0.00 days max=568.24 days [ 0- 2 days): 6 ( 4.80%) ( 4.80% cumulative) 0.09 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 1 ( 0.80%) ( 5.60% cumulative) 8.78 days ( 0.01%) ( 0.01% cumulative) [ 32- 64 days): 1 ( 0.80%) ( 6.40% cumulative) 56.72 days ( 0.09%) ( 0.11% cumulative) [ 64- 128 days): 2 ( 1.60%) ( 8.00% cumulative) 198.00 days ( 0.32%) ( 0.43% cumulative) [ 128- 256 days): 6 ( 4.80%) ( 12.80% cumulative) 1360.10 days ( 2.20%) ( 2.63% cumulative) [ 256- 512 days): 7 ( 5.60%) ( 18.40% cumulative) 2188.20 days ( 3.54%) ( 6.17% cumulative) [ 512-1024 days): 102 (81.60%) (100.00% cumulative) 57958.41 days (93.83%) (100.00% cumulative) time since creation (size of regular files by age): Range in days, count of bytes whose age is in range, count of bytes in range as a % total number of bytes, count of bytes in this range or smaller as a % of total number of bytes, total days in range (each weighted by its size), today weighted days in range as a % of total weighted days, total days in this range or smaller as a % of total weighted days. count=4329613 avg=318.50 days min=0.00 days max=568.24 days [ 0- 2 days): 480 ( 0.01%) ( 0.01% cumulative) 33.79 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 15 ( 0.00%) ( 0.01% cumulative) 138.48 days ( 0.00%) ( 0.00% cumulative) [ 32- 64 days): 0 ( 0.00%) ( 0.01% cumulative) 15.62 days ( 0.00%) ( 0.00% cumulative) [ 64- 128 days): 6 ( 0.00%) ( 0.01% cumulative) 621.22 days ( 0.00%) ( 0.00% cumulative) [ 128- 256 days): 105 ( 0.00%) ( 0.01% cumulative) 23875.08 days ( 0.00%) ( 0.00% cumulative) [ 256- 512 days): 4100706 (94.71%) ( 94.73% cumulative) 1249295168.89 days (90.60%) ( 90.60% cumulative) [ 512-1024 days): 228298 ( 5.27%) (100.00% cumulative) 129663288.88 days ( 9.40%) (100.00% cumulative) time since last access (Reguar files):Range in days, count of files in range, number of files in range as a % of total number of files, number of files in this range or smaller as a % of total number of files, total of age in range, total age in range as a % of total age of all files, total age in this range or smaller as a % of total age of all files. count=125 avg=466.05 days min=0.00 days max=568.24 days [ 0- 2 days): 11 ( 8.80%) ( 8.80% cumulative) 0.20 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 2 ( 1.60%) ( 10.40% cumulative) 18.80 days ( 0.03%) ( 0.03% cumulative) [ 32- 64 days): 1 ( 0.80%) ( 11.20% cumulative) 56.72 days ( 0.10%) ( 0.13% cumulative) [ 64- 128 days): 3 ( 2.40%) ( 13.60% cumulative) 313.71 days ( 0.54%) ( 0.67% cumulative) [ 128- 256 days): 5 ( 4.00%) ( 17.60% cumulative) 1129.61 days ( 1.94%) ( 2.61% cumulative) [ 256- 512 days): 7 ( 5.60%) ( 23.20% cumulative) 2186.88 days ( 3.75%) ( 6.36% cumulative) [ 512-1024 days): 96 (76.80%) (100.00% cumulative) 54550.21 days (93.64%) (100.00% cumulative) time since access (size of regular files by age): Range in days, count of bytes whose age is in range, count of bytes in range as a % total number of bytes, count of bytes in this range or smaller as a % of total number of bytes, total days in range (each weighted by its size), today weighted days in range as a % of total weighted days, total days in this range or smaller as a % of total weighted days. couaverage,318.210761,days min,0,days max,568,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,101,0.000024,0.000024,0.202194,0.000000,0.000000 8,16,0,0.000000,0.000024,4.394436,0.000000,0.000000 32,64,0,0.000000,0.000024,15.621501,0.000000,0.000000 64,128,6,0.000002,0.000025,667.215018,0.000000,0.000000 128,256,104,0.000024,0.000049,23514.947691,0.000017,0.000018 256,512,4100706,0.947130,0.947179,1247825750.863767,0.905712,0.905729 512,1024,228693,0.052821,1.000000,129879703.762636,0.094271,1.000000 nt=4329613 avg=318.21 days min=0.00 days max=568.24 days [ 0- 2 days): 101 ( 0.00%) ( 0.00% cumulative) 0.20 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 0 ( 0.00%) ( 0.00% cumulative) 4.39 days ( 0.00%) ( 0.00% cumulative) [ 32- 64 days): 0 ( 0.00%) ( 0.00% cumulative) 15.62 days ( 0.00%) ( 0.00% cumulative) [ 64- 128 days): 6 ( 0.00%) ( 0.00% cumulative) 667.22 days ( 0.00%) ( 0.00% cumulative) [ 128- 256 days): 104 ( 0.00%) ( 0.00% cumulative) 23514.95 days ( 0.00%) ( 0.00% cumulative) [ 256- 512 days): 4100706 (94.71%) ( 94.72% cumulative) 1247825750.86 days (90.57%) ( 90.57% cumulative) [ 512-1024 days): 228693 ( 5.28%) (100.00% cumulative) 129879703.76 days ( 9.43%) (100.00% cumulative) ----- END NORMAL ----- From adilger at whamcloud.com Fri Apr 22 17:06:22 2011 From: adilger at whamcloud.com (Andreas Dilger) Date: Fri, 22 Apr 2011 11:06:22 -0600 Subject: [Lustre-devel] 64-bit ext4 inodes In-Reply-To: <28DD036A-3277-4D65-87A5-ADB9ECF7B51C@xyratex.com> References: <2368DAF2-4CA9-41E4-8A56-E48AEA9D4549@xyratex.com> <73AED5C780AE05478241DB067651A92102378476@XYUS-EX22.xyus.xyratex.com> <50A7CDE4-73B3-4BEB-9BF4-BE0D077A3A00@whamcloud.com> <5E5695A6-CE4E-4253-B97C-85FAD3C96FFF@llnl.gov> <28DD036A-3277-4D65-87A5-ADB9ECF7B51C@xyratex.com> Message-ID: <711A9E1C-4C86-4871-97DA-7DC2362BB652@whamcloud.com> On 2011-04-22, at 9:33 AM, Alexey Lyashkov wrote: > Andreas, is possible to check different inodes groups on different cores in parallel? > It's should be dramatically increase time of checking on systems with 8-16 CPU cores. Yes, all of pass1 in e2fsck is trivially run in parallel. It would probably be best to make it event driven, by reading chunks from disk (maybe whole groups) and then having a task queue to process each chunk on a different core. The main effort would be adding locking to the data structures from many threads. Some of the other passes could also be parallelized, like tree walking, but getting good balance of disk and CPU usage is less trivial. Cheers, Andreas > On Apr 22, 2011, at 19:13, D. Marc Stearman wrote: > >> Andreas, LLNL often has file systems with more than 500M inodes in use. We create them with 1B inodes just for that reason. And yes, e2fsck times are horrible. Very painful. >> >> -Marc >> >> ---- >> D. Marc Stearman >> Lustre Operations Lead >> marc at llnl.gov >> 925.423.9670 >> Pager: 1.888.203.0641 >> >> >> >> >> On Apr 21, 2011, at 3:56 PM, Andreas Dilger wrote: >> >>> On Apr 20, 2011, at 11:07 AM, Nathan Rutman wrote: >>>> We're about to start looking into 64-bit inodes on ext4 -- anybody else working on this at the moment? >>> >>> I don't know of anyone other than Fujitsu that are looking at this. I haven't had any kind of technical discussion with them about it, nor looked at their code, but just implied that this is what they did based on their slides. >>> >>> One important consideration to note with 64-bit inode numbers is that they cannot be used safely on 1.8 MDS filesystems. That is because the IGIF FID namespace only has room for 2^32 inode numbers (mapped into the 2.x FID SEQ field, see ), so upgrading a 1.8 MDS with 64-bit inode numbers to 2.x would cause a huge world of hurt. If this is limited to 2.x filesystems that only identify MDS inodes via FIDs to the clients then this is not a concern. >>> >>> If you are concerned with being able to upgrade 32-bit inode filesystems into 64-bit inode filesystems, you should look at the data-in-dirent patch that is currently in the 2.x ldiskfs patchset. It was developed to allow storing the 128-bit Lustre FID in the directory entry in a compatible manner, but was also designed to allow storing the high 32 bits of a 64-bit inode number into the directory entry. That allows compatibility for upgrades by avoiding the need to atomically upgrade a whole directory full of dirents to have 64-bit inode number fields. It also avoids the need to store 64-bit inode numbers when most of them are 32-bit values. >>> >>> >>> I'm not against the idea of exploring this approach, but I am concerned that e2fsck time will continue to grow with the number of inodes in a single filesystem, and scalability of a single MDT will increasingly be an issue. >>> >>> AFAIK, e2fsck times are currently on the order of 1h/100M inodes, and no Lustre filesystems that I know of are above 500M inodes today just due to the average file size being so large and the long e2fsck times. With improvements in ext4 to reduce e2fsck time like flex_bg, and SSDs, it may be possible to reduce the e2fsck time/inode ratio noticeably, but I think it would take more effort on the e2fsck side than the ext4 side to make many billions of inodes in a single MDT a practical approach. >>> >>> Things like metadata prefetching that Dave Dillow was doing for e2scan with event-driven completion handlers that process the blocks whenever they arrive from the disk, and multi-threading some of the passes of e2fsck with an understanding of the underlying disk layout (using the s_raid_stride and s_raid_stripe_width). >>> >>> Cheers, Andreas >>> -- >>> Andreas Dilger >>> Principal Engineer >>> Whamcloud, Inc. >>> >>> >>> >>> _______________________________________________ >>> Lustre-devel mailing list >>> Lustre-devel at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-devel >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel > > -------------------------------------------- > Alexey Lyashkov > alexey_lyashkov at xyratex.com > > > > > ______________________________________________________________________ > This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. > > Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. > > Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. > > The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. > ______________________________________________________________________ > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From C.J.Walker at qmul.ac.uk Fri Apr 22 18:47:56 2011 From: C.J.Walker at qmul.ac.uk (Christopher J. Walker) Date: Fri, 22 Apr 2011 19:47:56 +0100 Subject: [Lustre-devel] Research on filesystem metadata operation distribution In-Reply-To: References: Message-ID: <4DB1CD5C.6080205@qmul.ac.uk> On 21/04/11 19:40, Andreas Dilger wrote: > I'm trying to get some data about the relative distribution of MDS operations in the wild, and I'd be grateful if some people with production filesystems that have been running for at least a week could collect some simple stats and email them to me. They can be collected by any regular user on the MDS node: The filesystem lustre_0 (lustre_1 is used for small tests) is used for storage of data from the LHC at CERN. We are a WLCG Tier-2 site and uses the "StoRM" SRM implementation. The filesystem is mostly used for storage of datasets around 2Gig in size, but the software area is also on Lustre (except for our main customer, ATLAS, where the mds was a bottleneck, so we now use the CVMFS caching filesystem for them). The filesystem has been up 18 days, but under somewhat reduced load from normal due to problems with our airconditioning. > > lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync" > [root at sn01 ~]# lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync" open 6483556050 samples [reqs] close 298937510 samples [reqs] link 207 samples [reqs] unlink 1642197 samples [reqs] rename 74790 samples [reqs] getattr 604826669 samples [reqs] setattr 732241 samples [reqs] getxattr 2500316 samples [reqs] setxattr 2869526 samples [reqs] sync 734 samples [reqs] open 78316 samples [reqs] close 22048 samples [reqs] getattr 139519 samples [reqs] getxattr 1 samples [reqs] > It would be useful to also include "lfs df" and "lfs df -i" information, as well as a brief description of what the filesystem is used for (scratch, home, project, archive, etc). [root at fe08 ~]# lfs df UUID 1K-blocks Used Available Use% Mounted on lustre_0-MDT0000_UUID 1279344884 16915292 1262429592 1% /mnt/lustre_0[MDT:0] lustre_0-OST0000_UUID 6486115712 5350727580 1135388004 82% /mnt/lustre_0[OST:0] lustre_0-OST0001_UUID 6486115712 5254948988 1231166596 81% /mnt/lustre_0[OST:1] lustre_0-OST0002_UUID 6486115712 5369069128 1117046456 82% /mnt/lustre_0[OST:2] lustre_0-OST0003_UUID 6486115712 5312081284 1174034300 81% /mnt/lustre_0[OST:3] lustre_0-OST0004_UUID 6486115712 5313935872 1172179776 81% /mnt/lustre_0[OST:4] lustre_0-OST0005_UUID 6486115712 5386002008 1100113576 83% /mnt/lustre_0[OST:5] lustre_0-OST0006_UUID 6486115712 5269309656 1216805928 81% /mnt/lustre_0[OST:6] lustre_0-OST0007_UUID 6486115712 5328370940 1157744644 82% /mnt/lustre_0[OST:7] lustre_0-OST0008_UUID 6486115712 5347529932 1138585652 82% /mnt/lustre_0[OST:8] lustre_0-OST0009_UUID 6486115712 5341504240 1144611344 82% /mnt/lustre_0[OST:9] lustre_0-OST000a_UUID 6486115712 5333294764 1152820820 82% /mnt/lustre_0[OST:10] lustre_0-OST000b_UUID 6486115712 5372246460 1113869124 82% /mnt/lustre_0[OST:11] lustre_0-OST000c_UUID 6486115712 5285503076 1200611484 81% /mnt/lustre_0[OST:12] lustre_0-OST000d_UUID 6486115712 5289776224 1196339360 81% /mnt/lustre_0[OST:13] lustre_0-OST000e_UUID 6486115712 5367755168 1118360416 82% /mnt/lustre_0[OST:14] lustre_0-OST000f_UUID 6486115712 5178737856 1307377792 79% /mnt/lustre_0[OST:15] lustre_0-OST0010_UUID 6486115712 5301360576 1184755072 81% /mnt/lustre_0[OST:16] lustre_0-OST0011_UUID 6486115712 5246933508 1239182076 80% /mnt/lustre_0[OST:17] lustre_0-OST0012_UUID 6486115712 5340756896 1145358688 82% /mnt/lustre_0[OST:18] lustre_0-OST0013_UUID 6486115712 5227080140 1259035444 80% /mnt/lustre_0[OST:19] lustre_0-OST0014_UUID 6486115712 5324360452 1161755132 82% /mnt/lustre_0[OST:20] lustre_0-OST0015_UUID 6486115712 5304634880 1181480704 81% /mnt/lustre_0[OST:21] lustre_0-OST0016_UUID 6486115712 5342578956 1143536628 82% /mnt/lustre_0[OST:22] lustre_0-OST0017_UUID 6486115712 5352854748 1133260900 82% /mnt/lustre_0[OST:23] lustre_0-OST0018_UUID 6486115712 5341708760 1144406760 82% /mnt/lustre_0[OST:24] lustre_0-OST0019_UUID 6486115712 5332733132 1153382452 82% /mnt/lustre_0[OST:25] lustre_0-OST001a_UUID 6486115712 5366956328 1119159256 82% /mnt/lustre_0[OST:26] lustre_0-OST001b_UUID 6486115712 5251430884 1234684764 80% /mnt/lustre_0[OST:27] lustre_0-OST001c_UUID 6486115712 5166970324 1319145260 79% /mnt/lustre_0[OST:28] lustre_0-OST001d_UUID 6486115712 5344269684 1141845900 82% /mnt/lustre_0[OST:29] lustre_0-OST001e_UUID 6486115712 5230794024 1255321624 80% /mnt/lustre_0[OST:30] lustre_0-OST001f_UUID 6486115712 5402916936 1083198648 83% /mnt/lustre_0[OST:31] lustre_0-OST0020_UUID 6486115712 5263884340 1222231308 81% /mnt/lustre_0[OST:32] lustre_0-OST0021_UUID 6486115712 5376336376 1109778888 82% /mnt/lustre_0[OST:33] lustre_0-OST0022_UUID 6486115712 5347838416 1138277168 82% /mnt/lustre_0[OST:34] lustre_0-OST0023_UUID 6486115712 5215511904 1270603744 80% /mnt/lustre_0[OST:35] lustre_0-OST0024_UUID 6486115712 5366457356 1119658228 82% /mnt/lustre_0[OST:36] lustre_0-OST0025_UUID 6486115712 5292414492 1193701092 81% /mnt/lustre_0[OST:37] lustre_0-OST0026_UUID 6486115712 5271663256 1214452328 81% /mnt/lustre_0[OST:38] lustre_0-OST0027_UUID 6486115712 5263536792 1222578792 81% /mnt/lustre_0[OST:39] lustre_0-OST0028_UUID 6486115712 5340305084 1145810500 82% /mnt/lustre_0[OST:40] lustre_0-OST0029_UUID 6486115712 5394190360 1091925224 83% /mnt/lustre_0[OST:41] lustre_0-OST002a_UUID 6486115712 5335685244 1150429316 82% /mnt/lustre_0[OST:42] lustre_0-OST002b_UUID 6486115712 5106503104 1379612480 78% /mnt/lustre_0[OST:43] lustre_0-OST002c_UUID 6486115712 5152124056 1333991592 79% /mnt/lustre_0[OST:44] lustre_0-OST002d_UUID 6486115712 5288720908 1197394740 81% /mnt/lustre_0[OST:45] lustre_0-OST002e_UUID 6486115712 5065686264 1420429384 78% /mnt/lustre_0[OST:46] lustre_0-OST002f_UUID 6486115712 5370371776 1115743872 82% /mnt/lustre_0[OST:47] filesystem summary: 311333554176 254430363132 56903183236 81% /mnt/lustre_0 UUID 1K-blocks Used Available Use% Mounted on lustre_1-MDT0000_UUID 1279344884 539236 1278805648 0% /mnt/lustre_1[MDT:0] lustre_1-OST0000_UUID 7930215400 4302650476 3627564860 54% /mnt/lustre_1[OST:0] lustre_1-OST0001_UUID 7930215400 4327747852 3602467484 54% /mnt/lustre_1[OST:1] lustre_1-OST0002_UUID 7930215400 4424687248 3505528088 55% /mnt/lustre_1[OST:2] lustre_1-OST0003_UUID 7930215400 4233067620 3697147716 53% /mnt/lustre_1[OST:3] filesystem summary: 31720861600 17288153196 14432708148 54% /mnt/lustre_1 [root at fe08 ~]# lfs df -i UUID Inodes IUsed IFree IUse% Mounted on lustre_0-MDT0000_UUID 344890671 29283273 315607398 8% /mnt/lustre_0[MDT:0] lustre_0-OST0000_UUID 284348159 501126 283847033 0% /mnt/lustre_0[OST:0] lustre_0-OST0001_UUID 308308277 516596 307791681 0% /mnt/lustre_0[OST:1] lustre_0-OST0002_UUID 279762988 501342 279261646 0% /mnt/lustre_0[OST:2] lustre_0-OST0003_UUID 294035715 527108 293508607 0% /mnt/lustre_0[OST:3] lustre_0-OST0004_UUID 293569946 524986 293044960 0% /mnt/lustre_0[OST:4] lustre_0-OST0005_UUID 275499122 470696 275028426 0% /mnt/lustre_0[OST:5] lustre_0-OST0006_UUID 304750117 548603 304201514 0% /mnt/lustre_0[OST:6] lustre_0-OST0007_UUID 289986672 550479 289436193 0% /mnt/lustre_0[OST:7] lustre_0-OST0008_UUID 285181040 534595 284646445 0% /mnt/lustre_0[OST:8] lustre_0-OST0009_UUID 286682099 529231 286152868 0% /mnt/lustre_0[OST:9] lustre_0-OST000a_UUID 288750615 545378 288205237 0% /mnt/lustre_0[OST:10] lustre_0-OST000b_UUID 278999894 532581 278467313 0% /mnt/lustre_0[OST:11] lustre_0-OST000c_UUID 300687110 533951 300153159 0% /mnt/lustre_0[OST:12] lustre_0-OST000d_UUID 299617234 532362 299084872 0% /mnt/lustre_0[OST:13] lustre_0-OST000e_UUID 280075762 485626 279590136 0% /mnt/lustre_0[OST:14] lustre_0-OST000f_UUID 327385455 540991 326844464 0% /mnt/lustre_0[OST:15] lustre_0-OST0010_UUID 296705745 516961 296188784 0% /mnt/lustre_0[OST:16] lustre_0-OST0011_UUID 310345458 549907 309795551 0% /mnt/lustre_0[OST:17] lustre_0-OST0012_UUID 286866260 526556 286339704 0% /mnt/lustre_0[OST:18] lustre_0-OST0013_UUID 315308235 549342 314758893 0% /mnt/lustre_0[OST:19] lustre_0-OST0014_UUID 290967510 528695 290438815 0% /mnt/lustre_0[OST:20] lustre_0-OST0015_UUID 295903066 532858 295370208 0% /mnt/lustre_0[OST:21] lustre_0-OST0016_UUID 286398762 514573 285884189 0% /mnt/lustre_0[OST:22] lustre_0-OST0017_UUID 283848065 532824 283315241 0% /mnt/lustre_0[OST:23] lustre_0-OST0018_UUID 286663505 561767 286101738 0% /mnt/lustre_0[OST:24] lustre_0-OST0019_UUID 288896723 551078 288345645 0% /mnt/lustre_0[OST:25] lustre_0-OST001a_UUID 280340068 550222 279789846 0% /mnt/lustre_0[OST:26] lustre_0-OST001b_UUID 309218724 547517 308671207 0% /mnt/lustre_0[OST:27] lustre_0-OST001c_UUID 330304954 518607 329786347 0% /mnt/lustre_0[OST:28] lustre_0-OST001d_UUID 285967857 506350 285461507 0% /mnt/lustre_0[OST:29] lustre_0-OST001e_UUID 314350650 520228 313830422 0% /mnt/lustre_0[OST:30] lustre_0-OST001f_UUID 271317367 517673 270799694 0% /mnt/lustre_0[OST:31] lustre_0-OST0020_UUID 306106565 548722 305557843 0% /mnt/lustre_0[OST:32] lustre_0-OST0021_UUID 277913188 468354 277444834 0% /mnt/lustre_0[OST:33] lustre_0-OST0022_UUID 285070365 501041 284569324 0% /mnt/lustre_0[OST:34] lustre_0-OST0023_UUID 318210059 559107 317650952 0% /mnt/lustre_0[OST:35] lustre_0-OST0024_UUID 280428253 513664 279914589 0% /mnt/lustre_0[OST:36] lustre_0-OST0025_UUID 298958894 533589 298425305 0% /mnt/lustre_0[OST:37] lustre_0-OST0026_UUID 304151287 538173 303613114 0% /mnt/lustre_0[OST:38] lustre_0-OST0027_UUID 306168127 523397 305644730 0% /mnt/lustre_0[OST:39] lustre_0-OST0028_UUID 286925194 472537 286452657 0% /mnt/lustre_0[OST:40] lustre_0-OST0029_UUID 273508145 526807 272981338 0% /mnt/lustre_0[OST:41] lustre_0-OST002a_UUID 288090585 482968 287607617 0% /mnt/lustre_0[OST:42] lustre_0-OST002b_UUID 345436810 533658 344903152 0% /mnt/lustre_0[OST:43] lustre_0-OST002c_UUID 334033778 535864 333497914 0% /mnt/lustre_0[OST:44] lustre_0-OST002d_UUID 299895732 547031 299348701 0% /mnt/lustre_0[OST:45] lustre_0-OST002e_UUID 355630432 523070 355107362 0% /mnt/lustre_0[OST:46] lustre_0-OST002f_UUID 279499831 563847 278935984 0% /mnt/lustre_0[OST:47] filesystem summary: 344890671 29283273 315607398 8% /mnt/lustre_0 UUID Inodes IUsed IFree IUse% Mounted on lustre_1-MDT0000_UUID 319898525 197113 319701412 0% /mnt/lustre_1[MDT:0] lustre_1-OST0000_UUID 503545856 48975 503496881 0% /mnt/lustre_1[OST:0] lustre_1-OST0001_UUID 503545856 49081 503496775 0% /mnt/lustre_1[OST:1] lustre_1-OST0002_UUID 503545856 49098 503496758 0% /mnt/lustre_1[OST:2] lustre_1-OST0003_UUID 503545856 49133 503496723 0% /mnt/lustre_1[OST:3] filesystem summary: 319898525 197113 319701412 0% /mnt/lustre_1 > > As a reminder, I'm also interested if some Lustre admins could run the "fsstats" tool from http://www.pdsi-scidac.org/fsstats/ and send me the output. Sending the output to PDSI via their submission form may also produce some positive results. > > http://www.pdsi-scidac.org/fsstats/files/fsstats-1.4.5.tar.gz > > > Thanks in advance for any data. I've set replies to go only to lustre-devel, to avoid clogging the larger readership of lustre-discuss, but it may be useful for others to have this in a list archive and/or searchable via Google in the future so I don't necessarily want to keep it all to myself. Hope that's useful for you. Chris PS I'm a different Chris Walker from the one who posted from Harvard... -- Dr Christopher J. Walker School of Physics Queen Mary, University of London From Nathan_Rutman at xyratex.com Fri Apr 22 22:41:35 2011 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Fri, 22 Apr 2011 15:41:35 -0700 Subject: [Lustre-devel] Research on filesystem metadata operation distribution In-Reply-To: References: Message-ID: <73AED5C780AE05478241DB067651A92102378482@XYUS-EX22.xyus.xyratex.com> Hmm - also 'uptime' would really be nice to have with these so we could estimate rough rates... On Apr 21, 2011, at 11:40 AM, Andreas Dilger wrote: > I'm trying to get some data about the relative distribution of MDS operations in the wild, and I'd be grateful if some people with production filesystems that have been running for at least a week could collect some simple stats and email them to me. They can be collected by any regular user on the MDS node: > > lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync" > > It would be useful to also include "lfs df" and "lfs df -i" information, as well as a brief description of what the filesystem is used for (scratch, home, project, archive, etc). > > > > As a reminder, I'm also interested if some Lustre admins could run the "fsstats" tool from http://www.pdsi-scidac.org/fsstats/ and send me the output. Sending the output to PDSI via their submission form may also produce some positive results. > > http://www.pdsi-scidac.org/fsstats/files/fsstats-1.4.5.tar.gz > > > Thanks in advance for any data. I've set replies to go only to lustre-devel, to avoid clogging the larger readership of lustre-discuss, but it may be useful for others to have this in a list archive and/or searchable via Google in the future so I don't necessarily want to keep it all to myself. > > Cheers, Andreas > -- > Andreas Dilger > Principal Engineer > Whamcloud, Inc. > > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________ From nirmal at fnal.gov Mon Apr 25 15:45:22 2011 From: nirmal at fnal.gov (Nirmal Seenu) Date: Mon, 25 Apr 2011 10:45:22 -0500 Subject: [Lustre-devel] Research on filesystem metadata operation distribution In-Reply-To: References: Message-ID: <4DB59712.8030907@fnal.gov> Attached is the stats from the cosmology cluster lustre file system. We use this filesystem as a general purpose storage system to store the outputs, binaries and code. The storage space is not backed up and users have to write the files manually to the tape if needed. Nirmal -------------- next part -------------- We use this filesystem for project and code storage without any backup. Users need to write these files manually to the tape if they need it a backup uptime 10:32:41 up 13 days, 14 min, 1 user, load average: 0.29, 0.46, 0.46 lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync" open 25405022 samples [reqs] close 9065872 samples [reqs] unlink 93666 samples [reqs] rename 12657 samples [reqs] getattr 178749678 samples [reqs] setattr 109256 samples [reqs] sync 121422 samples [reqs] lfs df output: UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 110092260 3352428 106739832 3% /lustre[MDT:0] lustre-OST0000_UUID 2884114628 2501561636 382552892 86% /lustre[OST:0] lustre-OST0001_UUID 2884114628 2527433380 356681044 87% /lustre[OST:1] lustre-OST0002_UUID 2884114628 2498952164 385162388 86% /lustre[OST:2] lustre-OST0003_UUID 2884114628 2490509464 393605056 86% /lustre[OST:3] lustre-OST0004_UUID 2884114628 2504063232 380051304 86% /lustre[OST:4] lustre-OST0005_UUID 2884114628 2489176372 394938176 86% /lustre[OST:5] lustre-OST0006_UUID 2884114628 2530726436 353386688 87% /lustre[OST:6] lustre-OST0007_UUID 2884114628 2486248244 397866316 86% /lustre[OST:7] lustre-OST0008_UUID 2884114628 2580039988 304073528 89% /lustre[OST:8] lustre-OST0009_UUID 2884085444 2409236584 474848852 83% /lustre[OST:9] lustre-OST000a_UUID 2884110020 2524712320 359397596 87% /lustre[OST:10] lustre-OST000b_UUID 2884110020 2471606976 412502964 85% /lustre[OST:11] lustre-OST000c_UUID 2884114628 2506496588 377617960 86% /lustre[OST:12] lustre-OST000d_UUID 2884114628 2386694152 497420256 82% /lustre[OST:13] lustre-OST000e_UUID 2884114628 2512284264 371808772 87% /lustre[OST:14] lustre-OST000f_UUID 2884114628 2508435980 375678576 86% /lustre[OST:15] lustre-OST0010_UUID 2884114628 2528948392 355138500 87% /lustre[OST:16] lustre-OST0011_UUID 2884114628 2475264400 408850124 85% /lustre[OST:17] lustre-OST0012_UUID 2884114628 2477997436 406117108 85% /lustre[OST:18] lustre-OST0013_UUID 2884114628 2411330600 472783940 83% /lustre[OST:19] lustre-OST0014_UUID 2884114628 2452501392 431613096 85% /lustre[OST:20] lustre-OST0015_UUID 2884110020 2425470236 458632292 84% /lustre[OST:21] lustre-OST0016_UUID 2884110020 2506874060 377235872 86% /lustre[OST:22] lustre-OST0017_UUID 2884110020 2526339148 357770776 87% /lustre[OST:23] lustre-OST0018_UUID 2884114628 2494217480 389897076 86% /lustre[OST:24] lustre-OST0019_UUID 2884114628 2550866328 333230572 88% /lustre[OST:25] lustre-OST001a_UUID 2884114628 2392512412 491602136 82% /lustre[OST:26] lustre-OST001b_UUID 2884114628 2546456908 337631132 88% /lustre[OST:27] lustre-OST001c_UUID 2884114628 2433956940 450157616 84% /lustre[OST:28] lustre-OST001d_UUID 2884114628 2518257840 365856700 87% /lustre[OST:29] lustre-OST001e_UUID 2884114628 2386002076 498112548 82% /lustre[OST:30] lustre-OST001f_UUID 2884114628 2551645032 332464392 88% /lustre[OST:31] lustre-OST0020_UUID 2884110020 2508140676 375969260 86% /lustre[OST:32] lustre-OST0021_UUID 2884110020 2480057388 404052560 85% /lustre[OST:33] lustre-OST0022_UUID 2884110020 2523920168 360189768 87% /lustre[OST:34] lustre-OST0023_UUID 2884114628 2483034600 401079940 86% /lustre[OST:35] lustre-OST0024_UUID 2883950796 2447210584 436740112 84% /lustre[OST:36] lustre-OST0025_UUID 2883950796 2516207332 367743360 87% /lustre[OST:37] lustre-OST0026_UUID 2883950796 2437923532 446027176 84% /lustre[OST:38] lustre-OST0027_UUID 2883950796 2476331728 407618984 85% /lustre[OST:39] lustre-OST0028_UUID 2883950796 2432972916 450977800 84% /lustre[OST:40] lustre-OST0029_UUID 2883950796 2460940916 423009752 85% /lustre[OST:41] lustre-OST002a_UUID 2883950796 2468612140 415338568 85% /lustre[OST:42] lustre-OST002b_UUID 2883950796 2458854996 425095716 85% /lustre[OST:43] lustre-OST002c_UUID 2883950796 2513629572 370321132 87% /lustre[OST:44] lustre-OST002d_UUID 2883950796 2363972180 519978512 81% /lustre[OST:45] lustre-OST002e_UUID 2883950796 2565237744 318712004 88% /lustre[OST:46] lustre-OST002f_UUID 2883950796 2459814044 424136640 85% /lustre[OST:47] filesystem summary: 138435470112 119203678976 19231677532 86% /lustre 'lfs df -i' output: UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 31457280 21307244 10150036 67% /lustre[MDT:0] lustre-OST0000_UUID 96011367 373119 95638248 0% /lustre[OST:0] lustre-OST0001_UUID 89519754 349443 89170311 0% /lustre[OST:1] lustre-OST0002_UUID 96636795 346179 96290616 0% /lustre[OST:2] lustre-OST0003_UUID 98732018 330727 98401291 0% /lustre[OST:3] lustre-OST0004_UUID 95343096 330247 95012849 0% /lustre[OST:4] lustre-OST0005_UUID 99061693 327130 98734563 0% /lustre[OST:5] lustre-OST0006_UUID 88716029 368981 88347048 0% /lustre[OST:6] lustre-OST0007_UUID 99842063 375467 99466596 0% /lustre[OST:7] lustre-OST0008_UUID 76312606 293948 76018658 0% /lustre[OST:8] lustre-OST0009_UUID 119025859 313645 118712214 0% /lustre[OST:9] lustre-OST000a_UUID 90165802 316377 89849425 0% /lustre[OST:10] lustre-OST000b_UUID 103444857 319102 103125755 0% /lustre[OST:11] lustre-OST000c_UUID 94650672 246165 94404507 0% /lustre[OST:12] lustre-OST000d_UUID 124683131 328012 124355119 0% /lustre[OST:13] lustre-OST000e_UUID 93312935 355345 92957590 0% /lustre[OST:14] lustre-OST000f_UUID 94242336 322674 93919662 0% /lustre[OST:15] lustre-OST0010_UUID 89125796 334243 88791553 0% /lustre[OST:16] lustre-OST0011_UUID 102566152 353598 102212554 0% /lustre[OST:17] lustre-OST0012_UUID 101814880 285582 101529298 0% /lustre[OST:18] lustre-OST0013_UUID 118530077 334070 118196007 0% /lustre[OST:19] lustre-OST0014_UUID 108236231 332922 107903309 0% /lustre[OST:20] lustre-OST0015_UUID 114429201 330920 114098281 0% /lustre[OST:21] lustre-OST0016_UUID 94649031 340041 94308990 0% /lustre[OST:22] lustre-OST0017_UUID 89793722 351004 89442718 0% /lustre[OST:23] lustre-OST0018_UUID 97915782 441495 97474287 0% /lustre[OST:24] lustre-OST0019_UUID 83744982 432907 83312075 0% /lustre[OST:25] lustre-OST001a_UUID 123338441 437889 122900552 0% /lustre[OST:26] lustre-OST001b_UUID 84867995 453565 84414430 0% /lustre[OST:27] lustre-OST001c_UUID 112988355 448933 112539422 0% /lustre[OST:28] lustre-OST001d_UUID 91907484 443287 91464197 0% /lustre[OST:29] lustre-OST001e_UUID 124982574 454436 124528138 0% /lustre[OST:30] lustre-OST001f_UUID 83550431 433032 83117399 0% /lustre[OST:31] lustre-OST0020_UUID 94407277 414941 93992336 0% /lustre[OST:32] lustre-OST0021_UUID 101467596 454438 101013158 0% /lustre[OST:33] lustre-OST0022_UUID 90505148 457685 90047463 0% /lustre[OST:34] lustre-OST0023_UUID 100697142 427135 100270007 0% /lustre[OST:35] lustre-OST0024_UUID 109852270 667217 109185053 0% /lustre[OST:36] lustre-OST0025_UUID 92594348 658482 91935866 0% /lustre[OST:37] lustre-OST0026_UUID 112144143 637327 111506816 0% /lustre[OST:38] lustre-OST0027_UUID 102548080 643313 101904767 0% /lustre[OST:39] lustre-OST0028_UUID 113378346 633876 112744470 0% /lustre[OST:40] lustre-OST0029_UUID 106409522 657053 105752469 0% /lustre[OST:41] lustre-OST002a_UUID 104465812 631148 103834664 0% /lustre[OST:42] lustre-OST002b_UUID 106921742 647792 106273950 0% /lustre[OST:43] lustre-OST002c_UUID 93233739 653433 92580306 0% /lustre[OST:44] lustre-OST002d_UUID 130641098 646444 129994654 0% /lustre[OST:45] lustre-OST002e_UUID 80262443 584180 79678263 0% /lustre[OST:46] lustre-OST002f_UUID 106696994 662808 106034186 0% /lustre[OST:47] filesystem summary: 31457280 21307244 10150036 67% /lustre -------------- next part -------------- #Generated by fsstats v1.4.5 (03/22/2008) #Comment: This is a comment line that can be modified or repeated before #uploading to record voluntarily added information. skipped special files,0 skipped duplicate hardlinks,113 skipped snapshot dirs,0 total capacity used,110.60 TB total user data,110.57 TB percent overhead,0.000277 histogram,file size count,20963690,items average,5663.305612,KB min,0,KB max,88080384,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,9988853,0.476484,0.476484,10644331.002930,0.000090,0.000090 2,4,2211694,0.105501,0.581985,5117879.170898,0.000043,0.000133 4,8,434744,0.020738,0.602723,2163080.032227,0.000018,0.000151 8,16,3308049,0.157799,0.760522,30315021.214844,0.000255,0.000406 16,32,246322,0.011750,0.772272,5634038.108398,0.000047,0.000454 32,64,549850,0.026229,0.798500,27251246.360352,0.000230,0.000683 64,128,1294831,0.061765,0.860266,105694690.291992,0.000890,0.001574 128,256,1206324,0.057543,0.917809,175351267.156250,0.001477,0.003051 256,512,283007,0.013500,0.931309,116443388.930664,0.000981,0.004031 512,1024,287128,0.013696,0.945005,213799662.544922,0.001801,0.005832 1024,2048,209030,0.009971,0.954977,297877006.204102,0.002509,0.008341 2048,4096,237039,0.011307,0.966284,726121881.948242,0.006116,0.014457 4096,8192,258313,0.012322,0.978606,1459982570.916992,0.012297,0.026755 8192,16384,154643,0.007377,0.985982,1743146719.902344,0.014682,0.041437 16384,32768,113993,0.005438,0.991420,2569819867.442383,0.021645,0.063082 32768,65536,35536,0.001695,0.993115,1593541789.721680,0.013422,0.076505 65536,131072,35113,0.001675,0.994790,3273789717.922852,0.027575,0.104079 131072,262144,44823,0.002138,0.996928,8623199382.573242,0.072632,0.176712 262144,524288,32315,0.001541,0.998470,12103445450.947266,0.101946,0.278658 524288,1048576,12379,0.000590,0.999060,9369780063.988281,0.078921,0.357579 1048576,2097152,9696,0.000463,0.999523,13605014835.129883,0.114594,0.472173 2097152,4194304,5779,0.000276,0.999798,15634270433.628906,0.131686,0.603859 4194304,8388608,2282,0.000109,0.999907,13913493530.226562,0.117192,0.721051 8388608,16777216,1214,0.000058,0.999965,14072262149.836914,0.118529,0.839580 16777216,33554432,547,0.000026,0.999991,11903128580.764648,0.100259,0.939839 33554432,67108864,183,0.000009,1.000000,6888917675.968750,0.058025,0.997864 67108864,134217728,3,0.000000,1.000000,253576972.794922,0.002136,1.000000 histogram,capacity used count,20963690,items average,5664.845682,KB min,0,KB max,88080472,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,42157,0.002011,0.002011,0.000000,0.000000,0.000000 4,8,12158464,0.579977,0.581988,48633856.000000,0.000410,0.000410 8,16,3645112,0.173877,0.755866,42002104.000000,0.000354,0.000763 16,32,292124,0.013935,0.769800,6121624.000000,0.000052,0.000815 32,64,434226,0.020713,0.790514,19392576.000000,0.000163,0.000978 64,128,1458240,0.069560,0.860074,118267028.000000,0.000996,0.001974 128,256,1204428,0.057453,0.917527,176857552.000000,0.001489,0.003463 256,512,278244,0.013273,0.930800,113045348.000000,0.000952,0.004415 512,1024,296902,0.014163,0.944962,218931408.000000,0.001844,0.006259 1024,2048,209705,0.010003,0.954966,298580660.000000,0.002514,0.008773 2048,4096,237215,0.011316,0.966281,726260988.000000,0.006116,0.014888 4096,8192,258606,0.012336,0.978617,1461346880.000000,0.012305,0.027194 8192,16384,154662,0.007378,0.985995,1743312812.000000,0.014680,0.041874 16384,32768,113731,0.005425,0.991420,2564630540.000000,0.021596,0.063469 32768,65536,35545,0.001696,0.993115,1593927936.000000,0.013422,0.076891 65536,131072,35111,0.001675,0.994790,3273579896.000000,0.027566,0.104457 131072,262144,44821,0.002138,0.996928,8622937300.000000,0.072610,0.177067 262144,524288,32316,0.001542,0.998470,12103831248.000000,0.101922,0.278989 524288,1048576,12378,0.000590,0.999060,9369247428.000000,0.078895,0.357884 1048576,2097152,9695,0.000462,0.999523,13603725128.000000,0.114552,0.472436 2097152,4194304,5779,0.000276,0.999798,15634291672.000000,0.131650,0.604086 4194304,8388608,2284,0.000109,0.999907,13926107864.000000,0.117266,0.721353 8388608,16777216,1212,0.000058,0.999965,14045385560.000000,0.118271,0.839624 16777216,33554432,547,0.000026,0.999991,11903144228.000000,0.100232,0.939856 33554432,67108864,183,0.000009,1.000000,6888929992.000000,0.058009,0.997865 67108864,134217728,3,0.000000,1.000000,253577152.000000,0.002135,1.000000 histogram,positive overhead count,21227725,items average,2.644709,KB min,0,KB max,625,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,10133167,0.477355,0.477355,29708405.123047,0.529173,0.529173 2,4,2211684,0.104188,0.581544,3728886.710938,0.066420,0.595593 4,8,545885,0.025716,0.607259,1314591.967773,0.023416,0.619009 8,16,3309882,0.155923,0.763182,9772114.785156,0.174063,0.793072 16,32,248098,0.011687,0.774869,591345.891602,0.010533,0.803605 32,64,551037,0.025958,0.800828,1201073.639648,0.021394,0.824999 64,128,1295993,0.061052,0.861880,2378769.356445,0.042371,0.867370 128,256,1208447,0.056928,0.918808,2525560.843750,0.044986,0.912356 256,512,284001,0.013379,0.932186,590533.716797,0.010519,0.922875 512,1024,287152,0.013527,0.945713,673432.544922,0.011995,0.934870 1024,2048,209068,0.009849,0.955562,475323.030273,0.008467,0.943337 2048,4096,237067,0.011168,0.966730,524272.072266,0.009338,0.952675 4096,8192,258301,0.012168,0.978898,584040.312500,0.010403,0.963079 8192,16384,154590,0.007282,0.986181,533597.967773,0.009505,0.972583 16384,32768,113663,0.005354,0.991535,540955.000000,0.009636,0.982219 32768,65536,35518,0.001673,0.993208,161950.297852,0.002885,0.985103 65536,131072,35097,0.001653,0.994862,160957.964844,0.002867,0.987970 131072,262144,44814,0.002111,0.996973,219240.236328,0.003905,0.991876 262144,524288,32307,0.001522,0.998495,189286.739258,0.003372,0.995247 524288,1048576,12368,0.000583,0.999077,70185.215820,0.001250,0.996497 1048576,2097152,9656,0.000455,0.999532,59877.355469,0.001067,0.997564 2097152,4194304,5743,0.000271,0.999803,40567.538086,0.000723,0.998287 4194304,8388608,2282,0.000108,0.999910,31413.773438,0.000560,0.998846 8388608,16777216,1172,0.000055,0.999965,36640.081055,0.000653,0.999499 16777216,33554432,547,0.000026,0.999991,15647.235352,0.000279,0.999777 33554432,67108864,183,0.000009,1.000000,12316.031250,0.000219,0.999997 67108864,134217728,3,0.000000,1.000000,179.205078,0.000003,1.000000 histogram,negative overhead count,67109,items average,346.763837,KB min,0,KB max,6291448,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,66403,0.989480,0.989480,2755.120117,0.000118,0.000118 2,4,10,0.000149,0.989629,25.881836,0.000001,0.000120 64,128,82,0.001222,0.990851,5871.648438,0.000252,0.000372 128,256,1,0.000015,0.990866,4.000000,0.000000,0.000372 256,512,2,0.000030,0.990895,22.647461,0.000001,0.000373 512,1024,3,0.000045,0.990940,2023.089844,0.000087,0.000460 1024,2048,2,0.000030,0.990970,2689.234375,0.000116,0.000575 2048,4096,3,0.000045,0.991015,3090.020508,0.000133,0.000708 4096,8192,30,0.000447,0.991462,94251.229492,0.004050,0.004758 8192,16384,57,0.000849,0.992311,276325.870117,0.011874,0.016633 16384,32768,332,0.004947,0.997258,5947938.442383,0.255595,0.272227 32768,65536,18,0.000268,0.997526,162896.019531,0.007000,0.279227 65536,131072,18,0.000268,0.997795,212183.887695,0.009118,0.288345 131072,262144,11,0.000164,0.997959,219554.809570,0.009435,0.297780 262144,524288,8,0.000119,0.998078,65637.686523,0.002821,0.300601 524288,1048576,11,0.000164,0.998242,1461989.204102,0.062825,0.363425 1048576,2097152,40,0.000596,0.998838,464076.485352,0.019942,0.383368 2097152,4194304,36,0.000536,0.999374,19329.166992,0.000831,0.384198 8388608,16777216,42,0.000626,1.000000,14330309.917969,0.615802,1.000000 histogram,directory size (entries) count,120427,items average,176.828668,ents min,0,ents max,1065715,ents bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,1,37665,0.312762,0.312762,23330.000000,0.001096,0.001096 2,3,13417,0.111412,0.424174,35582.000000,0.001671,0.002766 4,7,18135,0.150589,0.574763,97690.000000,0.004587,0.007354 8,15,17283,0.143514,0.718277,184931.000000,0.008684,0.016038 16,31,11260,0.093501,0.811778,254356.000000,0.011944,0.027983 32,63,7903,0.065625,0.877403,321699.000000,0.015107,0.043089 64,127,4892,0.040622,0.918025,404844.000000,0.019011,0.062101 128,255,2557,0.021233,0.939258,401205.000000,0.018840,0.080941 256,511,1768,0.014681,0.953939,607223.000000,0.028515,0.109456 512,1023,1769,0.014689,0.968628,1609130.000000,0.075564,0.185020 1024,2047,728,0.006045,0.974673,1090462.000000,0.051208,0.236228 2048,4095,2156,0.017903,0.992576,5721820.000000,0.268694,0.504921 4096,8191,788,0.006543,0.999120,3643213.000000,0.171083,0.676005 8192,16383,25,0.000208,0.999327,284845.000000,0.013376,0.689381 16384,32767,29,0.000241,0.999568,638579.000000,0.029987,0.719368 32768,65535,39,0.000324,0.999892,1739267.000000,0.081675,0.801043 65536,131071,7,0.000058,0.999950,666197.000000,0.031284,0.832328 131072,262143,2,0.000017,0.999967,378730.000000,0.017785,0.850113 524288,1048575,3,0.000025,0.999992,2126128.000000,0.099842,0.949955 1048576,2097151,1,0.000008,1.000000,1065715.000000,0.050045,1.000000 histogram,directory size count,120427,items average,18.335788,KB min,4,KB max,165476,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 4,8,111141,0.922891,0.922891,444564.000000,0.201331,0.201331 8,16,1833,0.015221,0.938112,21996.000000,0.009961,0.211292 16,32,1776,0.014748,0.952859,36340.000000,0.016457,0.227750 32,64,1187,0.009857,0.962716,59960.000000,0.027154,0.254904 64,128,1244,0.010330,0.973046,101984.000000,0.046186,0.301090 128,256,2124,0.017637,0.990683,392092.000000,0.177568,0.478658 256,512,996,0.008271,0.998954,299504.000000,0.135637,0.614295 512,1024,27,0.000224,0.999178,18812.000000,0.008519,0.622815 1024,2048,40,0.000332,0.999510,54788.000000,0.024812,0.647627 2048,4096,31,0.000257,0.999767,92764.000000,0.042010,0.689637 4096,8192,18,0.000149,0.999917,94156.000000,0.042641,0.732278 8192,16384,4,0.000033,0.999950,45704.000000,0.020698,0.752976 16384,32768,2,0.000017,0.999967,42732.000000,0.019352,0.772328 65536,131072,2,0.000017,0.999983,199124.000000,0.090178,0.862506 131072,262144,2,0.000017,1.000000,303604.000000,0.137494,1.000000 histogram,filename length count,21294947,items average,46.032158,chars min,1,chars max,156,chars bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,7,175229,0.008229,0.008229,924669.000000,0.000943,0.000943 8,15,1114735,0.052347,0.060576,13382449.000000,0.013652,0.014595 16,23,1434266,0.067352,0.127928,28106864.000000,0.028673,0.043268 24,31,2511232,0.117926,0.245855,68798838.000000,0.070185,0.113453 32,39,6338462,0.297651,0.543506,236275028.000000,0.241035,0.354488 40,47,5029532,0.236184,0.779690,209186020.000000,0.213400,0.567888 48,55,537420,0.025237,0.804927,27317483.000000,0.027868,0.595756 56,63,220318,0.010346,0.815273,13214545.000000,0.013481,0.609237 64,71,197292,0.009265,0.824538,13334976.000000,0.013604,0.622840 72,79,354371,0.016641,0.841179,26738700.000000,0.027277,0.650118 80,87,557251,0.026168,0.867347,45984154.000000,0.046911,0.697028 88,95,224094,0.010523,0.877870,20193481.000000,0.020600,0.717629 96,103,820745,0.038542,0.916412,82492989.000000,0.084155,0.801784 104,111,1273133,0.059786,0.976198,136763921.000000,0.139519,0.941303 112,119,506700,0.023794,0.999992,57515919.000000,0.058675,0.999977 120,127,59,0.000003,0.999995,7385.000000,0.000008,0.999985 127,-1,108,0.000005,1.000000,14949.000000,0.000015,1.000000 histogram,link count count,20963690,items average,1.000005,links min,1,links max,2,links bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,1,20963577,0.999995,0.999995,20963577.000000,0.999989,0.999989 2,3,113,0.000005,1.000000,226.000000,0.000011,1.000000 histogram,symlink target length count,210717,items average,68.612642,chars min,3,chars max,138,chars bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,7,41,0.000195,0.000195,257.000000,0.000018,0.000018 8,15,3263,0.015485,0.015680,39471.000000,0.002730,0.002748 16,23,6853,0.032522,0.048202,136114.000000,0.009415,0.012162 24,31,2921,0.013862,0.062064,79673.000000,0.005511,0.017673 32,39,2997,0.014223,0.076287,104318.000000,0.007215,0.024888 40,47,23609,0.112041,0.188328,1024366.000000,0.070852,0.095740 48,55,17186,0.081560,0.269888,886356.000000,0.061306,0.157047 56,63,15383,0.073003,0.342891,904491.000000,0.062561,0.219607 64,71,21767,0.103300,0.446191,1490971.000000,0.103125,0.322732 72,79,40367,0.191570,0.637761,3059852.000000,0.211639,0.534372 80,87,37398,0.177480,0.815240,3107834.000000,0.214958,0.749330 88,95,33925,0.160998,0.976238,3091387.000000,0.213821,0.963151 96,103,3265,0.015495,0.991733,318424.000000,0.022024,0.985175 104,111,334,0.001585,0.993318,35626.000000,0.002464,0.987639 112,119,154,0.000731,0.994049,17538.000000,0.001213,0.988852 127,-1,1254,0.005951,1.000000,161172.000000,0.011148,1.000000 relative symlink target pct,0.674663 absolute symlink target pct,0.325337 histogram,mtime (files) count,20963690,items average,310.842496,days min,0,days max,7029,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,79767,0.003805,0.003805,79142.951053,0.000012,0.000012 2,4,9027,0.000431,0.004236,25495.192731,0.000004,0.000016 4,8,19124,0.000912,0.005148,109156.502407,0.000017,0.000033 8,16,37509,0.001789,0.006937,475464.953854,0.000073,0.000106 16,32,389482,0.018579,0.025516,9461692.732789,0.001452,0.001558 32,64,523782,0.024985,0.050501,25947289.194106,0.003982,0.005540 64,128,2061189,0.098322,0.148823,207545525.539402,0.031850,0.037389 128,256,5977541,0.285138,0.433961,1263408411.723960,0.193881,0.231270 256,512,10741570,0.512389,0.946350,4143022927.489875,0.635783,0.867054 512,1024,955608,0.045584,0.991934,607877139.164743,0.093284,0.960338 1024,2048,141555,0.006752,0.998686,179353945.527176,0.027523,0.987861 2048,4096,24340,0.001161,0.999848,64552190.062745,0.009906,0.997768 4096,8192,3196,0.000152,1.000000,14547335.569120,0.002232,1.000000 histogram,mtime (KB) count,118723783234,items average,294.511304,days min,0,days max,7029,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,1236757924,0.010417,0.010417,719011583.094063,0.000021,0.000021 2,4,475882662,0.004008,0.014425,1459775355.422004,0.000042,0.000062 4,8,591887628,0.004985,0.019411,3739535344.818440,0.000107,0.000169 8,16,2327165996,0.019602,0.039012,26334104519.789959,0.000753,0.000922 16,32,4893100685,0.041214,0.080227,129481373506.183884,0.003703,0.004626 32,64,13335761868,0.112326,0.192552,645522888775.950073,0.018462,0.023087 64,128,14740751501,0.124160,0.316713,1409022412219.913330,0.040298,0.063385 128,256,27552389401,0.232071,0.548784,4994069520533.343750,0.142829,0.206213 256,512,29807088715,0.251062,0.799846,11943093434429.128906,0.341568,0.547781 512,1024,22539139104,0.189845,0.989692,14482586438757.146484,0.414197,0.961978 1024,2048,1220517137,0.010280,0.999972,1318741760524.159424,0.037716,0.999693 2048,4096,3265143,0.000028,0.999999,10383292341.656340,0.000297,0.999990 4096,8192,75464,0.000001,1.000000,342685847.613674,0.000010,1.000000 histogram,ctime (files) count,20963690,items average,293.499559,days min,0,days max,756,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,79885,0.003811,0.003811,79253.452176,0.000013,0.000013 2,4,8969,0.000428,0.004238,25324.493831,0.000004,0.000017 4,8,19248,0.000918,0.005157,110009.418600,0.000018,0.000035 8,16,39373,0.001878,0.007035,482019.497986,0.000078,0.000113 16,32,398845,0.019026,0.026060,9744007.218830,0.001584,0.001697 32,64,933387,0.044524,0.070584,43362984.055687,0.007048,0.008745 64,128,1691530,0.080689,0.151273,163796346.927605,0.026621,0.035366 128,256,6256058,0.298424,0.449696,1309069684.422790,0.212759,0.248125 256,512,10846311,0.517386,0.967082,4199850472.147315,0.682588,0.930713 512,1024,690084,0.032918,1.000000,426313678.083243,0.069287,1.000000 histogram,ctime (KB) count,118723783234,items average,214.794244,days min,0,days max,756,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,1246698470,0.010501,0.010501,729569442.251664,0.000029,0.000029 2,4,481961760,0.004060,0.014560,1474982666.503769,0.000058,0.000086 4,8,582017976,0.004902,0.019463,3687306936.767932,0.000145,0.000231 8,16,2625264219,0.022112,0.041575,30385845844.160862,0.001192,0.001423 16,32,8814482438,0.074244,0.115819,241192027256.134796,0.009458,0.010881 32,64,16623978134,0.140022,0.255841,769631076173.646118,0.030180,0.041061 64,128,22709982418,0.191284,0.447125,2216919728983.784180,0.086934,0.127995 128,256,30508794533,0.256973,0.704098,5315988506304.361328,0.208460,0.336455 256,512,22806938658,0.192101,0.896199,9427857837022.681641,0.369703,0.706158 512,1024,12323664623,0.103801,1.000000,7493318423583.877930,0.293842,1.000000 histogram,atime (files) count,20963690,items average,297.298404,days min,0,days max,1483,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,130565,0.006228,0.006228,117658.829595,0.000019,0.000019 2,4,10270,0.000490,0.006718,30030.969745,0.000005,0.000024 4,8,19384,0.000925,0.007643,112254.735139,0.000018,0.000042 8,16,141566,0.006753,0.014396,1983381.157083,0.000318,0.000360 16,32,397433,0.018958,0.033354,8955804.972650,0.001437,0.001797 32,64,1324101,0.063162,0.096515,59205539.542065,0.009500,0.011296 64,128,1295361,0.061791,0.158306,126677772.920356,0.020325,0.031622 128,256,5915624,0.282184,0.440490,1258923268.003366,0.201994,0.233616 256,512,10700363,0.510424,0.950914,4130193012.787154,0.662689,0.896306 512,1024,1027185,0.048998,0.999912,644239643.664700,0.103368,0.999674 1024,2048,1838,0.000088,1.000000,2033212.041910,0.000326,1.000000 histogram,atime (KB) count,118723783234,items average,264.905060,days min,0,days max,1483,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,1573515888,0.013254,0.013254,1111493822.377464,0.000035,0.000035 2,4,1223528168,0.010306,0.023559,3912931628.926414,0.000124,0.000160 4,8,566106726,0.004768,0.028328,3619277710.404666,0.000115,0.000275 8,16,11411559921,0.096119,0.124446,152103406104.956543,0.004836,0.005111 16,32,5448726055,0.045894,0.170340,127087813159.477859,0.004041,0.009152 32,64,10852246878,0.091408,0.261748,494091508331.668457,0.015710,0.024862 64,128,15148363744,0.127593,0.389341,1373167383029.253662,0.043661,0.068523 128,256,22267172233,0.187554,0.576896,4184072542221.459473,0.133037,0.201560 256,512,28639891237,0.241231,0.818127,11491813947766.080078,0.365393,0.566953 512,1024,21584252457,0.181802,0.999929,13610384318242.556641,0.432755,0.999709 1024,2048,8419923,0.000071,1.000000,9166255428.571865,0.000291,1.000000 -------------- next part -------------- ----- CUT HERE ----- REPORT FOLLOWS ----- CUT HERE ----- Generated by fsstats v1.4.5 (03/22/2008) complete : processed 21294947 files in 102228 secs (at 208.31 files/sec) Skipped 113 duplicate hardlinked files CSV WRITTEN TO cc-lustre-fsstats ----- BEGIN NORMAL ----- total 110.60 TB used to store 110.57 TB user data, overhead 0.03% file size (Regular file's last addressable byte): Range, Count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of files, total size of files in range, total size of files in range as a % of total size of files, total size of files in this range or smaller as a % of total size of files. count=20963690 avg=5663.31 KB min=0.00 KB max=88080384.03 KB [ 0- 2 KB): 9988853 (47.65%) ( 47.65% cumulative) 10644331.00 KB ( 0.01%) ( 0.01% cumulative) [ 2- 4 KB): 2211694 (10.55%) ( 58.20% cumulative) 5117879.17 KB ( 0.00%) ( 0.01% cumulative) [ 4- 8 KB): 434744 ( 2.07%) ( 60.27% cumulative) 2163080.03 KB ( 0.00%) ( 0.02% cumulative) [ 8- 16 KB): 3308049 (15.78%) ( 76.05% cumulative) 30315021.21 KB ( 0.03%) ( 0.04% cumulative) [ 16- 32 KB): 246322 ( 1.17%) ( 77.23% cumulative) 5634038.11 KB ( 0.00%) ( 0.05% cumulative) [ 32- 64 KB): 549850 ( 2.62%) ( 79.85% cumulative) 27251246.36 KB ( 0.02%) ( 0.07% cumulative) [ 64- 128 KB): 1294831 ( 6.18%) ( 86.03% cumulative) 105694690.29 KB ( 0.09%) ( 0.16% cumulative) [ 128- 256 KB): 1206324 ( 5.75%) ( 91.78% cumulative) 175351267.16 KB ( 0.15%) ( 0.31% cumulative) [ 256- 512 KB): 283007 ( 1.35%) ( 93.13% cumulative) 116443388.93 KB ( 0.10%) ( 0.40% cumulative) [ 512- 1024 KB): 287128 ( 1.37%) ( 94.50% cumulative) 213799662.54 KB ( 0.18%) ( 0.58% cumulative) [ 1024- 2048 KB): 209030 ( 1.00%) ( 95.50% cumulative) 297877006.20 KB ( 0.25%) ( 0.83% cumulative) [ 2048- 4096 KB): 237039 ( 1.13%) ( 96.63% cumulative) 726121881.95 KB ( 0.61%) ( 1.45% cumulative) [ 4096- 8192 KB): 258313 ( 1.23%) ( 97.86% cumulative) 1459982570.92 KB ( 1.23%) ( 2.68% cumulative) [ 8192- 16384 KB): 154643 ( 0.74%) ( 98.60% cumulative) 1743146719.90 KB ( 1.47%) ( 4.14% cumulative) [ 16384- 32768 KB): 113993 ( 0.54%) ( 99.14% cumulative) 2569819867.44 KB ( 2.16%) ( 6.31% cumulative) [ 32768- 65536 KB): 35536 ( 0.17%) ( 99.31% cumulative) 1593541789.72 KB ( 1.34%) ( 7.65% cumulative) [ 65536- 131072 KB): 35113 ( 0.17%) ( 99.48% cumulative) 3273789717.92 KB ( 2.76%) ( 10.41% cumulative) [ 131072- 262144 KB): 44823 ( 0.21%) ( 99.69% cumulative) 8623199382.57 KB ( 7.26%) ( 17.67% cumulative) [ 262144- 524288 KB): 32315 ( 0.15%) ( 99.85% cumulative) 12103445450.95 KB (10.19%) ( 27.87% cumulative) [ 524288- 1048576 KB): 12379 ( 0.06%) ( 99.91% cumulative) 9369780063.99 KB ( 7.89%) ( 35.76% cumulative) [ 1048576- 2097152 KB): 9696 ( 0.05%) ( 99.95% cumulative) 13605014835.13 KB (11.46%) ( 47.22% cumulative) [ 2097152- 4194304 KB): 5779 ( 0.03%) ( 99.98% cumulative) 15634270433.63 KB (13.17%) ( 60.39% cumulative) [ 4194304- 8388608 KB): 2282 ( 0.01%) ( 99.99% cumulative) 13913493530.23 KB (11.72%) ( 72.11% cumulative) [ 8388608- 16777216 KB): 1214 ( 0.01%) (100.00% cumulative) 14072262149.84 KB (11.85%) ( 83.96% cumulative) [ 16777216- 33554432 KB): 547 ( 0.00%) (100.00% cumulative) 11903128580.76 KB (10.03%) ( 93.98% cumulative) [ 33554432- 67108864 KB): 183 ( 0.00%) (100.00% cumulative) 6888917675.97 KB ( 5.80%) ( 99.79% cumulative) [ 67108864-134217728 KB): 3 ( 0.00%) (100.00% cumulative) 253576972.79 KB ( 0.21%) (100.00% cumulative) capacity used (Regualr file's space used on disk): Range, Count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of file, total capacity used in range, total capacity used in range as a % of total capacity used, total capacity used in this range or smaller as a % of total capacity. count=20963690 avg=5664.85 KB min=0.00 KB max=88080472.00 KB [ 0- 2 KB): 42157 ( 0.20%) ( 0.20% cumulative) 0.00 KB ( 0.00%) ( 0.00% cumulative) [ 4- 8 KB): 12158464 (58.00%) ( 58.20% cumulative) 48633856.00 KB ( 0.04%) ( 0.04% cumulative) [ 8- 16 KB): 3645112 (17.39%) ( 75.59% cumulative) 42002104.00 KB ( 0.04%) ( 0.08% cumulative) [ 16- 32 KB): 292124 ( 1.39%) ( 76.98% cumulative) 6121624.00 KB ( 0.01%) ( 0.08% cumulative) [ 32- 64 KB): 434226 ( 2.07%) ( 79.05% cumulative) 19392576.00 KB ( 0.02%) ( 0.10% cumulative) [ 64- 128 KB): 1458240 ( 6.96%) ( 86.01% cumulative) 118267028.00 KB ( 0.10%) ( 0.20% cumulative) [ 128- 256 KB): 1204428 ( 5.75%) ( 91.75% cumulative) 176857552.00 KB ( 0.15%) ( 0.35% cumulative) [ 256- 512 KB): 278244 ( 1.33%) ( 93.08% cumulative) 113045348.00 KB ( 0.10%) ( 0.44% cumulative) [ 512- 1024 KB): 296902 ( 1.42%) ( 94.50% cumulative) 218931408.00 KB ( 0.18%) ( 0.63% cumulative) [ 1024- 2048 KB): 209705 ( 1.00%) ( 95.50% cumulative) 298580660.00 KB ( 0.25%) ( 0.88% cumulative) [ 2048- 4096 KB): 237215 ( 1.13%) ( 96.63% cumulative) 726260988.00 KB ( 0.61%) ( 1.49% cumulative) [ 4096- 8192 KB): 258606 ( 1.23%) ( 97.86% cumulative) 1461346880.00 KB ( 1.23%) ( 2.72% cumulative) [ 8192- 16384 KB): 154662 ( 0.74%) ( 98.60% cumulative) 1743312812.00 KB ( 1.47%) ( 4.19% cumulative) [ 16384- 32768 KB): 113731 ( 0.54%) ( 99.14% cumulative) 2564630540.00 KB ( 2.16%) ( 6.35% cumulative) [ 32768- 65536 KB): 35545 ( 0.17%) ( 99.31% cumulative) 1593927936.00 KB ( 1.34%) ( 7.69% cumulative) [ 65536- 131072 KB): 35111 ( 0.17%) ( 99.48% cumulative) 3273579896.00 KB ( 2.76%) ( 10.45% cumulative) [ 131072- 262144 KB): 44821 ( 0.21%) ( 99.69% cumulative) 8622937300.00 KB ( 7.26%) ( 17.71% cumulative) [ 262144- 524288 KB): 32316 ( 0.15%) ( 99.85% cumulative) 12103831248.00 KB (10.19%) ( 27.90% cumulative) [ 524288- 1048576 KB): 12378 ( 0.06%) ( 99.91% cumulative) 9369247428.00 KB ( 7.89%) ( 35.79% cumulative) [ 1048576- 2097152 KB): 9695 ( 0.05%) ( 99.95% cumulative) 13603725128.00 KB (11.46%) ( 47.24% cumulative) [ 2097152- 4194304 KB): 5779 ( 0.03%) ( 99.98% cumulative) 15634291672.00 KB (13.17%) ( 60.41% cumulative) [ 4194304- 8388608 KB): 2284 ( 0.01%) ( 99.99% cumulative) 13926107864.00 KB (11.73%) ( 72.14% cumulative) [ 8388608- 16777216 KB): 1212 ( 0.01%) (100.00% cumulative) 14045385560.00 KB (11.83%) ( 83.96% cumulative) [ 16777216- 33554432 KB): 547 ( 0.00%) (100.00% cumulative) 11903144228.00 KB (10.02%) ( 93.99% cumulative) [ 33554432- 67108864 KB): 183 ( 0.00%) (100.00% cumulative) 6888929992.00 KB ( 5.80%) ( 99.79% cumulative) [ 67108864-134217728 KB): 3 ( 0.00%) (100.00% cumulative) 253577152.00 KB ( 0.21%) (100.00% cumulative) positive overhead (Files whose capacity used is larger than or equal to its size. Includes directories, symlinks and regular files): Range of file sizes, count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of files, total bytes that overflowed in range, total bytes that overflowed in this range as a % of total bytes that overflowed, total bytes that overflowed in this range or smaller as a % of total bytes that overflowed. count=21227725 avg=2.64 KB min=0.00 KB max=625.72 KB [ 0- 2 KB): 10133167 (47.74%) ( 47.74% cumulative) 29708405.12 KB (52.92%) ( 52.92% cumulative) [ 2- 4 KB): 2211684 (10.42%) ( 58.15% cumulative) 3728886.71 KB ( 6.64%) ( 59.56% cumulative) [ 4- 8 KB): 545885 ( 2.57%) ( 60.73% cumulative) 1314591.97 KB ( 2.34%) ( 61.90% cumulative) [ 8- 16 KB): 3309882 (15.59%) ( 76.32% cumulative) 9772114.79 KB (17.41%) ( 79.31% cumulative) [ 16- 32 KB): 248098 ( 1.17%) ( 77.49% cumulative) 591345.89 KB ( 1.05%) ( 80.36% cumulative) [ 32- 64 KB): 551037 ( 2.60%) ( 80.08% cumulative) 1201073.64 KB ( 2.14%) ( 82.50% cumulative) [ 64- 128 KB): 1295993 ( 6.11%) ( 86.19% cumulative) 2378769.36 KB ( 4.24%) ( 86.74% cumulative) [ 128- 256 KB): 1208447 ( 5.69%) ( 91.88% cumulative) 2525560.84 KB ( 4.50%) ( 91.24% cumulative) [ 256- 512 KB): 284001 ( 1.34%) ( 93.22% cumulative) 590533.72 KB ( 1.05%) ( 92.29% cumulative) [ 512- 1024 KB): 287152 ( 1.35%) ( 94.57% cumulative) 673432.54 KB ( 1.20%) ( 93.49% cumulative) [ 1024- 2048 KB): 209068 ( 0.98%) ( 95.56% cumulative) 475323.03 KB ( 0.85%) ( 94.33% cumulative) [ 2048- 4096 KB): 237067 ( 1.12%) ( 96.67% cumulative) 524272.07 KB ( 0.93%) ( 95.27% cumulative) [ 4096- 8192 KB): 258301 ( 1.22%) ( 97.89% cumulative) 584040.31 KB ( 1.04%) ( 96.31% cumulative) [ 8192- 16384 KB): 154590 ( 0.73%) ( 98.62% cumulative) 533597.97 KB ( 0.95%) ( 97.26% cumulative) [ 16384- 32768 KB): 113663 ( 0.54%) ( 99.15% cumulative) 540955.00 KB ( 0.96%) ( 98.22% cumulative) [ 32768- 65536 KB): 35518 ( 0.17%) ( 99.32% cumulative) 161950.30 KB ( 0.29%) ( 98.51% cumulative) [ 65536- 131072 KB): 35097 ( 0.17%) ( 99.49% cumulative) 160957.96 KB ( 0.29%) ( 98.80% cumulative) [ 131072- 262144 KB): 44814 ( 0.21%) ( 99.70% cumulative) 219240.24 KB ( 0.39%) ( 99.19% cumulative) [ 262144- 524288 KB): 32307 ( 0.15%) ( 99.85% cumulative) 189286.74 KB ( 0.34%) ( 99.52% cumulative) [ 524288- 1048576 KB): 12368 ( 0.06%) ( 99.91% cumulative) 70185.22 KB ( 0.13%) ( 99.65% cumulative) [ 1048576- 2097152 KB): 9656 ( 0.05%) ( 99.95% cumulative) 59877.36 KB ( 0.11%) ( 99.76% cumulative) [ 2097152- 4194304 KB): 5743 ( 0.03%) ( 99.98% cumulative) 40567.54 KB ( 0.07%) ( 99.83% cumulative) [ 4194304- 8388608 KB): 2282 ( 0.01%) ( 99.99% cumulative) 31413.77 KB ( 0.06%) ( 99.88% cumulative) [ 8388608- 16777216 KB): 1172 ( 0.01%) (100.00% cumulative) 36640.08 KB ( 0.07%) ( 99.95% cumulative) [ 16777216- 33554432 KB): 547 ( 0.00%) (100.00% cumulative) 15647.24 KB ( 0.03%) ( 99.98% cumulative) [ 33554432- 67108864 KB): 183 ( 0.00%) (100.00% cumulative) 12316.03 KB ( 0.02%) (100.00% cumulative) [ 67108864-134217728 KB): 3 ( 0.00%) (100.00% cumulative) 179.21 KB ( 0.00%) (100.00% cumulative) negative overhead (Files whose size is larger than capacity used. Includes directories, symlinks and regular files): Range in file size, count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of files, total bytes that underflowed in range, total bytes that underflowed in range as a % of total bytes that underflowed, total bytes that underflowed in this range or smaller as a % total bytes that underflowed. count=67109 avg=346.76 KB min=0.00 KB max=6291448.00 KB [ 0- 2 KB): 66403 (98.95%) ( 98.95% cumulative) 2755.12 KB ( 0.01%) ( 0.01% cumulative) [ 2- 4 KB): 10 ( 0.01%) ( 98.96% cumulative) 25.88 KB ( 0.00%) ( 0.01% cumulative) [ 64- 128 KB): 82 ( 0.12%) ( 99.09% cumulative) 5871.65 KB ( 0.03%) ( 0.04% cumulative) [ 128- 256 KB): 1 ( 0.00%) ( 99.09% cumulative) 4.00 KB ( 0.00%) ( 0.04% cumulative) [ 256- 512 KB): 2 ( 0.00%) ( 99.09% cumulative) 22.65 KB ( 0.00%) ( 0.04% cumulative) [ 512- 1024 KB): 3 ( 0.00%) ( 99.09% cumulative) 2023.09 KB ( 0.01%) ( 0.05% cumulative) [ 1024- 2048 KB): 2 ( 0.00%) ( 99.10% cumulative) 2689.23 KB ( 0.01%) ( 0.06% cumulative) [ 2048- 4096 KB): 3 ( 0.00%) ( 99.10% cumulative) 3090.02 KB ( 0.01%) ( 0.07% cumulative) [ 4096- 8192 KB): 30 ( 0.04%) ( 99.15% cumulative) 94251.23 KB ( 0.41%) ( 0.48% cumulative) [ 8192- 16384 KB): 57 ( 0.08%) ( 99.23% cumulative) 276325.87 KB ( 1.19%) ( 1.66% cumulative) [ 16384- 32768 KB): 332 ( 0.49%) ( 99.73% cumulative) 5947938.44 KB (25.56%) ( 27.22% cumulative) [ 32768- 65536 KB): 18 ( 0.03%) ( 99.75% cumulative) 162896.02 KB ( 0.70%) ( 27.92% cumulative) [ 65536- 131072 KB): 18 ( 0.03%) ( 99.78% cumulative) 212183.89 KB ( 0.91%) ( 28.83% cumulative) [ 131072- 262144 KB): 11 ( 0.02%) ( 99.80% cumulative) 219554.81 KB ( 0.94%) ( 29.78% cumulative) [ 262144- 524288 KB): 8 ( 0.01%) ( 99.81% cumulative) 65637.69 KB ( 0.28%) ( 30.06% cumulative) [ 524288- 1048576 KB): 11 ( 0.02%) ( 99.82% cumulative) 1461989.20 KB ( 6.28%) ( 36.34% cumulative) [ 1048576- 2097152 KB): 40 ( 0.06%) ( 99.88% cumulative) 464076.49 KB ( 1.99%) ( 38.34% cumulative) [ 2097152- 4194304 KB): 36 ( 0.05%) ( 99.94% cumulative) 19329.17 KB ( 0.08%) ( 38.42% cumulative) [ 8388608-16777216 KB): 42 ( 0.06%) (100.00% cumulative) 14330309.92 KB (61.58%) (100.00% cumulative) directory size (entries): Range of entries, count of directories in range, number of dirs in range as a % of total num of dirs, number of dirs in this range or smaller as a % total number of dirs, total entries in range, number of entries in range as a % of total number of entries, number of entries in this range or smaller as a % of total number of entries. count=120427 avg=176.83 ents min=0.00 ents max=1065715.00 ents [ 0- 1 ents]: 37665 (31.28%) ( 31.28% cumulative) 23330.00 ents ( 0.11%) ( 0.11% cumulative) [ 2- 3 ents]: 13417 (11.14%) ( 42.42% cumulative) 35582.00 ents ( 0.17%) ( 0.28% cumulative) [ 4- 7 ents]: 18135 (15.06%) ( 57.48% cumulative) 97690.00 ents ( 0.46%) ( 0.74% cumulative) [ 8- 15 ents]: 17283 (14.35%) ( 71.83% cumulative) 184931.00 ents ( 0.87%) ( 1.60% cumulative) [ 16- 31 ents]: 11260 ( 9.35%) ( 81.18% cumulative) 254356.00 ents ( 1.19%) ( 2.80% cumulative) [ 32- 63 ents]: 7903 ( 6.56%) ( 87.74% cumulative) 321699.00 ents ( 1.51%) ( 4.31% cumulative) [ 64- 127 ents]: 4892 ( 4.06%) ( 91.80% cumulative) 404844.00 ents ( 1.90%) ( 6.21% cumulative) [ 128- 255 ents]: 2557 ( 2.12%) ( 93.93% cumulative) 401205.00 ents ( 1.88%) ( 8.09% cumulative) [ 256- 511 ents]: 1768 ( 1.47%) ( 95.39% cumulative) 607223.00 ents ( 2.85%) ( 10.95% cumulative) [ 512- 1023 ents]: 1769 ( 1.47%) ( 96.86% cumulative) 1609130.00 ents ( 7.56%) ( 18.50% cumulative) [ 1024- 2047 ents]: 728 ( 0.60%) ( 97.47% cumulative) 1090462.00 ents ( 5.12%) ( 23.62% cumulative) [ 2048- 4095 ents]: 2156 ( 1.79%) ( 99.26% cumulative) 5721820.00 ents (26.87%) ( 50.49% cumulative) [ 4096- 8191 ents]: 788 ( 0.65%) ( 99.91% cumulative) 3643213.00 ents (17.11%) ( 67.60% cumulative) [ 8192- 16383 ents]: 25 ( 0.02%) ( 99.93% cumulative) 284845.00 ents ( 1.34%) ( 68.94% cumulative) [ 16384- 32767 ents]: 29 ( 0.02%) ( 99.96% cumulative) 638579.00 ents ( 3.00%) ( 71.94% cumulative) [ 32768- 65535 ents]: 39 ( 0.03%) ( 99.99% cumulative) 1739267.00 ents ( 8.17%) ( 80.10% cumulative) [ 65536- 131071 ents]: 7 ( 0.01%) (100.00% cumulative) 666197.00 ents ( 3.13%) ( 83.23% cumulative) [ 131072- 262143 ents]: 2 ( 0.00%) (100.00% cumulative) 378730.00 ents ( 1.78%) ( 85.01% cumulative) [ 524288-1048575 ents]: 3 ( 0.00%) (100.00% cumulative) 2126128.00 ents ( 9.98%) ( 95.00% cumulative) [1048576-2097151 ents]: 1 ( 0.00%) (100.00% cumulative) 1065715.00 ents ( 5.00%) (100.00% cumulative) directory size (KB's): Range of dir sizes, count of directories in range, number of dirs in range as a % of total num of dirs, number of dirs in this range or smaller as a % total number of dirs, total size of dirs in range, total size of dirs in range as a % of total size of dirs, total size of dirs in this range or smaller as a % of total size of dirs. count=120427 avg=18.34 KB min=4.00 KB max=165476.00 KB [ 4- 8 KB): 111141 (92.29%) ( 92.29% cumulative) 444564.00 KB (20.13%) ( 20.13% cumulative) [ 8- 16 KB): 1833 ( 1.52%) ( 93.81% cumulative) 21996.00 KB ( 1.00%) ( 21.13% cumulative) [ 16- 32 KB): 1776 ( 1.47%) ( 95.29% cumulative) 36340.00 KB ( 1.65%) ( 22.77% cumulative) [ 32- 64 KB): 1187 ( 0.99%) ( 96.27% cumulative) 59960.00 KB ( 2.72%) ( 25.49% cumulative) [ 64- 128 KB): 1244 ( 1.03%) ( 97.30% cumulative) 101984.00 KB ( 4.62%) ( 30.11% cumulative) [ 128- 256 KB): 2124 ( 1.76%) ( 99.07% cumulative) 392092.00 KB (17.76%) ( 47.87% cumulative) [ 256- 512 KB): 996 ( 0.83%) ( 99.90% cumulative) 299504.00 KB (13.56%) ( 61.43% cumulative) [ 512- 1024 KB): 27 ( 0.02%) ( 99.92% cumulative) 18812.00 KB ( 0.85%) ( 62.28% cumulative) [ 1024- 2048 KB): 40 ( 0.03%) ( 99.95% cumulative) 54788.00 KB ( 2.48%) ( 64.76% cumulative) [ 2048- 4096 KB): 31 ( 0.03%) ( 99.98% cumulative) 92764.00 KB ( 4.20%) ( 68.96% cumulative) [ 4096- 8192 KB): 18 ( 0.01%) ( 99.99% cumulative) 94156.00 KB ( 4.26%) ( 73.23% cumulative) [ 8192- 16384 KB): 4 ( 0.00%) (100.00% cumulative) 45704.00 KB ( 2.07%) ( 75.30% cumulative) [ 16384- 32768 KB): 2 ( 0.00%) (100.00% cumulative) 42732.00 KB ( 1.94%) ( 77.23% cumulative) [ 65536-131072 KB): 2 ( 0.00%) (100.00% cumulative) 199124.00 KB ( 9.02%) ( 86.25% cumulative) [131072-262144 KB): 2 ( 0.00%) (100.00% cumulative) 303604.00 KB (13.75%) (100.00% cumulative) filename length (Includes regular files, dirs, symlinks and special files): Range of filename lengths, count of files in range, count of files in range as a % of total number of files, total bytes of filenames in range, total bytes of filenames in range as a % of total number of bytes of filenames, total number of bytes of filenames in this range or smaller as a % of total number bytes of filenames. count=21294947 avg=46.03 chars min=1.00 chars max=156.00 chars [ 0- 7 chars]: 175229 ( 0.82%) ( 0.82% cumulative) 924669.00 chars ( 0.09%) ( 0.09% cumulative) [ 8- 15 chars]: 1114735 ( 5.23%) ( 6.06% cumulative) 13382449.00 chars ( 1.37%) ( 1.46% cumulative) [ 16- 23 chars]: 1434266 ( 6.74%) ( 12.79% cumulative) 28106864.00 chars ( 2.87%) ( 4.33% cumulative) [ 24- 31 chars]: 2511232 (11.79%) ( 24.59% cumulative) 68798838.00 chars ( 7.02%) ( 11.35% cumulative) [ 32- 39 chars]: 6338462 (29.77%) ( 54.35% cumulative) 236275028.00 chars (24.10%) ( 35.45% cumulative) [ 40- 47 chars]: 5029532 (23.62%) ( 77.97% cumulative) 209186020.00 chars (21.34%) ( 56.79% cumulative) [ 48- 55 chars]: 537420 ( 2.52%) ( 80.49% cumulative) 27317483.00 chars ( 2.79%) ( 59.58% cumulative) [ 56- 63 chars]: 220318 ( 1.03%) ( 81.53% cumulative) 13214545.00 chars ( 1.35%) ( 60.92% cumulative) [ 64- 71 chars]: 197292 ( 0.93%) ( 82.45% cumulative) 13334976.00 chars ( 1.36%) ( 62.28% cumulative) [ 72- 79 chars]: 354371 ( 1.66%) ( 84.12% cumulative) 26738700.00 chars ( 2.73%) ( 65.01% cumulative) [ 80- 87 chars]: 557251 ( 2.62%) ( 86.73% cumulative) 45984154.00 chars ( 4.69%) ( 69.70% cumulative) [ 88- 95 chars]: 224094 ( 1.05%) ( 87.79% cumulative) 20193481.00 chars ( 2.06%) ( 71.76% cumulative) [ 96-103 chars]: 820745 ( 3.85%) ( 91.64% cumulative) 82492989.00 chars ( 8.42%) ( 80.18% cumulative) [104-111 chars]: 1273133 ( 5.98%) ( 97.62% cumulative) 136763921.00 chars (13.95%) ( 94.13% cumulative) [112-119 chars]: 506700 ( 2.38%) (100.00% cumulative) 57515919.00 chars ( 5.87%) (100.00% cumulative) [120-127 chars]: 59 ( 0.00%) (100.00% cumulative) 7385.00 chars ( 0.00%) (100.00% cumulative) [ >127 chars]: 108 ( 0.00%) (100.00% cumulative) 14949.00 chars ( 0.00%) (100.00% cumulative) link count(of regular files): Range of link counts, count of files in range, number of files in range as a % of total files, number of files in this range or smaller as a % of total files, total links in range, total links in range as a % of total links, total links in this range or smaller as a % of total links. count=20963690 avg=1.00 links min=1.00 links max=2.00 links [0-1 links]: 20963577 (100.00%) (100.00% cumulative) 20963577.00 links (100.00%) (100.00% cumulative) [2-3 links]: 113 ( 0.00%) (100.00% cumulative) 226.00 links ( 0.00%) (100.00% cumulative) symlink target length: Range of symlink file size, count of files in range, number of files in range as a % of total files, number of files in this range or smaller as a % of total files, total size of targets in range, total size of targets in range as a % of total size of targets, total size of targets in this range or smaller as a % of total size of targets. count=210717 avg=68.61 chars min=3.00 chars max=138.00 chars [ 0- 7 chars]: 41 ( 0.02%) ( 0.02% cumulative) 257.00 chars ( 0.00%) ( 0.00% cumulative) [ 8- 15 chars]: 3263 ( 1.55%) ( 1.57% cumulative) 39471.00 chars ( 0.27%) ( 0.27% cumulative) [ 16- 23 chars]: 6853 ( 3.25%) ( 4.82% cumulative) 136114.00 chars ( 0.94%) ( 1.22% cumulative) [ 24- 31 chars]: 2921 ( 1.39%) ( 6.21% cumulative) 79673.00 chars ( 0.55%) ( 1.77% cumulative) [ 32- 39 chars]: 2997 ( 1.42%) ( 7.63% cumulative) 104318.00 chars ( 0.72%) ( 2.49% cumulative) [ 40- 47 chars]: 23609 (11.20%) ( 18.83% cumulative) 1024366.00 chars ( 7.09%) ( 9.57% cumulative) [ 48- 55 chars]: 17186 ( 8.16%) ( 26.99% cumulative) 886356.00 chars ( 6.13%) ( 15.70% cumulative) [ 56- 63 chars]: 15383 ( 7.30%) ( 34.29% cumulative) 904491.00 chars ( 6.26%) ( 21.96% cumulative) [ 64- 71 chars]: 21767 (10.33%) ( 44.62% cumulative) 1490971.00 chars (10.31%) ( 32.27% cumulative) [ 72- 79 chars]: 40367 (19.16%) ( 63.78% cumulative) 3059852.00 chars (21.16%) ( 53.44% cumulative) [ 80- 87 chars]: 37398 (17.75%) ( 81.52% cumulative) 3107834.00 chars (21.50%) ( 74.93% cumulative) [ 88- 95 chars]: 33925 (16.10%) ( 97.62% cumulative) 3091387.00 chars (21.38%) ( 96.32% cumulative) [ 96-103 chars]: 3265 ( 1.55%) ( 99.17% cumulative) 318424.00 chars ( 2.20%) ( 98.52% cumulative) [104-111 chars]: 334 ( 0.16%) ( 99.33% cumulative) 35626.00 chars ( 0.25%) ( 98.76% cumulative) [112-119 chars]: 154 ( 0.07%) ( 99.40% cumulative) 17538.00 chars ( 0.12%) ( 98.89% cumulative) [ >127 chars]: 1254 ( 0.60%) (100.00% cumulative) 161172.00 chars ( 1.11%) (100.00% cumulative) symlink targets: 67.47% relative, 32.53% absolute time since last modification (Reguar files):Range in days, count of files in range, number of files in range as a % of total number of files, number of files in this range or smaller as a % of total number of files, total of age in range, total age in range as a % of total age of all files, total age in this range or smaller as a % of total age of all files. count=20963690 avg=310.84 days min=0.00 days max=7029.15 days [ 0- 2 days): 79767 ( 0.38%) ( 0.38% cumulative) 79142.95 days ( 0.00%) ( 0.00% cumulative) [ 2- 4 days): 9027 ( 0.04%) ( 0.42% cumulative) 25495.19 days ( 0.00%) ( 0.00% cumulative) [ 4- 8 days): 19124 ( 0.09%) ( 0.51% cumulative) 109156.50 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 37509 ( 0.18%) ( 0.69% cumulative) 475464.95 days ( 0.01%) ( 0.01% cumulative) [ 16- 32 days): 389482 ( 1.86%) ( 2.55% cumulative) 9461692.73 days ( 0.15%) ( 0.16% cumulative) [ 32- 64 days): 523782 ( 2.50%) ( 5.05% cumulative) 25947289.19 days ( 0.40%) ( 0.55% cumulative) [ 64- 128 days): 2061189 ( 9.83%) ( 14.88% cumulative) 207545525.54 days ( 3.18%) ( 3.74% cumulative) [ 128- 256 days): 5977541 (28.51%) ( 43.40% cumulative) 1263408411.72 days (19.39%) ( 23.13% cumulative) [ 256- 512 days): 10741570 (51.24%) ( 94.64% cumulative) 4143022927.49 days (63.58%) ( 86.71% cumulative) [ 512-1024 days): 955608 ( 4.56%) ( 99.19% cumulative) 607877139.16 days ( 9.33%) ( 96.03% cumulative) [1024-2048 days): 141555 ( 0.68%) ( 99.87% cumulative) 179353945.53 days ( 2.75%) ( 98.79% cumulative) [2048-4096 days): 24340 ( 0.12%) ( 99.98% cumulative) 64552190.06 days ( 0.99%) ( 99.78% cumulative) [4096-8192 days): 3196 ( 0.02%) (100.00% cumulative) 14547335.57 days ( 0.22%) (100.00% cumulative) time since last modification (size of regular files by age): Range in days, count of bytes whose age is in range, count of bytes in range as a % total number of bytes, count of bytes in this range or smaller as a % of total number of bytes, total days in range (each weighted by its size), today weighted days in range as a % of total weighted days, total days in this range or smaller as a % of total weighted days. count=118723783234 avg=294.51 days min=0.00 days max=7029.15 days [ 0- 2 days): 1236757924 ( 1.04%) ( 1.04% cumulative) 719011583.09 days ( 0.00%) ( 0.00% cumulative) [ 2- 4 days): 475882662 ( 0.40%) ( 1.44% cumulative) 1459775355.42 days ( 0.00%) ( 0.01% cumulative) [ 4- 8 days): 591887628 ( 0.50%) ( 1.94% cumulative) 3739535344.82 days ( 0.01%) ( 0.02% cumulative) [ 8- 16 days): 2327165996 ( 1.96%) ( 3.90% cumulative) 26334104519.79 days ( 0.08%) ( 0.09% cumulative) [ 16- 32 days): 4893100685 ( 4.12%) ( 8.02% cumulative) 129481373506.18 days ( 0.37%) ( 0.46% cumulative) [ 32- 64 days): 13335761868 (11.23%) ( 19.26% cumulative) 645522888775.95 days ( 1.85%) ( 2.31% cumulative) [ 64- 128 days): 14740751501 (12.42%) ( 31.67% cumulative) 1409022412219.91 days ( 4.03%) ( 6.34% cumulative) [ 128- 256 days): 27552389401 (23.21%) ( 54.88% cumulative) 4994069520533.34 days (14.28%) ( 20.62% cumulative) [ 256- 512 days): 29807088715 (25.11%) ( 79.98% cumulative) 11943093434429.13 days (34.16%) ( 54.78% cumulative) [ 512-1024 days): 22539139104 (18.98%) ( 98.97% cumulative) 14482586438757.15 days (41.42%) ( 96.20% cumulative) [1024-2048 days): 1220517137 ( 1.03%) (100.00% cumulative) 1318741760524.16 days ( 3.77%) ( 99.97% cumulative) [2048-4096 days): 3265143 ( 0.00%) (100.00% cumulative) 10383292341.66 days ( 0.03%) (100.00% cumulative) [4096-8192 days): 75464 ( 0.00%) (100.00% cumulative) 342685847.61 days ( 0.00%) (100.00% cumulative) time since creation (Reguar files):Range in days, count of files in range, number of files in range as a % of total number of files, number of files in this range or smaller as a % of total number of files, total of age in range, total age in range as a % of total age of all files, total age in this range or smaller as a % of total age of all files. count=20963690 avg=293.50 days min=0.00 days max=756.08 days [ 0- 2 days): 79885 ( 0.38%) ( 0.38% cumulative) 79253.45 days ( 0.00%) ( 0.00% cumulative) [ 2- 4 days): 8969 ( 0.04%) ( 0.42% cumulative) 25324.49 days ( 0.00%) ( 0.00% cumulative) [ 4- 8 days): 19248 ( 0.09%) ( 0.52% cumulative) 110009.42 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 39373 ( 0.19%) ( 0.70% cumulative) 482019.50 days ( 0.01%) ( 0.01% cumulative) [ 16- 32 days): 398845 ( 1.90%) ( 2.61% cumulative) 9744007.22 days ( 0.16%) ( 0.17% cumulative) [ 32- 64 days): 933387 ( 4.45%) ( 7.06% cumulative) 43362984.06 days ( 0.70%) ( 0.87% cumulative) [ 64- 128 days): 1691530 ( 8.07%) ( 15.13% cumulative) 163796346.93 days ( 2.66%) ( 3.54% cumulative) [ 128- 256 days): 6256058 (29.84%) ( 44.97% cumulative) 1309069684.42 days (21.28%) ( 24.81% cumulative) [ 256- 512 days): 10846311 (51.74%) ( 96.71% cumulative) 4199850472.15 days (68.26%) ( 93.07% cumulative) [ 512-1024 days): 690084 ( 3.29%) (100.00% cumulative) 426313678.08 days ( 6.93%) (100.00% cumulative) time since creation (size of regular files by age): Range in days, count of bytes whose age is in range, count of bytes in range as a % total number of bytes, count of bytes in this range or smaller as a % of total number of bytes, total days in range (each weighted by its size), today weighted days in range as a % of total weighted days, total days in this range or smaller as a % of total weighted days. count=118723783234 avg=214.79 days min=0.00 days max=756.08 days [ 0- 2 days): 1246698470 ( 1.05%) ( 1.05% cumulative) 729569442.25 days ( 0.00%) ( 0.00% cumulative) [ 2- 4 days): 481961760 ( 0.41%) ( 1.46% cumulative) 1474982666.50 days ( 0.01%) ( 0.01% cumulative) [ 4- 8 days): 582017976 ( 0.49%) ( 1.95% cumulative) 3687306936.77 days ( 0.01%) ( 0.02% cumulative) [ 8- 16 days): 2625264219 ( 2.21%) ( 4.16% cumulative) 30385845844.16 days ( 0.12%) ( 0.14% cumulative) [ 16- 32 days): 8814482438 ( 7.42%) ( 11.58% cumulative) 241192027256.13 days ( 0.95%) ( 1.09% cumulative) [ 32- 64 days): 16623978134 (14.00%) ( 25.58% cumulative) 769631076173.65 days ( 3.02%) ( 4.11% cumulative) [ 64- 128 days): 22709982418 (19.13%) ( 44.71% cumulative) 2216919728983.78 days ( 8.69%) ( 12.80% cumulative) [ 128- 256 days): 30508794533 (25.70%) ( 70.41% cumulative) 5315988506304.36 days (20.85%) ( 33.65% cumulative) [ 256- 512 days): 22806938658 (19.21%) ( 89.62% cumulative) 9427857837022.68 days (36.97%) ( 70.62% cumulative) [ 512-1024 days): 12323664623 (10.38%) (100.00% cumulative) 7493318423583.88 days (29.38%) (100.00% cumulative) time since last access (Reguar files):Range in days, count of files in range, number of files in range as a % of total number of files, number of files in this range or smaller as a % of total number of files, total of age in range, total age in range as a % of total age of all files, total age in this range or smaller as a % of total age of all files. count=20963690 avg=297.30 days min=0.00 days max=1483.87 days [ 0- 2 days): 130565 ( 0.62%) ( 0.62% cumulative) 117658.83 days ( 0.00%) ( 0.00% cumulative) [ 2- 4 days): 10270 ( 0.05%) ( 0.67% cumulative) 30030.97 days ( 0.00%) ( 0.00% cumulative) [ 4- 8 days): 19384 ( 0.09%) ( 0.76% cumulative) 112254.74 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 141566 ( 0.68%) ( 1.44% cumulative) 1983381.16 days ( 0.03%) ( 0.04% cumulative) [ 16- 32 days): 397433 ( 1.90%) ( 3.34% cumulative) 8955804.97 days ( 0.14%) ( 0.18% cumulative) [ 32- 64 days): 1324101 ( 6.32%) ( 9.65% cumulative) 59205539.54 days ( 0.95%) ( 1.13% cumulative) [ 64- 128 days): 1295361 ( 6.18%) ( 15.83% cumulative) 126677772.92 days ( 2.03%) ( 3.16% cumulative) [ 128- 256 days): 5915624 (28.22%) ( 44.05% cumulative) 1258923268.00 days (20.20%) ( 23.36% cumulative) [ 256- 512 days): 10700363 (51.04%) ( 95.09% cumulative) 4130193012.79 days (66.27%) ( 89.63% cumulative) [ 512-1024 days): 1027185 ( 4.90%) ( 99.99% cumulative) 644239643.66 days (10.34%) ( 99.97% cumulative) [1024-2048 days): 1838 ( 0.01%) (100.00% cumulative) 2033212.04 days ( 0.03%) (100.00% cumulative) time since access (size of regular files by age): Range in days, count of bytes whose age is in range, count of bytes in range as a % total number of bytes, count of bytes in this range or smaller as a % of total number of bytes, total days in range (each weighted by its size), today weighted days in range as a % of total weighted days, total days in this range or smaller as a % of total weighted days. count=118723783234 avg=264.91 days min=0.00 days max=1483.87 days [ 0- 2 days): 1573515888 ( 1.33%) ( 1.33% cumulative) 1111493822.38 days ( 0.00%) ( 0.00% cumulative) [ 2- 4 days): 1223528168 ( 1.03%) ( 2.36% cumulative) 3912931628.93 days ( 0.01%) ( 0.02% cumulative) [ 4- 8 days): 566106726 ( 0.48%) ( 2.83% cumulative) 3619277710.40 days ( 0.01%) ( 0.03% cumulative) [ 8- 16 days): 11411559921 ( 9.61%) ( 12.44% cumulative) 152103406104.96 days ( 0.48%) ( 0.51% cumulative) [ 16- 32 days): 5448726055 ( 4.59%) ( 17.03% cumulative) 127087813159.48 days ( 0.40%) ( 0.92% cumulative) [ 32- 64 days): 10852246878 ( 9.14%) ( 26.17% cumulative) 494091508331.67 days ( 1.57%) ( 2.49% cumulative) [ 64- 128 days): 15148363744 (12.76%) ( 38.93% cumulative) 1373167383029.25 days ( 4.37%) ( 6.85% cumulative) [ 128- 256 days): 22267172233 (18.76%) ( 57.69% cumulative) 4184072542221.46 days (13.30%) ( 20.16% cumulative) [ 256- 512 days): 28639891237 (24.12%) ( 81.81% cumulative) 11491813947766.08 days (36.54%) ( 56.70% cumulative) [ 512-1024 days): 21584252457 (18.18%) ( 99.99% cumulative) 13610384318242.56 days (43.28%) ( 99.97% cumulative) [1024-2048 days): 8419923 ( 0.01%) (100.00% cumulative) 9166255428.57 days ( 0.03%) (100.00% cumulative) ----- END NORMAL ----- From nirmal at fnal.gov Mon Apr 25 15:48:11 2011 From: nirmal at fnal.gov (Nirmal Seenu) Date: Mon, 25 Apr 2011 10:48:11 -0500 Subject: [Lustre-devel] Research on filesystem metadata operation distribution In-Reply-To: References: Message-ID: <4DB597BB.5090300@fnal.gov> Attached is the stats from the LQCD cluster lustre file system. We use this filesystem as a general purpose storage system to store the outputs. The storage space is not backed up and users have to write the files manually to the tape if needed. Nirmal -------------- next part -------------- uptime 10:37:33 up 86 days, 21:34, 1 user, load average: 0.14, 0.51, 1.50 lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync" open 613755772 samples [reqs] close 301722516 samples [reqs] link 150 samples [reqs] unlink 13231029 samples [reqs] rename 2405891 samples [reqs] getattr 2629766124 samples [reqs] setattr 137068401 samples [reqs] getxattr 13042 samples [reqs] sync 7388 samples [reqs] 'lfs df' output: UUID 1K-blocks Used Available Use% Mounted on lqcdproj-MDT0000_UUID 293579372 7889780 285689592 2% /lqcdproj[MDT:0] lqcdproj-OST0000_UUID 2884110020 2613132412 270964064 90% /lqcdproj[OST:0] lqcdproj-OST0001_UUID 2884114628 2601758076 282312900 90% /lqcdproj[OST:1] lqcdproj-OST0002_UUID 2884114628 2662345648 221751180 92% /lqcdproj[OST:2] lqcdproj-OST0003_UUID 2884114628 2600940396 283172184 90% /lqcdproj[OST:3] lqcdproj-OST0004_UUID 2884114628 2640066844 244038820 91% /lqcdproj[OST:4] lqcdproj-OST0005_UUID 2884114628 2623291648 260811660 90% /lqcdproj[OST:5] lqcdproj-OST0006_UUID 2884114628 2606213580 277879556 90% /lqcdproj[OST:6] lqcdproj-OST0007_UUID 2884114628 2587228924 296855532 89% /lqcdproj[OST:7] lqcdproj-OST0008_UUID 2884114628 2680527240 203546344 92% /lqcdproj[OST:8] lqcdproj-OST0009_UUID 2884114628 2735554856 148515592 94% /lqcdproj[OST:9] lqcdproj-OST000a_UUID 2884110020 2579392516 304701876 89% /lqcdproj[OST:10] lqcdproj-OST000b_UUID 2884110020 2566274564 317820536 88% /lqcdproj[OST:11] lqcdproj-OST000c_UUID 2884110020 2655604816 228487100 92% /lqcdproj[OST:12] lqcdproj-OST000d_UUID 2884114628 2613190276 270915456 90% /lqcdproj[OST:13] lqcdproj-OST000e_UUID 2884114628 2594037276 290065888 89% /lqcdproj[OST:14] lqcdproj-OST000f_UUID 2884114628 2622882308 261213984 90% /lqcdproj[OST:15] lqcdproj-OST0010_UUID 2884114628 2647251136 236853376 91% /lqcdproj[OST:16] lqcdproj-OST0011_UUID 2884114628 2637580292 246522124 91% /lqcdproj[OST:17] lqcdproj-OST0012_UUID 2884114628 2622168844 261935528 90% /lqcdproj[OST:18] lqcdproj-OST0013_UUID 2884114628 2649462692 234642584 91% /lqcdproj[OST:19] lqcdproj-OST0014_UUID 2884114628 2652161080 231942528 91% /lqcdproj[OST:20] lqcdproj-OST0015_UUID 2884114628 2612201188 271903736 90% /lqcdproj[OST:21] lqcdproj-OST0016_UUID 2884110020 2635654212 248436400 91% /lqcdproj[OST:22] lqcdproj-OST0017_UUID 2884110020 2663990304 220111912 92% /lqcdproj[OST:23] lqcdproj-OST0018_UUID 2884114628 2701387532 182712248 93% /lqcdproj[OST:24] lqcdproj-OST0019_UUID 2884114628 2559541432 324556280 88% /lqcdproj[OST:25] lqcdproj-OST001a_UUID 2884114628 2641939044 242164628 91% /lqcdproj[OST:26] lqcdproj-OST001b_UUID 2884114628 2577809756 306289560 89% /lqcdproj[OST:27] lqcdproj-OST001c_UUID 2884114628 2650922984 233191640 91% /lqcdproj[OST:28] lqcdproj-OST001d_UUID 2884114628 2627144748 256958916 91% /lqcdproj[OST:29] lqcdproj-OST001e_UUID 2884114628 2633465196 250639572 91% /lqcdproj[OST:30] lqcdproj-OST001f_UUID 2884114628 2571269944 312842636 89% /lqcdproj[OST:31] lqcdproj-OST0020_UUID 2884114628 2668096448 216006484 92% /lqcdproj[OST:32] lqcdproj-OST0021_UUID 2884114628 2603719180 280375808 90% /lqcdproj[OST:33] lqcdproj-OST0022_UUID 2884114628 2562060984 322041880 88% /lqcdproj[OST:34] lqcdproj-OST0023_UUID 2884114628 2532849300 351253272 87% /lqcdproj[OST:35] lqcdproj-OST0024_UUID 2884114628 2533017120 351079260 87% /lqcdproj[OST:36] lqcdproj-OST0025_UUID 2884110020 2542586604 341523412 88% /lqcdproj[OST:37] lqcdproj-OST0026_UUID 2884114628 2560582444 323519408 88% /lqcdproj[OST:38] lqcdproj-OST0027_UUID 2884110020 2639694916 244405168 91% /lqcdproj[OST:39] lqcdproj-OST0028_UUID 2884114628 2571309884 312790236 89% /lqcdproj[OST:40] lqcdproj-OST0029_UUID 2884110020 2530861100 353238768 87% /lqcdproj[OST:41] lqcdproj-OST002a_UUID 2884110020 2613576308 270516880 90% /lqcdproj[OST:42] lqcdproj-OST002b_UUID 2884114628 2577598700 306500656 89% /lqcdproj[OST:43] lqcdproj-OST002c_UUID 2884114628 2616246212 267860328 90% /lqcdproj[OST:44] lqcdproj-OST002d_UUID 2884114628 2620559488 263548004 90% /lqcdproj[OST:45] lqcdproj-OST002e_UUID 2884110020 2537877392 346209092 87% /lqcdproj[OST:46] lqcdproj-OST002f_UUID 2884110020 2554455452 329637328 88% /lqcdproj[OST:47] lqcdproj-OST0030_UUID 2884059332 2632733724 251312600 91% /lqcdproj[OST:48] lqcdproj-OST0031_UUID 2884059332 2516044436 367980028 87% /lqcdproj[OST:49] lqcdproj-OST0032_UUID 2884271796 2566742004 317518808 88% /lqcdproj[OST:50] lqcdproj-OST0033_UUID 2884059332 2551671972 332368176 88% /lqcdproj[OST:51] lqcdproj-OST0034_UUID 2884059332 2569364380 314639132 89% /lqcdproj[OST:52] lqcdproj-OST0035_UUID 2884059332 2561217876 322818344 88% /lqcdproj[OST:53] lqcdproj-OST0036_UUID 2884059332 2637389232 246650132 91% /lqcdproj[OST:54] lqcdproj-OST0037_UUID 2884059332 2568451184 315584936 89% /lqcdproj[OST:55] lqcdproj-OST0038_UUID 2884271796 2548496208 335735244 88% /lqcdproj[OST:56] lqcdproj-OST0039_UUID 2884059332 2615917480 268133520 90% /lqcdproj[OST:57] lqcdproj-OST003a_UUID 2884271796 2554940060 329316668 88% /lqcdproj[OST:58] lqcdproj-OST003b_UUID 2884059332 2566522852 317514016 88% /lqcdproj[OST:59] lqcdproj-OST003c_UUID 2883950796 2573962996 309987796 89% /lqcdproj[OST:60] lqcdproj-OST003d_UUID 2883950796 2548389948 335545164 88% /lqcdproj[OST:61] lqcdproj-OST003e_UUID 2883950796 2540884640 343054152 88% /lqcdproj[OST:62] lqcdproj-OST003f_UUID 2883950796 2545441668 338497680 88% /lqcdproj[OST:63] lqcdproj-OST0040_UUID 2883950796 2586539516 297409232 89% /lqcdproj[OST:64] lqcdproj-OST0041_UUID 2883950796 2576799572 307141016 89% /lqcdproj[OST:65] lqcdproj-OST0042_UUID 2883950796 2613981588 269960088 90% /lqcdproj[OST:66] lqcdproj-OST0043_UUID 2883950796 2551743768 332202556 88% /lqcdproj[OST:67] lqcdproj-OST0044_UUID 2883950796 2633945760 250000332 91% /lqcdproj[OST:68] lqcdproj-OST0045_UUID 2883950796 2564084016 319866776 88% /lqcdproj[OST:69] lqcdproj-OST0046_UUID 2883950796 2531576920 352362112 87% /lqcdproj[OST:70] lqcdproj-OST0047_UUID 2883950796 2512748556 371202236 87% /lqcdproj[OST:71] lqcdproj-OST0048_UUID 5768390140 5114772644 653617492 88% /lqcdproj[OST:72] lqcdproj-OST0049_UUID 5768390140 5097073520 671289028 88% /lqcdproj[OST:73] lqcdproj-OST004a_UUID 5768390140 5060154412 708203960 87% /lqcdproj[OST:74] lqcdproj-OST004b_UUID 5768390140 5318191208 450186716 92% /lqcdproj[OST:75] lqcdproj-OST004c_UUID 5768390140 5300020760 468359056 91% /lqcdproj[OST:76] lqcdproj-OST004d_UUID 5768390140 5360640116 407738112 92% /lqcdproj[OST:77] lqcdproj-OST004e_UUID 5768390140 5301105280 467271816 91% /lqcdproj[OST:78] lqcdproj-OST004f_UUID 5768390140 5339516016 428846708 92% /lqcdproj[OST:79] lqcdproj-OST0050_UUID 5768390140 5276760312 491610244 91% /lqcdproj[OST:80] lqcdproj-OST0051_UUID 5768390140 5359078976 409293600 92% /lqcdproj[OST:81] lqcdproj-OST0052_UUID 5768390140 5321914412 446455080 92% /lqcdproj[OST:82] lqcdproj-OST0053_UUID 5768390140 5331386640 436987492 92% /lqcdproj[OST:83] lqcdproj-OST0054_UUID 5768227844 5092295640 675891472 88% /lqcdproj[OST:84] lqcdproj-OST0055_UUID 5768227844 4923076972 845150868 85% /lqcdproj[OST:85] lqcdproj-OST0056_UUID 5768226308 4960545996 807656120 85% /lqcdproj[OST:86] lqcdproj-OST0057_UUID 5768227844 5019124704 749090056 87% /lqcdproj[OST:87] lqcdproj-OST0058_UUID 5768227844 4984953948 783247872 86% /lqcdproj[OST:88] lqcdproj-OST0059_UUID 5768227844 4883190456 885022244 84% /lqcdproj[OST:89] lqcdproj-OST005a_UUID 5768227844 5409659176 358531528 93% /lqcdproj[OST:90] lqcdproj-OST005b_UUID 5768226308 5145960848 622242932 89% /lqcdproj[OST:91] lqcdproj-OST005c_UUID 5768226308 5312932100 455269532 92% /lqcdproj[OST:92] lqcdproj-OST005d_UUID 5768226308 5209470128 558749120 90% /lqcdproj[OST:93] lqcdproj-OST005e_UUID 5768226308 5358982744 409158808 92% /lqcdproj[OST:94] lqcdproj-OST005f_UUID 5768226308 5253169944 515056360 91% /lqcdproj[OST:95] lqcdproj-OST0060_UUID 5768390140 5383982244 384394216 93% /lqcdproj[OST:96] lqcdproj-OST0061_UUID 5768390140 5534252044 234123524 95% /lqcdproj[OST:97] lqcdproj-OST0062_UUID 5768390140 5376327400 392048612 93% /lqcdproj[OST:98] lqcdproj-OST0063_UUID 5768390140 5377268688 391100300 93% /lqcdproj[OST:99] filesystem summary: 369167172928 333406880980 35758655936 90% /lqcdproj 'lfs df -i' output: UUID Inodes IUsed IFree IUse% Mounted on lqcdproj-MDT0000_UUID 83886080 58832324 25053756 70% /lqcdproj[MDT:0] lqcdproj-OST0000_UUID 68134346 389944 67744402 0% /lqcdproj[OST:0] lqcdproj-OST0001_UUID 70905173 316025 70589148 0% /lqcdproj[OST:1] lqcdproj-OST0002_UUID 55795428 353126 55442302 0% /lqcdproj[OST:2] lqcdproj-OST0003_UUID 71153930 377280 70776650 0% /lqcdproj[OST:3] lqcdproj-OST0004_UUID 61386934 374953 61011981 0% /lqcdproj[OST:4] lqcdproj-OST0005_UUID 65583451 377688 65205763 0% /lqcdproj[OST:5] lqcdproj-OST0006_UUID 69870179 394917 69475262 0% /lqcdproj[OST:6] lqcdproj-OST0007_UUID 74565634 344198 74221436 0% /lqcdproj[OST:7] lqcdproj-OST0008_UUID 51265044 368158 50896886 0% /lqcdproj[OST:8] lqcdproj-OST0009_UUID 37516562 376584 37139978 1% /lqcdproj[OST:9] lqcdproj-OST000a_UUID 76502815 323436 76179379 0% /lqcdproj[OST:10] lqcdproj-OST000b_UUID 79820620 361768 79458852 0% /lqcdproj[OST:11] lqcdproj-OST000c_UUID 57481420 355119 57126301 0% /lqcdproj[OST:12] lqcdproj-OST000d_UUID 68140928 409808 67731120 0% /lqcdproj[OST:13] lqcdproj-OST000e_UUID 72882337 362967 72519370 0% /lqcdproj[OST:14] lqcdproj-OST000f_UUID 65717420 409310 65308110 0% /lqcdproj[OST:15] lqcdproj-OST0010_UUID 59622919 407046 59215873 0% /lqcdproj[OST:16] lqcdproj-OST0011_UUID 62022573 388980 61633593 0% /lqcdproj[OST:17] lqcdproj-OST0012_UUID 65900282 413804 65486478 0% /lqcdproj[OST:18] lqcdproj-OST0013_UUID 59061605 398606 58662999 0% /lqcdproj[OST:19] lqcdproj-OST0014_UUID 58397293 408891 57988402 0% /lqcdproj[OST:20] lqcdproj-OST0015_UUID 68441778 463416 67978362 0% /lqcdproj[OST:21] lqcdproj-OST0016_UUID 62449347 335365 62113982 0% /lqcdproj[OST:22] lqcdproj-OST0017_UUID 55373958 344035 55029923 0% /lqcdproj[OST:23] lqcdproj-OST0018_UUID 46072948 391164 45681784 0% /lqcdproj[OST:24] lqcdproj-OST0019_UUID 81616557 473258 81143299 0% /lqcdproj[OST:25] lqcdproj-OST001a_UUID 61013671 469754 60543917 0% /lqcdproj[OST:26] lqcdproj-OST001b_UUID 77000769 424546 76576223 0% /lqcdproj[OST:27] lqcdproj-OST001c_UUID 58781886 483966 58297920 0% /lqcdproj[OST:28] lqcdproj-OST001d_UUID 64729751 487272 64242479 0% /lqcdproj[OST:29] lqcdproj-OST001e_UUID 63134721 472345 62662376 0% /lqcdproj[OST:30] lqcdproj-OST001f_UUID 78651009 456672 78194337 0% /lqcdproj[OST:31] lqcdproj-OST0020_UUID 54435337 430757 54004580 0% /lqcdproj[OST:32] lqcdproj-OST0021_UUID 70536201 437337 70098864 0% /lqcdproj[OST:33] lqcdproj-OST0022_UUID 80994999 481579 80513420 0% /lqcdproj[OST:34] lqcdproj-OST0023_UUID 88248309 431961 87816348 0% /lqcdproj[OST:35] lqcdproj-OST0024_UUID 88214773 440396 87774377 0% /lqcdproj[OST:36] lqcdproj-OST0025_UUID 85744562 435639 85308923 0% /lqcdproj[OST:37] lqcdproj-OST0026_UUID 81336772 453698 80883074 0% /lqcdproj[OST:38] lqcdproj-OST0027_UUID 61552230 448454 61103776 0% /lqcdproj[OST:39] lqcdproj-OST0028_UUID 78601674 400458 78201216 0% /lqcdproj[OST:40] lqcdproj-OST0029_UUID 88815507 503275 88312232 0% /lqcdproj[OST:41] lqcdproj-OST002a_UUID 68121842 488407 67633435 0% /lqcdproj[OST:42] lqcdproj-OST002b_UUID 77088217 459231 76628986 0% /lqcdproj[OST:43] lqcdproj-OST002c_UUID 67381123 413986 66967137 0% /lqcdproj[OST:44] lqcdproj-OST002d_UUID 66340987 452226 65888761 0% /lqcdproj[OST:45] lqcdproj-OST002e_UUID 87029491 471330 86558161 0% /lqcdproj[OST:46] lqcdproj-OST002f_UUID 82880608 466967 82413641 0% /lqcdproj[OST:47] lqcdproj-OST0030_UUID 63278033 446629 62831404 0% /lqcdproj[OST:48] lqcdproj-OST0031_UUID 92471245 467499 92003746 0% /lqcdproj[OST:49] lqcdproj-OST0032_UUID 79817657 435175 79382482 0% /lqcdproj[OST:50] lqcdproj-OST0033_UUID 83615223 518355 83096868 0% /lqcdproj[OST:51] lqcdproj-OST0034_UUID 79104347 430571 78673776 0% /lqcdproj[OST:52] lqcdproj-OST0035_UUID 81193974 483610 80710364 0% /lqcdproj[OST:53] lqcdproj-OST0036_UUID 62118319 450757 61667562 0% /lqcdproj[OST:54] lqcdproj-OST0037_UUID 79440695 538658 78902037 0% /lqcdproj[OST:55] lqcdproj-OST0038_UUID 84388253 444314 83943939 0% /lqcdproj[OST:56] lqcdproj-OST0039_UUID 67527521 492058 67035463 0% /lqcdproj[OST:57] lqcdproj-OST003a_UUID 82818499 485536 82332963 0% /lqcdproj[OST:58] lqcdproj-OST003b_UUID 79803730 419606 79384124 0% /lqcdproj[OST:59] lqcdproj-OST003c_UUID 78033218 536265 77496953 0% /lqcdproj[OST:60] lqcdproj-OST003d_UUID 84442817 552605 83890212 0% /lqcdproj[OST:61] lqcdproj-OST003e_UUID 86259843 493272 85766571 0% /lqcdproj[OST:62] lqcdproj-OST003f_UUID 85183502 556212 84627290 0% /lqcdproj[OST:63] lqcdproj-OST0040_UUID 74797227 461299 74335928 0% /lqcdproj[OST:64] lqcdproj-OST0041_UUID 77292670 504863 76787807 0% /lqcdproj[OST:65] lqcdproj-OST0042_UUID 68040780 548464 67492316 0% /lqcdproj[OST:66] lqcdproj-OST0043_UUID 83531122 479362 83051760 0% /lqcdproj[OST:67] lqcdproj-OST0044_UUID 63085241 583977 62501264 0% /lqcdproj[OST:68] lqcdproj-OST0045_UUID 80491631 524898 79966733 0% /lqcdproj[OST:69] lqcdproj-OST0046_UUID 88596519 515565 88080954 0% /lqcdproj[OST:70] lqcdproj-OST0047_UUID 93337404 536811 92800593 0% /lqcdproj[OST:71] lqcdproj-OST0048_UUID 163697829 294432 163403397 0% /lqcdproj[OST:72] lqcdproj-OST0049_UUID 168056920 227725 167829195 0% /lqcdproj[OST:73] lqcdproj-OST004a_UUID 177363178 304244 177058934 0% /lqcdproj[OST:74] lqcdproj-OST004b_UUID 112882331 332592 112549739 0% /lqcdproj[OST:75] lqcdproj-OST004c_UUID 117430752 338304 117092448 0% /lqcdproj[OST:76] lqcdproj-OST004d_UUID 102227261 289660 101937601 0% /lqcdproj[OST:77] lqcdproj-OST004e_UUID 117126345 305097 116821248 0% /lqcdproj[OST:78] lqcdproj-OST004f_UUID 107493346 275573 107217773 0% /lqcdproj[OST:79] lqcdproj-OST0050_UUID 123242579 335057 122907522 0% /lqcdproj[OST:80] lqcdproj-OST0051_UUID 102612033 284195 102327838 0% /lqcdproj[OST:81] lqcdproj-OST0052_UUID 111948022 329028 111618994 0% /lqcdproj[OST:82] lqcdproj-OST0053_UUID 109921804 670914 109250890 0% /lqcdproj[OST:83] lqcdproj-OST0054_UUID 169674053 690918 168983135 0% /lqcdproj[OST:84] lqcdproj-OST0055_UUID 211960472 672696 211287776 0% /lqcdproj[OST:85] lqcdproj-OST0056_UUID 202621560 701380 201920180 0% /lqcdproj[OST:86] lqcdproj-OST0057_UUID 187913696 637891 187275805 0% /lqcdproj[OST:87] lqcdproj-OST0058_UUID 196479918 661338 195818580 0% /lqcdproj[OST:88] lqcdproj-OST0059_UUID 221924473 665078 221259395 0% /lqcdproj[OST:89] lqcdproj-OST005a_UUID 90849842 1207613 89642229 1% /lqcdproj[OST:90] lqcdproj-OST005b_UUID 156881564 1317417 155564147 0% /lqcdproj[OST:91] lqcdproj-OST005c_UUID 115108358 1284790 113823568 1% /lqcdproj[OST:92] lqcdproj-OST005d_UUID 140920614 1231522 139689092 0% /lqcdproj[OST:93] lqcdproj-OST005e_UUID 104563999 2269779 102294220 2% /lqcdproj[OST:94] lqcdproj-OST005f_UUID 131045574 2281480 128764094 1% /lqcdproj[OST:95] lqcdproj-OST0060_UUID 98398451 2296477 96101974 2% /lqcdproj[OST:96] lqcdproj-OST0061_UUID 60841563 2307039 58534524 3% /lqcdproj[OST:97] lqcdproj-OST0062_UUID 100334859 2319175 98015684 2% /lqcdproj[OST:98] lqcdproj-OST0063_UUID 100100892 2320532 97780360 2% /lqcdproj[OST:99] filesystem summary: 83886080 58832324 25053756 70% /lqcdproj -------------- next part -------------- #Generated by fsstats v1.4.5 (03/22/2008) #Comment: This is a comment line that can be modified or repeated before #uploading to record voluntarily added information. skipped special files,28 skipped duplicate hardlinks,87 skipped snapshot dirs,0 total capacity used,308.19 TB total user data,308.06 TB percent overhead,0.000435 histogram,file size count,59934337,items average,5518.788700,KB min,0,KB max,220555264,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,7239631,0.120793,0.120793,8331921.766602,0.000025,0.000025 2,4,5650798,0.094283,0.215076,16761219.459961,0.000051,0.000076 4,8,8378680,0.139798,0.354874,49784206.946289,0.000151,0.000226 8,16,7583802,0.126535,0.481409,78424902.477539,0.000237,0.000463 16,32,4407781,0.073544,0.554952,100330648.391602,0.000303,0.000767 32,64,9505899,0.158605,0.713557,416140865.510742,0.001258,0.002025 64,128,6293548,0.105007,0.818565,555242846.333008,0.001679,0.003704 128,256,3554484,0.059306,0.877871,641521188.999023,0.001940,0.005643 256,512,972568,0.016227,0.894098,377965059.421875,0.001143,0.006786 512,1024,1126589,0.018797,0.912895,817273695.046875,0.002471,0.009257 1024,2048,2136930,0.035655,0.948550,2244063826.294922,0.006784,0.016041 2048,4096,743957,0.012413,0.960963,2091234680.057617,0.006322,0.022364 4096,8192,454489,0.007583,0.968546,2809914289.699219,0.008495,0.030859 8192,16384,53098,0.000886,0.969432,669513076.252930,0.002024,0.032883 16384,32768,961986,0.016051,0.985482,19041429218.568359,0.057568,0.090451 32768,65536,418646,0.006985,0.992468,16517027089.158203,0.049936,0.140387 65536,131072,33220,0.000554,0.993022,2682990338.608398,0.008111,0.148498 131072,262144,260643,0.004349,0.997371,39459804685.265625,0.119299,0.267797 262144,524288,69264,0.001156,0.998526,23084309692.799805,0.069791,0.337587 524288,1048576,25245,0.000421,0.998948,18200044499.855469,0.055024,0.392611 1048576,2097152,36208,0.000604,0.999552,61461741871.820312,0.185817,0.578428 2097152,4194304,16615,0.000277,0.999829,42466967024.889648,0.128390,0.706819 4194304,8388608,8541,0.000143,0.999971,45861839774.806641,0.138654,0.845472 8388608,16777216,280,0.000005,0.999976,3681509558.092773,0.011130,0.856603 16777216,33554432,751,0.000013,0.999989,15376447541.249023,0.046488,0.903090 33554432,67108864,646,0.000011,0.999999,27523378769.218750,0.083211,0.986302 67108864,134217728,20,0.000000,1.000000,1651003773.913086,0.004991,0.991293 134217728,268435456,18,0.000000,1.000000,2879945511.585938,0.008707,1.000000 histogram,capacity used count,59934337,items average,5521.186633,KB min,0,KB max,220555348,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,299584,0.004999,0.004999,0.000000,0.000000,0.000000 4,8,12593332,0.210119,0.215117,50373328.000000,0.000152,0.000152 8,16,13843488,0.230978,0.446095,132609324.000000,0.000401,0.000553 16,32,6278025,0.104748,0.550843,138281212.000000,0.000418,0.000971 32,64,9579236,0.159829,0.710672,435966032.000000,0.001317,0.002288 64,128,6436169,0.107387,0.818059,582412400.000000,0.001760,0.004048 128,256,3560058,0.059399,0.877458,650283864.000000,0.001965,0.006014 256,512,995959,0.016618,0.894076,386528508.000000,0.001168,0.007182 512,1024,1127852,0.018818,0.912894,821320296.000000,0.002482,0.009664 1024,2048,2136832,0.035653,0.948547,2251982384.000000,0.006805,0.016469 2048,4096,744089,0.012415,0.960962,2092982692.000000,0.006325,0.022794 4096,8192,454530,0.007584,0.968546,2811313820.000000,0.008496,0.031290 8192,16384,53107,0.000886,0.969432,669804240.000000,0.002024,0.033314 16384,32768,961979,0.016051,0.985482,19045380536.000000,0.057555,0.090869 32768,65536,418645,0.006985,0.992468,16519196460.000000,0.049921,0.140789 65536,131072,33223,0.000554,0.993022,2683382476.000000,0.008109,0.148899 131072,262144,260643,0.004349,0.997371,39461284736.000000,0.119251,0.268150 262144,524288,69264,0.001156,0.998526,23084702376.000000,0.069762,0.337911 524288,1048576,25246,0.000421,0.998948,18201243276.000000,0.055004,0.392915 1048576,2097152,36206,0.000604,0.999552,61458936304.000000,0.185728,0.578643 2097152,4194304,16615,0.000277,0.999829,42466980604.000000,0.128334,0.706977 4194304,8388608,8541,0.000143,0.999971,45861900636.000000,0.138594,0.845571 8388608,16777216,279,0.000005,0.999976,3670989160.000000,0.011094,0.856665 16777216,33554432,751,0.000013,0.999989,15376456720.000000,0.046467,0.903132 33554432,67108864,646,0.000011,0.999999,27523397012.000000,0.083175,0.986308 67108864,134217728,20,0.000000,1.000000,1651004852.000000,0.004989,0.991297 134217728,268435456,18,0.000000,1.000000,2879947068.000000,0.008703,1.000000 histogram,positive overhead count,60189966,items average,2.600233,KB min,0,KB max,348,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,7266889,0.120733,0.120733,19534879.025391,0.124817,0.124817 2,4,5650798,0.093883,0.214615,5841972.540039,0.037327,0.162144 4,8,8563923,0.142282,0.356897,17235366.162109,0.110125,0.272269 8,16,7594069,0.126168,0.483065,21039137.498047,0.134429,0.406697 16,32,4414329,0.073340,0.556405,11969910.451172,0.076481,0.483179 32,64,9517996,0.158133,0.714538,23155179.557617,0.147949,0.631127 64,128,6299483,0.104660,0.819198,19866348.781250,0.126935,0.758063 128,256,3559085,0.059131,0.878329,11233679.503906,0.071777,0.829840 256,512,975293,0.016204,0.894532,2930022.809570,0.018721,0.848561 512,1024,1127192,0.018727,0.913259,3442788.484375,0.021998,0.870558 1024,2048,2137149,0.035507,0.948766,8201783.161133,0.052405,0.922963 2048,4096,744002,0.012361,0.961127,1568006.220703,0.010019,0.932982 4096,8192,454539,0.007552,0.968679,1240414.223633,0.007926,0.940908 8192,16384,53236,0.000884,0.969563,166869.873047,0.001066,0.941974 16384,32768,961954,0.015982,0.985545,4115411.148438,0.026295,0.968269 32768,65536,418620,0.006955,0.992500,2239686.912109,0.014310,0.982579 65536,131072,33207,0.000552,0.993052,150006.844727,0.000958,0.983538 131072,262144,260639,0.004330,0.997382,1492316.187500,0.009535,0.993073 262144,524288,69263,0.001151,0.998533,399746.551758,0.002554,0.995627 524288,1048576,25244,0.000419,0.998952,162715.746094,0.001040,0.996667 1048576,2097152,36200,0.000601,0.999554,287085.722656,0.001834,0.998501 2097152,4194304,16606,0.000276,0.999830,113718.391602,0.000727,0.999228 4194304,8388608,8536,0.000142,0.999972,88597.248047,0.000566,0.999794 8388608,16777216,279,0.000005,0.999976,2225.907227,0.000014,0.999808 16777216,33554432,751,0.000012,0.999989,9178.750977,0.000059,0.999867 33554432,67108864,646,0.000011,0.999999,18242.781250,0.000117,0.999983 67108864,134217728,20,0.000000,1.000000,1078.086914,0.000007,0.999990 134217728,268435456,18,0.000000,1.000000,1556.414062,0.000010,1.000000 histogram,negative overhead count,5204,items average,2424.471249,KB min,0,KB max,9484284,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,5032,0.966949,0.966949,143.698242,0.000011,0.000011 4,8,5,0.000961,0.967909,25.108398,0.000002,0.000013 8,16,6,0.001153,0.969062,39.975586,0.000003,0.000017 16,32,7,0.001345,0.970407,186.842773,0.000015,0.000031 32,64,6,0.001153,0.971560,293.068359,0.000023,0.000055 64,128,2,0.000384,0.971945,231.114258,0.000018,0.000073 128,256,1,0.000192,0.972137,128.502930,0.000010,0.000083 256,512,5,0.000961,0.973098,678.231445,0.000054,0.000137 512,1024,1,0.000192,0.973290,3.531250,0.000000,0.000137 1024,2048,16,0.003075,0.976364,673.456055,0.000053,0.000191 2048,4096,10,0.001922,0.978286,6110.278320,0.000484,0.000675 4096,8192,2,0.000384,0.978670,67.922852,0.000005,0.000680 8192,16384,8,0.001537,0.980208,446.125977,0.000035,0.000716 16384,32768,35,0.006726,0.986933,35905.716797,0.002846,0.003561 32768,65536,26,0.004996,0.991929,22184.070312,0.001758,0.005320 65536,131072,13,0.002498,0.994427,23097.453125,0.001831,0.007150 131072,262144,4,0.000769,0.995196,12265.453125,0.000972,0.008122 262144,524288,1,0.000192,0.995388,7063.351562,0.000560,0.008682 524288,1048576,1,0.000192,0.995580,2279.601562,0.000181,0.008863 1048576,2097152,8,0.001537,0.997118,2892965.542969,0.229292,0.238155 2097152,4194304,9,0.001729,0.998847,100139.281250,0.007937,0.246092 4194304,8388608,5,0.000961,0.999808,27736.054688,0.002198,0.248290 8388608,16777216,1,0.000192,1.000000,9484284.000000,0.751710,1.000000 histogram,directory size (entries) count,228543,items average,263.387126,ents min,0,ents max,288415,ents bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,1,38730,0.169465,0.169465,18448.000000,0.000306,0.000306 2,3,27366,0.119741,0.289206,67758.000000,0.001126,0.001432 4,7,31986,0.139956,0.429162,159107.000000,0.002643,0.004075 8,15,42698,0.186827,0.615989,431811.000000,0.007174,0.011249 16,31,23960,0.104838,0.720827,505863.000000,0.008404,0.019652 32,63,16548,0.072407,0.793234,711434.000000,0.011819,0.031471 64,127,10943,0.047882,0.841115,919782.000000,0.015280,0.046751 128,255,7530,0.032948,0.874063,1306575.000000,0.021706,0.068457 256,511,5591,0.024464,0.898527,2258238.000000,0.037515,0.105972 512,1023,10391,0.045466,0.943993,7008296.000000,0.116426,0.222398 1024,2047,5497,0.024052,0.968045,7930133.000000,0.131740,0.354138 2048,4095,4933,0.021585,0.989630,13612099.000000,0.226132,0.580270 4096,8191,1631,0.007137,0.996766,8585977.000000,0.142635,0.722906 8192,16383,515,0.002253,0.999020,5516225.000000,0.091639,0.814545 16384,32767,117,0.000512,0.999532,2709560.000000,0.045013,0.859557 32768,65535,49,0.000214,0.999746,2377922.000000,0.039503,0.899061 65536,131071,43,0.000188,0.999934,3795052.000000,0.063046,0.962107 131072,262143,14,0.000061,0.999996,1992589.000000,0.033102,0.995209 262144,524287,1,0.000004,1.000000,288415.000000,0.004791,1.000000 histogram,directory size count,228543,items average,29.017069,KB min,4,KB max,26988,KB bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 4,8,185248,0.810561,0.810561,740992.000000,0.111736,0.111736 8,16,10273,0.044950,0.855511,123276.000000,0.018589,0.130325 16,32,6555,0.028682,0.884192,133784.000000,0.020174,0.150498 32,64,12103,0.052957,0.937150,485908.000000,0.073271,0.223769 64,128,5937,0.025978,0.963127,537204.000000,0.081006,0.304776 128,256,4602,0.020136,0.983264,800756.000000,0.120748,0.425523 256,512,2730,0.011945,0.995209,908160.000000,0.136943,0.562467 512,1024,604,0.002643,0.997852,409624.000000,0.061768,0.624235 1024,2048,235,0.001028,0.998880,295492.000000,0.044558,0.668792 2048,4096,55,0.000241,0.999121,165772.000000,0.024997,0.693790 4096,8192,52,0.000228,0.999348,331200.000000,0.049942,0.743732 8192,16384,146,0.000639,0.999987,1637404.000000,0.246908,0.990639 16384,32768,3,0.000013,1.000000,62076.000000,0.009361,1.000000 histogram,filename length count,60195285,items average,41.713233,chars min,1,chars max,220,chars bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,7,433803,0.007207,0.007207,2272792.000000,0.000905,0.000905 8,15,1628761,0.027058,0.034265,19094598.000000,0.007605,0.008510 16,23,2771542,0.046043,0.080307,54241521.000000,0.021602,0.030112 24,31,3856254,0.064062,0.144369,106932976.000000,0.042587,0.072699 32,39,14899176,0.247514,0.391883,521744672.000000,0.207789,0.280487 40,47,17344424,0.288136,0.680019,734036771.000000,0.292335,0.572823 48,55,13751667,0.228451,0.908470,704938238.000000,0.280747,0.853569 56,63,2346500,0.038981,0.947452,138069584.000000,0.054987,0.908557 64,71,1278004,0.021231,0.968683,85515974.000000,0.034057,0.942614 72,79,1444484,0.023997,0.992679,105925289.000000,0.042186,0.984800 80,87,324627,0.005393,0.998072,26883013.000000,0.010706,0.995506 88,95,37223,0.000618,0.998691,3329140.000000,0.001326,0.996832 96,103,59953,0.000996,0.999687,5916560.000000,0.002356,0.999188 104,111,15596,0.000259,0.999946,1654085.000000,0.000659,0.999847 112,119,3144,0.000052,0.999998,367669.000000,0.000146,0.999993 120,127,86,0.000001,0.999999,10498.000000,0.000004,0.999997 127,-1,41,0.000001,1.000000,6570.000000,0.000003,1.000000 histogram,link count count,59934337,items average,1.000001,links min,1,links max,7,links bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,1,59934278,0.999999,0.999999,59934278.000000,0.999998,0.999998 2,3,52,0.000001,1.000000,111.000000,0.000002,0.999999 4,7,7,0.000000,1.000000,35.000000,0.000001,1.000000 histogram,symlink target length count,32290,items average,84.320842,chars min,5,chars max,154,chars bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,7,21,0.000650,0.000650,119.000000,0.000044,0.000044 8,15,651,0.020161,0.020811,8328.000000,0.003059,0.003102 16,23,1686,0.052214,0.073026,31548.000000,0.011587,0.014689 24,31,1229,0.038061,0.111087,33849.000000,0.012432,0.027121 32,39,256,0.007928,0.119015,8633.000000,0.003171,0.030292 40,47,191,0.005915,0.124930,8240.000000,0.003026,0.033319 48,55,538,0.016662,0.141592,28122.000000,0.010329,0.043647 56,63,1141,0.035336,0.176928,68604.000000,0.025197,0.068844 64,71,1564,0.048436,0.225364,105964.000000,0.038918,0.107762 72,79,1451,0.044937,0.270300,109262.000000,0.040130,0.147892 80,87,1534,0.047507,0.317807,126146.000000,0.046331,0.194223 88,95,16648,0.515578,0.833385,1483365.000000,0.544810,0.739033 96,103,679,0.021028,0.854413,67409.000000,0.024758,0.763791 104,111,637,0.019727,0.874141,67713.000000,0.024870,0.788661 112,119,760,0.023537,0.897677,87181.000000,0.032020,0.820680 120,127,319,0.009879,0.907557,39180.000000,0.014390,0.835070 127,-1,2985,0.092443,1.000000,449057.000000,0.164930,1.000000 relative symlink target pct,0.208640 absolute symlink target pct,0.791360 histogram,mtime (files) count,59934337,items average,619.753158,days min,-1063,days max,15087,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct -1,0,56,0.000001,0.000001,-15888.501505,-0.000000,-0.000000 0,2,35412,0.000591,0.000592,34078.029919,0.000001,0.000000 2,4,29491,0.000492,0.001084,77373.040208,0.000002,0.000003 4,8,49796,0.000831,0.001915,301158.255359,0.000008,0.000011 8,16,673987,0.011245,0.013160,7950711.065798,0.000214,0.000225 16,32,1105851,0.018451,0.031611,27403040.467512,0.000738,0.000962 32,64,2338818,0.039023,0.070634,115616097.341384,0.003113,0.004075 64,128,6245411,0.104204,0.174838,562851103.337436,0.015153,0.019228 128,256,9189205,0.153321,0.328160,1718705208.960418,0.046271,0.065499 256,512,10686750,0.178308,0.506467,3778926028.425153,0.101736,0.167235 512,1024,18277796,0.304964,0.811431,12128383407.198038,0.326519,0.493754 1024,2048,9548349,0.159314,0.970744,14533495124.026802,0.391269,0.885023 2048,4096,1735825,0.028962,0.999707,4043797267.039390,0.108867,0.993890 4096,8192,3673,0.000061,0.999768,17041460.210277,0.000459,0.994348 8192,16384,13917,0.000232,1.000000,209928463.075278,0.005652,1.000000 histogram,mtime (KB) count,330764941776,items average,258.427355,days min,-1063,days max,15087,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct -1,0,76603,0.000000,0.000000,-115.152030,-0.000000,-0.000000 0,2,1106041649,0.003344,0.003344,1366476314.626506,0.000016,0.000016 2,4,1385813726,0.004190,0.007534,3966654932.773013,0.000046,0.000062 4,8,2677064724,0.008094,0.015627,15363660036.922892,0.000180,0.000242 8,16,10772940907,0.032570,0.048197,135762404168.481506,0.001588,0.001830 16,32,20415381579,0.061722,0.109919,467250388017.326294,0.005466,0.007297 32,64,23160971600,0.070022,0.179941,1121567142378.384521,0.013121,0.020418 64,128,83081569774,0.251180,0.431121,8257626650042.665039,0.096604,0.117022 128,256,65411975006,0.197760,0.628881,11578341200630.927734,0.135453,0.252475 256,512,63106694110,0.190790,0.819671,22594455048531.875000,0.264328,0.516804 512,1024,58833004426,0.177870,0.997541,38996626749493.195312,0.456215,0.973018 1024,2048,643646022,0.001946,0.999487,986397036736.210327,0.011540,0.984558 2048,4096,97400319,0.000294,0.999781,228338238534.346313,0.002671,0.987229 4096,8192,5014,0.000000,0.999781,24886886.571377,0.000000,0.987229 8192,16384,72356309,0.000219,1.000000,1091622329101.521118,0.012771,1.000000 histogram,ctime (files) count,59934337,items average,129.305593,days min,0,days max,773,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct -1,0,40,0.000001,0.000001,-0.039352,-0.000000,-0.000000 0,2,357834,0.005970,0.005971,665384.618843,0.000086,0.000086 2,4,809439,0.013505,0.019477,1817661.354687,0.000235,0.000320 4,8,41020,0.000684,0.020161,247938.627558,0.000032,0.000352 8,16,560317,0.009349,0.029510,6581167.903299,0.000849,0.001202 16,32,918100,0.015318,0.044828,23949354.713726,0.003090,0.004292 32,64,2168158,0.036176,0.081004,109801036.524535,0.014168,0.018460 64,128,39481597,0.658748,0.739751,2842059951.338632,0.366725,0.385185 128,256,7722024,0.128841,0.868593,1537800781.225099,0.198430,0.583615 256,512,5770445,0.096279,0.964872,1991800261.622381,0.257012,0.840626 512,1024,2105363,0.035128,1.000000,1235121475.975745,0.159374,1.000000 histogram,ctime (KB) count,330764941776,items average,182.457305,days min,0,days max,773,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct -1,0,76603,0.000000,0.000000,-115.152030,-0.000000,-0.000000 0,2,3442662193,0.010408,0.010408,4057745143.277389,0.000067,0.000067 2,4,5051567075,0.015272,0.025681,15346835252.232616,0.000254,0.000322 4,8,5102405849,0.015426,0.041107,30530827648.932316,0.000506,0.000827 8,16,25230584561,0.076280,0.117386,299866572253.154358,0.004969,0.005796 16,32,19596489529,0.059246,0.176632,462425687390.829590,0.007662,0.013459 32,64,29725835959,0.089870,0.266502,1351978211181.035400,0.022402,0.035861 64,128,74853917442,0.226305,0.492808,7400118058516.604492,0.122619,0.158480 128,256,91711062311,0.277270,0.770077,16388147495021.320312,0.271550,0.430029 256,512,46591104178,0.140859,0.910936,17169818592094.595703,0.284502,0.714531 512,1024,29459236070,0.089064,1.000000,17228189779033.849609,0.285469,1.000000 histogram,atime (files) count,59934337,items average,96.989003,days min,0,days max,892,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,22500822,0.375425,0.375425,20742470.051829,0.003568,0.003568 2,4,5070236,0.084597,0.460021,11414473.060420,0.001964,0.005532 4,8,5858988,0.097757,0.557778,40226580.718701,0.006920,0.012452 8,16,1829136,0.030519,0.588297,19643192.356613,0.003379,0.015831 16,32,785070,0.013099,0.601396,19712871.393561,0.003391,0.019222 32,64,2789384,0.046541,0.647936,128951841.125802,0.022183,0.041406 64,128,5747891,0.095903,0.743839,499724084.617960,0.085967,0.127373 128,256,5243228,0.087483,0.831322,1000102852.628599,0.172047,0.299420 256,512,7652623,0.127683,0.959006,2581858985.146958,0.444155,0.743574 512,1024,2456959,0.040994,1.000000,1490594239.302162,0.256426,1.000000 histogram,atime (KB) count,330764941776,items average,177.003264,days min,0,days max,892,days bucket min,bucket max,count,percent,cumulative pct,val count,percent,cumulative pct 0,2,5827271517,0.017618,0.017618,5872231564.030893,0.000100,0.000100 2,4,6266426014,0.018945,0.036563,17948937149.600842,0.000307,0.000407 4,8,8352467112,0.025252,0.061815,52384514577.010384,0.000895,0.001302 8,16,28923846298,0.087445,0.149260,347359021276.677917,0.005933,0.007235 16,32,26633629587,0.080521,0.229781,650880664462.987427,0.011117,0.018352 32,64,37262787520,0.112656,0.342438,1752906299042.393066,0.029940,0.048292 64,128,67195085361,0.203151,0.545588,6635947781217.270508,0.113345,0.161637 128,256,70775892016,0.213976,0.759565,12534779725391.322266,0.214100,0.375737 256,512,42183779470,0.127534,0.887099,14510417023055.207031,0.247844,0.623581 512,1024,37343756875,0.112901,1.000000,22037978254075.386719,0.376419,1.000000 -------------- next part -------------- ----- CUT HERE ----- REPORT FOLLOWS ----- CUT HERE ----- Generated by fsstats v1.4.5 (03/22/2008) complete : processed 60195285 files in 76547 secs (at 786.38 files/sec) Skipped 87 duplicate hardlinked files Skipped 28 special files CSV WRITTEN TO lqcd-lustre-fsstats ----- BEGIN NORMAL ----- total 308.19 TB used to store 308.06 TB user data, overhead 0.04% file size (Regular file's last addressable byte): Range, Count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of files, total size of files in range, total size of files in range as a % of total size of files, total size of files in this range or smaller as a % of total size of files. count=59934337 avg=5518.79 KB min=0.00 KB max=220555264.00 KB [ 0- 2 KB): 7239631 (12.08%) ( 12.08% cumulative) 8331921.77 KB ( 0.00%) ( 0.00% cumulative) [ 2- 4 KB): 5650798 ( 9.43%) ( 21.51% cumulative) 16761219.46 KB ( 0.01%) ( 0.01% cumulative) [ 4- 8 KB): 8378680 (13.98%) ( 35.49% cumulative) 49784206.95 KB ( 0.02%) ( 0.02% cumulative) [ 8- 16 KB): 7583802 (12.65%) ( 48.14% cumulative) 78424902.48 KB ( 0.02%) ( 0.05% cumulative) [ 16- 32 KB): 4407781 ( 7.35%) ( 55.50% cumulative) 100330648.39 KB ( 0.03%) ( 0.08% cumulative) [ 32- 64 KB): 9505899 (15.86%) ( 71.36% cumulative) 416140865.51 KB ( 0.13%) ( 0.20% cumulative) [ 64- 128 KB): 6293548 (10.50%) ( 81.86% cumulative) 555242846.33 KB ( 0.17%) ( 0.37% cumulative) [ 128- 256 KB): 3554484 ( 5.93%) ( 87.79% cumulative) 641521189.00 KB ( 0.19%) ( 0.56% cumulative) [ 256- 512 KB): 972568 ( 1.62%) ( 89.41% cumulative) 377965059.42 KB ( 0.11%) ( 0.68% cumulative) [ 512- 1024 KB): 1126589 ( 1.88%) ( 91.29% cumulative) 817273695.05 KB ( 0.25%) ( 0.93% cumulative) [ 1024- 2048 KB): 2136930 ( 3.57%) ( 94.85% cumulative) 2244063826.29 KB ( 0.68%) ( 1.60% cumulative) [ 2048- 4096 KB): 743957 ( 1.24%) ( 96.10% cumulative) 2091234680.06 KB ( 0.63%) ( 2.24% cumulative) [ 4096- 8192 KB): 454489 ( 0.76%) ( 96.85% cumulative) 2809914289.70 KB ( 0.85%) ( 3.09% cumulative) [ 8192- 16384 KB): 53098 ( 0.09%) ( 96.94% cumulative) 669513076.25 KB ( 0.20%) ( 3.29% cumulative) [ 16384- 32768 KB): 961986 ( 1.61%) ( 98.55% cumulative) 19041429218.57 KB ( 5.76%) ( 9.05% cumulative) [ 32768- 65536 KB): 418646 ( 0.70%) ( 99.25% cumulative) 16517027089.16 KB ( 4.99%) ( 14.04% cumulative) [ 65536- 131072 KB): 33220 ( 0.06%) ( 99.30% cumulative) 2682990338.61 KB ( 0.81%) ( 14.85% cumulative) [ 131072- 262144 KB): 260643 ( 0.43%) ( 99.74% cumulative) 39459804685.27 KB (11.93%) ( 26.78% cumulative) [ 262144- 524288 KB): 69264 ( 0.12%) ( 99.85% cumulative) 23084309692.80 KB ( 6.98%) ( 33.76% cumulative) [ 524288- 1048576 KB): 25245 ( 0.04%) ( 99.89% cumulative) 18200044499.86 KB ( 5.50%) ( 39.26% cumulative) [ 1048576- 2097152 KB): 36208 ( 0.06%) ( 99.96% cumulative) 61461741871.82 KB (18.58%) ( 57.84% cumulative) [ 2097152- 4194304 KB): 16615 ( 0.03%) ( 99.98% cumulative) 42466967024.89 KB (12.84%) ( 70.68% cumulative) [ 4194304- 8388608 KB): 8541 ( 0.01%) (100.00% cumulative) 45861839774.81 KB (13.87%) ( 84.55% cumulative) [ 8388608- 16777216 KB): 280 ( 0.00%) (100.00% cumulative) 3681509558.09 KB ( 1.11%) ( 85.66% cumulative) [ 16777216- 33554432 KB): 751 ( 0.00%) (100.00% cumulative) 15376447541.25 KB ( 4.65%) ( 90.31% cumulative) [ 33554432- 67108864 KB): 646 ( 0.00%) (100.00% cumulative) 27523378769.22 KB ( 8.32%) ( 98.63% cumulative) [ 67108864-134217728 KB): 20 ( 0.00%) (100.00% cumulative) 1651003773.91 KB ( 0.50%) ( 99.13% cumulative) [134217728-268435456 KB): 18 ( 0.00%) (100.00% cumulative) 2879945511.59 KB ( 0.87%) (100.00% cumulative) capacity used (Regualr file's space used on disk): Range, Count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of file, total capacity used in range, total capacity used in range as a % of total capacity used, total capacity used in this range or smaller as a % of total capacity. count=59934337 avg=5521.19 KB min=0.00 KB max=220555348.00 KB [ 0- 2 KB): 299584 ( 0.50%) ( 0.50% cumulative) 0.00 KB ( 0.00%) ( 0.00% cumulative) [ 4- 8 KB): 12593332 (21.01%) ( 21.51% cumulative) 50373328.00 KB ( 0.02%) ( 0.02% cumulative) [ 8- 16 KB): 13843488 (23.10%) ( 44.61% cumulative) 132609324.00 KB ( 0.04%) ( 0.06% cumulative) [ 16- 32 KB): 6278025 (10.47%) ( 55.08% cumulative) 138281212.00 KB ( 0.04%) ( 0.10% cumulative) [ 32- 64 KB): 9579236 (15.98%) ( 71.07% cumulative) 435966032.00 KB ( 0.13%) ( 0.23% cumulative) [ 64- 128 KB): 6436169 (10.74%) ( 81.81% cumulative) 582412400.00 KB ( 0.18%) ( 0.40% cumulative) [ 128- 256 KB): 3560058 ( 5.94%) ( 87.75% cumulative) 650283864.00 KB ( 0.20%) ( 0.60% cumulative) [ 256- 512 KB): 995959 ( 1.66%) ( 89.41% cumulative) 386528508.00 KB ( 0.12%) ( 0.72% cumulative) [ 512- 1024 KB): 1127852 ( 1.88%) ( 91.29% cumulative) 821320296.00 KB ( 0.25%) ( 0.97% cumulative) [ 1024- 2048 KB): 2136832 ( 3.57%) ( 94.85% cumulative) 2251982384.00 KB ( 0.68%) ( 1.65% cumulative) [ 2048- 4096 KB): 744089 ( 1.24%) ( 96.10% cumulative) 2092982692.00 KB ( 0.63%) ( 2.28% cumulative) [ 4096- 8192 KB): 454530 ( 0.76%) ( 96.85% cumulative) 2811313820.00 KB ( 0.85%) ( 3.13% cumulative) [ 8192- 16384 KB): 53107 ( 0.09%) ( 96.94% cumulative) 669804240.00 KB ( 0.20%) ( 3.33% cumulative) [ 16384- 32768 KB): 961979 ( 1.61%) ( 98.55% cumulative) 19045380536.00 KB ( 5.76%) ( 9.09% cumulative) [ 32768- 65536 KB): 418645 ( 0.70%) ( 99.25% cumulative) 16519196460.00 KB ( 4.99%) ( 14.08% cumulative) [ 65536- 131072 KB): 33223 ( 0.06%) ( 99.30% cumulative) 2683382476.00 KB ( 0.81%) ( 14.89% cumulative) [ 131072- 262144 KB): 260643 ( 0.43%) ( 99.74% cumulative) 39461284736.00 KB (11.93%) ( 26.81% cumulative) [ 262144- 524288 KB): 69264 ( 0.12%) ( 99.85% cumulative) 23084702376.00 KB ( 6.98%) ( 33.79% cumulative) [ 524288- 1048576 KB): 25246 ( 0.04%) ( 99.89% cumulative) 18201243276.00 KB ( 5.50%) ( 39.29% cumulative) [ 1048576- 2097152 KB): 36206 ( 0.06%) ( 99.96% cumulative) 61458936304.00 KB (18.57%) ( 57.86% cumulative) [ 2097152- 4194304 KB): 16615 ( 0.03%) ( 99.98% cumulative) 42466980604.00 KB (12.83%) ( 70.70% cumulative) [ 4194304- 8388608 KB): 8541 ( 0.01%) (100.00% cumulative) 45861900636.00 KB (13.86%) ( 84.56% cumulative) [ 8388608- 16777216 KB): 279 ( 0.00%) (100.00% cumulative) 3670989160.00 KB ( 1.11%) ( 85.67% cumulative) [ 16777216- 33554432 KB): 751 ( 0.00%) (100.00% cumulative) 15376456720.00 KB ( 4.65%) ( 90.31% cumulative) [ 33554432- 67108864 KB): 646 ( 0.00%) (100.00% cumulative) 27523397012.00 KB ( 8.32%) ( 98.63% cumulative) [ 67108864-134217728 KB): 20 ( 0.00%) (100.00% cumulative) 1651004852.00 KB ( 0.50%) ( 99.13% cumulative) [134217728-268435456 KB): 18 ( 0.00%) (100.00% cumulative) 2879947068.00 KB ( 0.87%) (100.00% cumulative) positive overhead (Files whose capacity used is larger than or equal to its size. Includes directories, symlinks and regular files): Range of file sizes, count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of files, total bytes that overflowed in range, total bytes that overflowed in this range as a % of total bytes that overflowed, total bytes that overflowed in this range or smaller as a % of total bytes that overflowed. count=60189966 avg=2.60 KB min=0.00 KB max=348.03 KB [ 0- 2 KB): 7266889 (12.07%) ( 12.07% cumulative) 19534879.03 KB (12.48%) ( 12.48% cumulative) [ 2- 4 KB): 5650798 ( 9.39%) ( 21.46% cumulative) 5841972.54 KB ( 3.73%) ( 16.21% cumulative) [ 4- 8 KB): 8563923 (14.23%) ( 35.69% cumulative) 17235366.16 KB (11.01%) ( 27.23% cumulative) [ 8- 16 KB): 7594069 (12.62%) ( 48.31% cumulative) 21039137.50 KB (13.44%) ( 40.67% cumulative) [ 16- 32 KB): 4414329 ( 7.33%) ( 55.64% cumulative) 11969910.45 KB ( 7.65%) ( 48.32% cumulative) [ 32- 64 KB): 9517996 (15.81%) ( 71.45% cumulative) 23155179.56 KB (14.79%) ( 63.11% cumulative) [ 64- 128 KB): 6299483 (10.47%) ( 81.92% cumulative) 19866348.78 KB (12.69%) ( 75.81% cumulative) [ 128- 256 KB): 3559085 ( 5.91%) ( 87.83% cumulative) 11233679.50 KB ( 7.18%) ( 82.98% cumulative) [ 256- 512 KB): 975293 ( 1.62%) ( 89.45% cumulative) 2930022.81 KB ( 1.87%) ( 84.86% cumulative) [ 512- 1024 KB): 1127192 ( 1.87%) ( 91.33% cumulative) 3442788.48 KB ( 2.20%) ( 87.06% cumulative) [ 1024- 2048 KB): 2137149 ( 3.55%) ( 94.88% cumulative) 8201783.16 KB ( 5.24%) ( 92.30% cumulative) [ 2048- 4096 KB): 744002 ( 1.24%) ( 96.11% cumulative) 1568006.22 KB ( 1.00%) ( 93.30% cumulative) [ 4096- 8192 KB): 454539 ( 0.76%) ( 96.87% cumulative) 1240414.22 KB ( 0.79%) ( 94.09% cumulative) [ 8192- 16384 KB): 53236 ( 0.09%) ( 96.96% cumulative) 166869.87 KB ( 0.11%) ( 94.20% cumulative) [ 16384- 32768 KB): 961954 ( 1.60%) ( 98.55% cumulative) 4115411.15 KB ( 2.63%) ( 96.83% cumulative) [ 32768- 65536 KB): 418620 ( 0.70%) ( 99.25% cumulative) 2239686.91 KB ( 1.43%) ( 98.26% cumulative) [ 65536- 131072 KB): 33207 ( 0.06%) ( 99.31% cumulative) 150006.84 KB ( 0.10%) ( 98.35% cumulative) [ 131072- 262144 KB): 260639 ( 0.43%) ( 99.74% cumulative) 1492316.19 KB ( 0.95%) ( 99.31% cumulative) [ 262144- 524288 KB): 69263 ( 0.12%) ( 99.85% cumulative) 399746.55 KB ( 0.26%) ( 99.56% cumulative) [ 524288- 1048576 KB): 25244 ( 0.04%) ( 99.90% cumulative) 162715.75 KB ( 0.10%) ( 99.67% cumulative) [ 1048576- 2097152 KB): 36200 ( 0.06%) ( 99.96% cumulative) 287085.72 KB ( 0.18%) ( 99.85% cumulative) [ 2097152- 4194304 KB): 16606 ( 0.03%) ( 99.98% cumulative) 113718.39 KB ( 0.07%) ( 99.92% cumulative) [ 4194304- 8388608 KB): 8536 ( 0.01%) (100.00% cumulative) 88597.25 KB ( 0.06%) ( 99.98% cumulative) [ 8388608- 16777216 KB): 279 ( 0.00%) (100.00% cumulative) 2225.91 KB ( 0.00%) ( 99.98% cumulative) [ 16777216- 33554432 KB): 751 ( 0.00%) (100.00% cumulative) 9178.75 KB ( 0.01%) ( 99.99% cumulative) [ 33554432- 67108864 KB): 646 ( 0.00%) (100.00% cumulative) 18242.78 KB ( 0.01%) (100.00% cumulative) [ 67108864-134217728 KB): 20 ( 0.00%) (100.00% cumulative) 1078.09 KB ( 0.00%) (100.00% cumulative) [134217728-268435456 KB): 18 ( 0.00%) (100.00% cumulative) 1556.41 KB ( 0.00%) (100.00% cumulative) negative overhead (Files whose size is larger than capacity used. Includes directories, symlinks and regular files): Range in file size, count of files in range, number of files in range as a percent of total number of files, number of files in this range or smaller as a % of total # of files, total bytes that underflowed in range, total bytes that underflowed in range as a % of total bytes that underflowed, total bytes that underflowed in this range or smaller as a % total bytes that underflowed. count=5204 avg=2424.47 KB min=0.00 KB max=9484284.00 KB [ 0- 2 KB): 5032 (96.69%) ( 96.69% cumulative) 143.70 KB ( 0.00%) ( 0.00% cumulative) [ 4- 8 KB): 5 ( 0.10%) ( 96.79% cumulative) 25.11 KB ( 0.00%) ( 0.00% cumulative) [ 8- 16 KB): 6 ( 0.12%) ( 96.91% cumulative) 39.98 KB ( 0.00%) ( 0.00% cumulative) [ 16- 32 KB): 7 ( 0.13%) ( 97.04% cumulative) 186.84 KB ( 0.00%) ( 0.00% cumulative) [ 32- 64 KB): 6 ( 0.12%) ( 97.16% cumulative) 293.07 KB ( 0.00%) ( 0.01% cumulative) [ 64- 128 KB): 2 ( 0.04%) ( 97.19% cumulative) 231.11 KB ( 0.00%) ( 0.01% cumulative) [ 128- 256 KB): 1 ( 0.02%) ( 97.21% cumulative) 128.50 KB ( 0.00%) ( 0.01% cumulative) [ 256- 512 KB): 5 ( 0.10%) ( 97.31% cumulative) 678.23 KB ( 0.01%) ( 0.01% cumulative) [ 512- 1024 KB): 1 ( 0.02%) ( 97.33% cumulative) 3.53 KB ( 0.00%) ( 0.01% cumulative) [ 1024- 2048 KB): 16 ( 0.31%) ( 97.64% cumulative) 673.46 KB ( 0.01%) ( 0.02% cumulative) [ 2048- 4096 KB): 10 ( 0.19%) ( 97.83% cumulative) 6110.28 KB ( 0.05%) ( 0.07% cumulative) [ 4096- 8192 KB): 2 ( 0.04%) ( 97.87% cumulative) 67.92 KB ( 0.00%) ( 0.07% cumulative) [ 8192- 16384 KB): 8 ( 0.15%) ( 98.02% cumulative) 446.13 KB ( 0.00%) ( 0.07% cumulative) [ 16384- 32768 KB): 35 ( 0.67%) ( 98.69% cumulative) 35905.72 KB ( 0.28%) ( 0.36% cumulative) [ 32768- 65536 KB): 26 ( 0.50%) ( 99.19% cumulative) 22184.07 KB ( 0.18%) ( 0.53% cumulative) [ 65536- 131072 KB): 13 ( 0.25%) ( 99.44% cumulative) 23097.45 KB ( 0.18%) ( 0.72% cumulative) [ 131072- 262144 KB): 4 ( 0.08%) ( 99.52% cumulative) 12265.45 KB ( 0.10%) ( 0.81% cumulative) [ 262144- 524288 KB): 1 ( 0.02%) ( 99.54% cumulative) 7063.35 KB ( 0.06%) ( 0.87% cumulative) [ 524288- 1048576 KB): 1 ( 0.02%) ( 99.56% cumulative) 2279.60 KB ( 0.02%) ( 0.89% cumulative) [ 1048576- 2097152 KB): 8 ( 0.15%) ( 99.71% cumulative) 2892965.54 KB (22.93%) ( 23.82% cumulative) [ 2097152- 4194304 KB): 9 ( 0.17%) ( 99.88% cumulative) 100139.28 KB ( 0.79%) ( 24.61% cumulative) [ 4194304- 8388608 KB): 5 ( 0.10%) ( 99.98% cumulative) 27736.05 KB ( 0.22%) ( 24.83% cumulative) [ 8388608-16777216 KB): 1 ( 0.02%) (100.00% cumulative) 9484284.00 KB (75.17%) (100.00% cumulative) directory size (entries): Range of entries, count of directories in range, number of dirs in range as a % of total num of dirs, number of dirs in this range or smaller as a % total number of dirs, total entries in range, number of entries in range as a % of total number of entries, number of entries in this range or smaller as a % of total number of entries. count=228543 avg=263.39 ents min=0.00 ents max=288415.00 ents [ 0- 1 ents]: 38730 (16.95%) ( 16.95% cumulative) 18448.00 ents ( 0.03%) ( 0.03% cumulative) [ 2- 3 ents]: 27366 (11.97%) ( 28.92% cumulative) 67758.00 ents ( 0.11%) ( 0.14% cumulative) [ 4- 7 ents]: 31986 (14.00%) ( 42.92% cumulative) 159107.00 ents ( 0.26%) ( 0.41% cumulative) [ 8- 15 ents]: 42698 (18.68%) ( 61.60% cumulative) 431811.00 ents ( 0.72%) ( 1.12% cumulative) [ 16- 31 ents]: 23960 (10.48%) ( 72.08% cumulative) 505863.00 ents ( 0.84%) ( 1.97% cumulative) [ 32- 63 ents]: 16548 ( 7.24%) ( 79.32% cumulative) 711434.00 ents ( 1.18%) ( 3.15% cumulative) [ 64- 127 ents]: 10943 ( 4.79%) ( 84.11% cumulative) 919782.00 ents ( 1.53%) ( 4.68% cumulative) [ 128- 255 ents]: 7530 ( 3.29%) ( 87.41% cumulative) 1306575.00 ents ( 2.17%) ( 6.85% cumulative) [ 256- 511 ents]: 5591 ( 2.45%) ( 89.85% cumulative) 2258238.00 ents ( 3.75%) ( 10.60% cumulative) [ 512- 1023 ents]: 10391 ( 4.55%) ( 94.40% cumulative) 7008296.00 ents (11.64%) ( 22.24% cumulative) [ 1024- 2047 ents]: 5497 ( 2.41%) ( 96.80% cumulative) 7930133.00 ents (13.17%) ( 35.41% cumulative) [ 2048- 4095 ents]: 4933 ( 2.16%) ( 98.96% cumulative) 13612099.00 ents (22.61%) ( 58.03% cumulative) [ 4096- 8191 ents]: 1631 ( 0.71%) ( 99.68% cumulative) 8585977.00 ents (14.26%) ( 72.29% cumulative) [ 8192- 16383 ents]: 515 ( 0.23%) ( 99.90% cumulative) 5516225.00 ents ( 9.16%) ( 81.45% cumulative) [ 16384- 32767 ents]: 117 ( 0.05%) ( 99.95% cumulative) 2709560.00 ents ( 4.50%) ( 85.96% cumulative) [ 32768- 65535 ents]: 49 ( 0.02%) ( 99.97% cumulative) 2377922.00 ents ( 3.95%) ( 89.91% cumulative) [ 65536-131071 ents]: 43 ( 0.02%) ( 99.99% cumulative) 3795052.00 ents ( 6.30%) ( 96.21% cumulative) [131072-262143 ents]: 14 ( 0.01%) (100.00% cumulative) 1992589.00 ents ( 3.31%) ( 99.52% cumulative) [262144-524287 ents]: 1 ( 0.00%) (100.00% cumulative) 288415.00 ents ( 0.48%) (100.00% cumulative) directory size (KB's): Range of dir sizes, count of directories in range, number of dirs in range as a % of total num of dirs, number of dirs in this range or smaller as a % total number of dirs, total size of dirs in range, total size of dirs in range as a % of total size of dirs, total size of dirs in this range or smaller as a % of total size of dirs. count=228543 avg=29.02 KB min=4.00 KB max=26988.00 KB [ 4- 8 KB): 185248 (81.06%) ( 81.06% cumulative) 740992.00 KB (11.17%) ( 11.17% cumulative) [ 8- 16 KB): 10273 ( 4.49%) ( 85.55% cumulative) 123276.00 KB ( 1.86%) ( 13.03% cumulative) [ 16- 32 KB): 6555 ( 2.87%) ( 88.42% cumulative) 133784.00 KB ( 2.02%) ( 15.05% cumulative) [ 32- 64 KB): 12103 ( 5.30%) ( 93.71% cumulative) 485908.00 KB ( 7.33%) ( 22.38% cumulative) [ 64- 128 KB): 5937 ( 2.60%) ( 96.31% cumulative) 537204.00 KB ( 8.10%) ( 30.48% cumulative) [ 128- 256 KB): 4602 ( 2.01%) ( 98.33% cumulative) 800756.00 KB (12.07%) ( 42.55% cumulative) [ 256- 512 KB): 2730 ( 1.19%) ( 99.52% cumulative) 908160.00 KB (13.69%) ( 56.25% cumulative) [ 512- 1024 KB): 604 ( 0.26%) ( 99.79% cumulative) 409624.00 KB ( 6.18%) ( 62.42% cumulative) [ 1024- 2048 KB): 235 ( 0.10%) ( 99.89% cumulative) 295492.00 KB ( 4.46%) ( 66.88% cumulative) [ 2048- 4096 KB): 55 ( 0.02%) ( 99.91% cumulative) 165772.00 KB ( 2.50%) ( 69.38% cumulative) [ 4096- 8192 KB): 52 ( 0.02%) ( 99.93% cumulative) 331200.00 KB ( 4.99%) ( 74.37% cumulative) [ 8192-16384 KB): 146 ( 0.06%) (100.00% cumulative) 1637404.00 KB (24.69%) ( 99.06% cumulative) [16384-32768 KB): 3 ( 0.00%) (100.00% cumulative) 62076.00 KB ( 0.94%) (100.00% cumulative) filename length (Includes regular files, dirs, symlinks and special files): Range of filename lengths, count of files in range, count of files in range as a % of total number of files, total bytes of filenames in range, total bytes of filenames in range as a % of total number of bytes of filenames, total number of bytes of filenames in this range or smaller as a % of total number bytes of filenames. count=60195285 avg=41.71 chars min=1.00 chars max=220.00 chars [ 0- 7 chars]: 433803 ( 0.72%) ( 0.72% cumulative) 2272792.00 chars ( 0.09%) ( 0.09% cumulative) [ 8- 15 chars]: 1628761 ( 2.71%) ( 3.43% cumulative) 19094598.00 chars ( 0.76%) ( 0.85% cumulative) [ 16- 23 chars]: 2771542 ( 4.60%) ( 8.03% cumulative) 54241521.00 chars ( 2.16%) ( 3.01% cumulative) [ 24- 31 chars]: 3856254 ( 6.41%) ( 14.44% cumulative) 106932976.00 chars ( 4.26%) ( 7.27% cumulative) [ 32- 39 chars]: 14899176 (24.75%) ( 39.19% cumulative) 521744672.00 chars (20.78%) ( 28.05% cumulative) [ 40- 47 chars]: 17344424 (28.81%) ( 68.00% cumulative) 734036771.00 chars (29.23%) ( 57.28% cumulative) [ 48- 55 chars]: 13751667 (22.85%) ( 90.85% cumulative) 704938238.00 chars (28.07%) ( 85.36% cumulative) [ 56- 63 chars]: 2346500 ( 3.90%) ( 94.75% cumulative) 138069584.00 chars ( 5.50%) ( 90.86% cumulative) [ 64- 71 chars]: 1278004 ( 2.12%) ( 96.87% cumulative) 85515974.00 chars ( 3.41%) ( 94.26% cumulative) [ 72- 79 chars]: 1444484 ( 2.40%) ( 99.27% cumulative) 105925289.00 chars ( 4.22%) ( 98.48% cumulative) [ 80- 87 chars]: 324627 ( 0.54%) ( 99.81% cumulative) 26883013.00 chars ( 1.07%) ( 99.55% cumulative) [ 88- 95 chars]: 37223 ( 0.06%) ( 99.87% cumulative) 3329140.00 chars ( 0.13%) ( 99.68% cumulative) [ 96-103 chars]: 59953 ( 0.10%) ( 99.97% cumulative) 5916560.00 chars ( 0.24%) ( 99.92% cumulative) [104-111 chars]: 15596 ( 0.03%) ( 99.99% cumulative) 1654085.00 chars ( 0.07%) ( 99.98% cumulative) [112-119 chars]: 3144 ( 0.01%) (100.00% cumulative) 367669.00 chars ( 0.01%) (100.00% cumulative) [120-127 chars]: 86 ( 0.00%) (100.00% cumulative) 10498.00 chars ( 0.00%) (100.00% cumulative) [ >127 chars]: 41 ( 0.00%) (100.00% cumulative) 6570.00 chars ( 0.00%) (100.00% cumulative) link count(of regular files): Range of link counts, count of files in range, number of files in range as a % of total files, number of files in this range or smaller as a % of total files, total links in range, total links in range as a % of total links, total links in this range or smaller as a % of total links. count=59934337 avg=1.00 links min=1.00 links max=7.00 links [0-1 links]: 59934278 (100.00%) (100.00% cumulative) 59934278.00 links (100.00%) (100.00% cumulative) [2-3 links]: 52 ( 0.00%) (100.00% cumulative) 111.00 links ( 0.00%) (100.00% cumulative) [4-7 links]: 7 ( 0.00%) (100.00% cumulative) 35.00 links ( 0.00%) (100.00% cumulative) symlink target length: Range of symlink file size, count of files in range, number of files in range as a % of total files, number of files in this range or smaller as a % of total files, total size of targets in range, total size of targets in range as a % of total size of targets, total size of targets in this range or smaller as a % of total size of targets. count=32290 avg=84.32 chars min=5.00 chars max=154.00 chars [ 0- 7 chars]: 21 ( 0.07%) ( 0.07% cumulative) 119.00 chars ( 0.00%) ( 0.00% cumulative) [ 8- 15 chars]: 651 ( 2.02%) ( 2.08% cumulative) 8328.00 chars ( 0.31%) ( 0.31% cumulative) [ 16- 23 chars]: 1686 ( 5.22%) ( 7.30% cumulative) 31548.00 chars ( 1.16%) ( 1.47% cumulative) [ 24- 31 chars]: 1229 ( 3.81%) ( 11.11% cumulative) 33849.00 chars ( 1.24%) ( 2.71% cumulative) [ 32- 39 chars]: 256 ( 0.79%) ( 11.90% cumulative) 8633.00 chars ( 0.32%) ( 3.03% cumulative) [ 40- 47 chars]: 191 ( 0.59%) ( 12.49% cumulative) 8240.00 chars ( 0.30%) ( 3.33% cumulative) [ 48- 55 chars]: 538 ( 1.67%) ( 14.16% cumulative) 28122.00 chars ( 1.03%) ( 4.36% cumulative) [ 56- 63 chars]: 1141 ( 3.53%) ( 17.69% cumulative) 68604.00 chars ( 2.52%) ( 6.88% cumulative) [ 64- 71 chars]: 1564 ( 4.84%) ( 22.54% cumulative) 105964.00 chars ( 3.89%) ( 10.78% cumulative) [ 72- 79 chars]: 1451 ( 4.49%) ( 27.03% cumulative) 109262.00 chars ( 4.01%) ( 14.79% cumulative) [ 80- 87 chars]: 1534 ( 4.75%) ( 31.78% cumulative) 126146.00 chars ( 4.63%) ( 19.42% cumulative) [ 88- 95 chars]: 16648 (51.56%) ( 83.34% cumulative) 1483365.00 chars (54.48%) ( 73.90% cumulative) [ 96-103 chars]: 679 ( 2.10%) ( 85.44% cumulative) 67409.00 chars ( 2.48%) ( 76.38% cumulative) [104-111 chars]: 637 ( 1.97%) ( 87.41% cumulative) 67713.00 chars ( 2.49%) ( 78.87% cumulative) [112-119 chars]: 760 ( 2.35%) ( 89.77% cumulative) 87181.00 chars ( 3.20%) ( 82.07% cumulative) [120-127 chars]: 319 ( 0.99%) ( 90.76% cumulative) 39180.00 chars ( 1.44%) ( 83.51% cumulative) [ >127 chars]: 2985 ( 9.24%) (100.00% cumulative) 449057.00 chars (16.49%) (100.00% cumulative) symlink targets: 20.86% relative, 79.14% absolute time since last modification (Reguar files):Range in days, count of files in range, number of files in range as a % of total number of files, number of files in this range or smaller as a % of total number of files, total of age in range, total age in range as a % of total age of all files, total age in this range or smaller as a % of total age of all files. count=59934337 avg=619.75 days min=-1063.77 days max=15087.35 days [ < 0 days]: 56 ( 0.00%) ( 0.00% cumulative) -15888.50 days (-0.00%) ( -0.00% cumulative) [ 0- 2 days): 35412 ( 0.06%) ( 0.06% cumulative) 34078.03 days ( 0.00%) ( 0.00% cumulative) [ 2- 4 days): 29491 ( 0.05%) ( 0.11% cumulative) 77373.04 days ( 0.00%) ( 0.00% cumulative) [ 4- 8 days): 49796 ( 0.08%) ( 0.19% cumulative) 301158.26 days ( 0.00%) ( 0.00% cumulative) [ 8- 16 days): 673987 ( 1.12%) ( 1.32% cumulative) 7950711.07 days ( 0.02%) ( 0.02% cumulative) [ 16- 32 days): 1105851 ( 1.85%) ( 3.16% cumulative) 27403040.47 days ( 0.07%) ( 0.10% cumulative) [ 32- 64 days): 2338818 ( 3.90%) ( 7.06% cumulative) 115616097.34 days ( 0.31%) ( 0.41% cumulative) [ 64- 128 days): 6245411 (10.42%) ( 17.48% cumulative) 562851103.34 days ( 1.52%) ( 1.92% cumulative) [ 128- 256 days): 9189205 (15.33%) ( 32.82% cumulative) 1718705208.96 days ( 4.63%) ( 6.55% cumulative) [ 256- 512 days): 10686750 (17.83%) ( 50.65% cumulative) 3778926028.43 days (10.17%) ( 16.72% cumulative) [ 512- 1024 days): 18277796 (30.50%) ( 81.14% cumulative) 12128383407.20 days (32.65%) ( 49.38% cumulative) [ 1024- 2048 days): 9548349 (15.93%) ( 97.07% cumulative) 14533495124.03 days (39.13%) ( 88.50% cumulative) [ 2048- 4096 days): 1735825 ( 2.90%) ( 99.97% cumulative) 4043797267.04 days (10.89%) ( 99.39% cumulative) [ 4096- 8192 days): 3673 ( 0.01%) ( 99.98% cumulative) 17041460.21 days ( 0.05%) ( 99.43% cumulative) [ 8192-16384 days): 13917 ( 0.02%) (100.00% cumulative) 209928463.08 days ( 0.57%) (100.00% cumulative) time since last modification (size of regular files by age): Range in days, count of bytes whose age is in range, count of bytes in range as a % total number of bytes, count of bytes in this range or smaller as a % of total number of bytes, total days in range (each weighted by its size), today weighted days in range as a % of total weighted days, total days in this range or smaller as a % of total weighted days. count=330764941776 avg=258.43 days min=-1063.77 days max=15087.35 days [ < 0 days]: 76603 ( 0.00%) ( 0.00% cumulative) -115.15 days (-0.00%) ( -0.00% cumulative) [ 0- 2 days): 1106041649 ( 0.33%) ( 0.33% cumulative) 1366476314.63 days ( 0.00%) ( 0.00% cumulative) [ 2- 4 days): 1385813726 ( 0.42%) ( 0.75% cumulative) 3966654932.77 days ( 0.00%) ( 0.01% cumulative) [ 4- 8 days): 2677064724 ( 0.81%) ( 1.56% cumulative) 15363660036.92 days ( 0.02%) ( 0.02% cumulative) [ 8- 16 days): 10772940907 ( 3.26%) ( 4.82% cumulative) 135762404168.48 days ( 0.16%) ( 0.18% cumulative) [ 16- 32 days): 20415381579 ( 6.17%) ( 10.99% cumulative) 467250388017.33 days ( 0.55%) ( 0.73% cumulative) [ 32- 64 days): 23160971600 ( 7.00%) ( 17.99% cumulative) 1121567142378.38 days ( 1.31%) ( 2.04% cumulative) [ 64- 128 days): 83081569774 (25.12%) ( 43.11% cumulative) 8257626650042.67 days ( 9.66%) ( 11.70% cumulative) [ 128- 256 days): 65411975006 (19.78%) ( 62.89% cumulative) 11578341200630.93 days (13.55%) ( 25.25% cumulative) [ 256- 512 days): 63106694110 (19.08%) ( 81.97% cumulative) 22594455048531.88 days (26.43%) ( 51.68% cumulative) [ 512- 1024 days): 58833004426 (17.79%) ( 99.75% cumulative) 38996626749493.20 days (45.62%) ( 97.30% cumulative) [ 1024- 2048 days): 643646022 ( 0.19%) ( 99.95% cumulative) 986397036736.21 days ( 1.15%) ( 98.46% cumulative) [ 2048- 4096 days): 97400319 ( 0.03%) ( 99.98% cumulative) 228338238534.35 days ( 0.27%) ( 98.72% cumulative) [ 4096- 8192 days): 5014 ( 0.00%) ( 99.98% cumulative) 24886886.57 days ( 0.00%) ( 98.72% cumulative) [ 8192-16384 days): 72356309 ( 0.02%) (100.00% cumulative) 1091622329101.52 days ( 1.28%) (100.00% cumulative) time since creation (Reguar files):Range in days, count of files in range, number of files in range as a % of total number of files, number of files in this range or smaller as a % of total number of files, total of age in range, total age in range as a % of total age of all files, total age in this range or smaller as a % of total age of all files. count=59934337 avg=129.31 days min=-0.00 days max=773.44 days [ < 0 days]: 40 ( 0.00%) ( 0.00% cumulative) -0.04 days (-0.00%) ( -0.00% cumulative) [ 0- 2 days): 357834 ( 0.60%) ( 0.60% cumulative) 665384.62 days ( 0.01%) ( 0.01% cumulative) [ 2- 4 days): 809439 ( 1.35%) ( 1.95% cumulative) 1817661.35 days ( 0.02%) ( 0.03% cumulative) [ 4- 8 days): 41020 ( 0.07%) ( 2.02% cumulative) 247938.63 days ( 0.00%) ( 0.04% cumulative) [ 8- 16 days): 560317 ( 0.93%) ( 2.95% cumulative) 6581167.90 days ( 0.08%) ( 0.12% cumulative) [ 16- 32 days): 918100 ( 1.53%) ( 4.48% cumulative) 23949354.71 days ( 0.31%) ( 0.43% cumulative) [ 32- 64 days): 2168158 ( 3.62%) ( 8.10% cumulative) 109801036.52 days ( 1.42%) ( 1.85% cumulative) [ 64- 128 days): 39481597 (65.87%) ( 73.98% cumulative) 2842059951.34 days (36.67%) ( 38.52% cumulative) [ 128- 256 days): 7722024 (12.88%) ( 86.86% cumulative) 1537800781.23 days (19.84%) ( 58.36% cumulative) [ 256- 512 days): 5770445 ( 9.63%) ( 96.49% cumulative) 1991800261.62 days (25.70%) ( 84.06% cumulative) [ 512-1024 days): 2105363 ( 3.51%) (100.00% cumulative) 1235121475.98 days (15.94%) (100.00% cumulative) time since creation (size of regular files by age): Range in days, count of bytes whose age is in range, count of bytes in range as a % total number of bytes, count of bytes in this range or smaller as a % of total number of bytes, total days in range (each weighted by its size), today weighted days in range as a % of total weighted days, total days in this range or smaller as a % of total weighted days. count=330764941776 avg=182.46 days min=-0.00 days max=773.44 days [ < 0 days]: 76603 ( 0.00%) ( 0.00% cumulative) -115.15 days (-0.00%) ( -0.00% cumulative) [ 0- 2 days): 3442662193 ( 1.04%) ( 1.04% cumulative) 4057745143.28 days ( 0.01%) ( 0.01% cumulative) [ 2- 4 days): 5051567075 ( 1.53%) ( 2.57% cumulative) 15346835252.23 days ( 0.03%) ( 0.03% cumulative) [ 4- 8 days): 5102405849 ( 1.54%) ( 4.11% cumulative) 30530827648.93 days ( 0.05%) ( 0.08% cumulative) [ 8- 16 days): 25230584561 ( 7.63%) ( 11.74% cumulative) 299866572253.15 days ( 0.50%) ( 0.58% cumulative) [ 16- 32 days): 19596489529 ( 5.92%) ( 17.66% cumulative) 462425687390.83 days ( 0.77%) ( 1.35% cumulative) [ 32- 64 days): 29725835959 ( 8.99%) ( 26.65% cumulative) 1351978211181.04 days ( 2.24%) ( 3.59% cumulative) [ 64- 128 days): 74853917442 (22.63%) ( 49.28% cumulative) 7400118058516.60 days (12.26%) ( 15.85% cumulative) [ 128- 256 days): 91711062311 (27.73%) ( 77.01% cumulative) 16388147495021.32 days (27.15%) ( 43.00% cumulative) [ 256- 512 days): 46591104178 (14.09%) ( 91.09% cumulative) 17169818592094.60 days (28.45%) ( 71.45% cumulative) [ 512-1024 days): 29459236070 ( 8.91%) (100.00% cumulative) 17228189779033.85 days (28.55%) (100.00% cumulative) time since last access (Reguar files):Range in days, count of files in range, number of files in range as a % of total number of files, number of files in this range or smaller as a % of total number of files, total of age in range, total age in range as a % of total age of all files, total age in this range or smaller as a % of total age of all files. count=59934337 avg=96.99 days min=0.00 days max=892.70 days [ 0- 2 days): 22500822 (37.54%) ( 37.54% cumulative) 20742470.05 days ( 0.36%) ( 0.36% cumulative) [ 2- 4 days): 5070236 ( 8.46%) ( 46.00% cumulative) 11414473.06 days ( 0.20%) ( 0.55% cumulative) [ 4- 8 days): 5858988 ( 9.78%) ( 55.78% cumulative) 40226580.72 days ( 0.69%) ( 1.25% cumulative) [ 8- 16 days): 1829136 ( 3.05%) ( 58.83% cumulative) 19643192.36 days ( 0.34%) ( 1.58% cumulative) [ 16- 32 days): 785070 ( 1.31%) ( 60.14% cumulative) 19712871.39 days ( 0.34%) ( 1.92% cumulative) [ 32- 64 days): 2789384 ( 4.65%) ( 64.79% cumulative) 128951841.13 days ( 2.22%) ( 4.14% cumulative) [ 64- 128 days): 5747891 ( 9.59%) ( 74.38% cumulative) 499724084.62 days ( 8.60%) ( 12.74% cumulative) [ 128- 256 days): 5243228 ( 8.75%) ( 83.13% cumulative) 1000102852.63 days (17.20%) ( 29.94% cumulative) [ 256- 512 days): 7652623 (12.77%) ( 95.90% cumulative) 2581858985.15 days (44.42%) ( 74.36% cumulative) [ 512-1024 days): 2456959 ( 4.10%) (100.00% cumulative) 1490594239.30 days (25.64%) (100.00% cumulative) time since access (size of regular files by age): Range in days, count of bytes whose age is in range, count of bytes in range as a % total number of bytes, count of bytes in this range or smaller as a % of total number of bytes, total days in range (each weighted by its size), today weighted days in range as a % of total weighted days, total days in this range or smaller as a % of total weighted days. count=330764941776 avg=177.00 days min=0.00 days max=892.70 days [ 0- 2 days): 5827271517 ( 1.76%) ( 1.76% cumulative) 5872231564.03 days ( 0.01%) ( 0.01% cumulative) [ 2- 4 days): 6266426014 ( 1.89%) ( 3.66% cumulative) 17948937149.60 days ( 0.03%) ( 0.04% cumulative) [ 4- 8 days): 8352467112 ( 2.53%) ( 6.18% cumulative) 52384514577.01 days ( 0.09%) ( 0.13% cumulative) [ 8- 16 days): 28923846298 ( 8.74%) ( 14.93% cumulative) 347359021276.68 days ( 0.59%) ( 0.72% cumulative) [ 16- 32 days): 26633629587 ( 8.05%) ( 22.98% cumulative) 650880664462.99 days ( 1.11%) ( 1.84% cumulative) [ 32- 64 days): 37262787520 (11.27%) ( 34.24% cumulative) 1752906299042.39 days ( 2.99%) ( 4.83% cumulative) [ 64- 128 days): 67195085361 (20.32%) ( 54.56% cumulative) 6635947781217.27 days (11.33%) ( 16.16% cumulative) [ 128- 256 days): 70775892016 (21.40%) ( 75.96% cumulative) 12534779725391.32 days (21.41%) ( 37.57% cumulative) [ 256- 512 days): 42183779470 (12.75%) ( 88.71% cumulative) 14510417023055.21 days (24.78%) ( 62.36% cumulative) [ 512-1024 days): 37343756875 (11.29%) (100.00% cumulative) 22037978254075.39 days (37.64%) (100.00% cumulative) ----- END NORMAL ----- From gmpc at sanger.ac.uk Tue Apr 26 08:13:48 2011 From: gmpc at sanger.ac.uk (Guy Coates) Date: Tue, 26 Apr 2011 09:13:48 +0100 Subject: [Lustre-devel] Fwd: Re: [Lustre-discuss] Research on filesystem metadata operation distribution Message-ID: <4DB67EBC.60900@sanger.ac.uk> Hi, Please find the stats attached. We happen to log our MDS operations, so I have included graphs for the past week & month, showing average and peak operations. Both filesystems are multi-user scratch filesystems, running a wide mix of embarrassingly parallel bio-informatics code. Cheers, Guy -- Dr. Guy Coates, Informatics Systems Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK Tel: +44 (0)1223 834244 x 6925 Fax: +44 (0)1223 496802 -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. -------------- next part -------------- A non-text attachment was scrubbed... Name: stats.tar.gz Type: application/x-gzip Size: 172356 bytes Desc: not available URL: From Mark.Hills at framestore.com Tue Apr 26 16:06:05 2011 From: Mark.Hills at framestore.com (Mark Hills) Date: Tue, 26 Apr 2011 17:06:05 +0100 (BST) Subject: [Lustre-devel] [PATCH] Fix task IO accounting on reads Message-ID: We've often found it inconvenient that reads from Lustre mounts do not correctly increment IO counts (/proc//io). Below is a patch which aims to fix this functionality. The symptom is that writes are accounted for, but not reads. It seems that the accounting it normally done in the kernels page writeback and readahead functionality. Therefore as Lustre implements its own readahead, it must also maintain its own accounting on reads (but not writes). Is anyone able to confirm this rationale for the location of this particular call, and consider the following patch? Thanks. -- Mark --- lustre/llite/rw.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/lustre/llite/rw.c b/lustre/llite/rw.c index ccd6a42..c7a84b7 100644 --- a/lustre/llite/rw.c +++ b/lustre/llite/rw.c @@ -58,6 +58,7 @@ #include #include #include +#include #define DEBUG_SUBSYSTEM S_LLITE @@ -1311,6 +1312,8 @@ static int ll_issue_page_read(struct obd_export *exp, if (rc) { LL_CDEBUG_PAGE(D_ERROR, page, "read queue failed: rc %d\n", rc); page_cache_release(page); + } else { + task_io_account_read(CFS_PAGE_SIZE); } RETURN(rc); } -- 1.7.1.1 From adilger at whamcloud.com Tue Apr 26 16:38:14 2011 From: adilger at whamcloud.com (Andreas Dilger) Date: Tue, 26 Apr 2011 10:38:14 -0600 Subject: [Lustre-devel] [PATCH] Fix task IO accounting on reads In-Reply-To: References: Message-ID: <5870C1C1-3775-4CC8-B619-C79B6799D6E5@whamcloud.com> On 2011-04-26, at 10:06 AM, Mark Hills wrote: > We've often found it inconvenient that reads from Lustre mounts do not > correctly increment IO counts (/proc//io). > > Below is a patch which aims to fix this functionality. > > The symptom is that writes are accounted for, but not reads. It seems that > the accounting it normally done in the kernels page writeback and > readahead functionality. Therefore as Lustre implements its own readahead, > it must also maintain its own accounting on reads (but not writes). > > Is anyone able to confirm this rationale for the location of this > particular call, and consider the following patch? Thanks. Mark, It looks like this functionality was added to the kernel in Git commit hash 7c3ab7381e79dfc7db14a67c6f4f3285664e1ec2, which was between 2.6.19 and 2.6.20. This call needs to be made conditional so that it doesn't break the client build on older kernels (it looks like it would break RHEL5 and SLES10 clients, which don't have this functionality AFAICS). It looks like it may be possible to create a compatibility macro in lustre/include/linux/lustre_compat25.h so that it doesn't break for clients running on older kernels: #ifdef CONFIG_TASK_IO_ACCOUNTING #include #else #define task_io_accounting_read(bytes) do {} while (0) #endif > lustre/llite/rw.c | 3 +++ > 1 files changed, 3 insertions(+), 0 deletions(-) > > diff --git a/lustre/llite/rw.c b/lustre/llite/rw.c > index ccd6a42..c7a84b7 100644 > --- a/lustre/llite/rw.c > +++ b/lustre/llite/rw.c > @@ -58,6 +58,7 @@ > #include > #include > #include > +#include > > #define DEBUG_SUBSYSTEM S_LLITE > > @@ -1311,6 +1312,8 @@ static int ll_issue_page_read(struct obd_export *exp, > if (rc) { > LL_CDEBUG_PAGE(D_ERROR, page, "read queue failed: rc %d\n", rc); > page_cache_release(page); > + } else { > + task_io_account_read(CFS_PAGE_SIZE); > } > RETURN(rc); > } > -- > 1.7.1.1 > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. From kit.westneat at nyu.edu Tue Apr 26 17:49:24 2011 From: kit.westneat at nyu.edu (Kit Westneat) Date: Tue, 26 Apr 2011 13:49:24 -0400 Subject: [Lustre-devel] MDS stats at NYU Message-ID: <30B98843-241A-4FBB-A813-6CE084D1C3CF@nyu.edu> These are the stats for NYU's scratch filesystem. [root at mds0 ~]# lctl get_param mds.*.stats | egrep "open|close|rename|link|attr|sync" open 9447471556 samples [reqs] close 4096767574 samples [reqs] link 1021879 samples [reqs] unlink 124252665 samples [reqs] rename 6081440 samples [reqs] getxattr 2728382 samples [reqs] setxattr 11866 samples [reqs] setattr 463972324 samples [reqs] getattr 14395503114 samples [reqs] [root at mds0 ~]# uptime 12:58:40 up 143 days, 22:18, 1 user, load average: 0.48, 0.56, 0.66 [westneat at login-0-1 ~]$ lfs df UUID 1K-blocks Used Available Use% Mounted on scratch-MDT0000_UUID 366974128 3381180 342621428 0% /scratch[MDT:0] scratch-OST0000_UUID 7691221300 6119095796 1181430780 79% /scratch[OST:0] scratch-OST0001_UUID 7691221300 6158628776 1141900888 80% /scratch[OST:1] scratch-OST0002_UUID 7691221300 6284721592 1015803352 81% /scratch[OST:2] scratch-OST0003_UUID 7691221300 6140176888 1160347996 79% /scratch[OST:3] scratch-OST0004_UUID 7691221300 6168458884 1132066684 80% /scratch[OST:4] scratch-OST0005_UUID 7691221300 6127417412 1173104884 79% /scratch[OST:5] scratch-OST0006_UUID 7691221300 6255178448 1045348128 81% /scratch[OST:6] scratch-OST0007_UUID 7691221300 6334097184 966402364 82% /scratch[OST:7] scratch-OST0008_UUID 7691221300 6394367032 906162596 83% /scratch[OST:8] scratch-OST0009_UUID 7691221300 6212551752 1087977856 80% /scratch[OST:9] scratch-OST000a_UUID 7691221300 6145601928 1154907900 79% /scratch[OST:10] scratch-OST000b_UUID 7691221300 6241638640 1058856436 81% /scratch[OST:11] scratch-OST000c_UUID 7691221300 6173697080 1126808920 80% /scratch[OST:12] scratch-OST000d_UUID 7691221300 670477364 6942604588 8% /scratch[OST:13] filesystem summary: 107677098200 81426108776 21093723372 75% /scratch [westneat at login-0-1 ~]$ lfs df -i UUID Inodes IUsed IFree IUse% Mounted on scratch-MDT0000_UUID 104857600 18631841 86225759 17% /scratch[MDT:0] scratch-OST0000_UUID 397882034 4853346 393028688 1% /scratch[OST:0] scratch-OST0001_UUID 388003577 4857918 383145659 1% /scratch[OST:1] scratch-OST0002_UUID 356214257 4590433 351623824 1% /scratch[OST:2] scratch-OST0003_UUID 392647284 4887652 387759632 1% /scratch[OST:3] scratch-OST0004_UUID 385627716 4939587 380688129 1% /scratch[OST:4] scratch-OST0005_UUID 395955612 5006122 390949490 1% /scratch[OST:5] scratch-OST0006_UUID 363765711 4766685 358999026 1% /scratch[OST:6] scratch-OST0007_UUID 343837518 4558701 339278817 1% /scratch[OST:7] scratch-OST0008_UUID 328746176 4532738 324213438 1% /scratch[OST:8] scratch-OST0009_UUID 374504226 4838488 369665738 1% /scratch[OST:9] scratch-OST000a_UUID 391271939 4868488 386403451 1% /scratch[OST:10] scratch-OST000b_UUID 367140930 4746770 362394160 1% /scratch[OST:11] scratch-OST000c_UUID 384194773 4815470 379379303 1% /scratch[OST:12] scratch-OST000d_UUID 488366080 176786 488189294 0% /scratch[OST:13] filesystem summary: 104857600 18631841 86225759 17% /scratch - Kit From Mark.Hills at framestore.com Wed Apr 27 11:23:57 2011 From: Mark.Hills at framestore.com (Mark Hills) Date: Wed, 27 Apr 2011 12:23:57 +0100 (BST) Subject: [Lustre-devel] [PATCH] Fix task IO accounting on reads In-Reply-To: <5870C1C1-3775-4CC8-B619-C79B6799D6E5@whamcloud.com> References: <5870C1C1-3775-4CC8-B619-C79B6799D6E5@whamcloud.com> Message-ID: On Tue, 26 Apr 2011, Andreas Dilger wrote: > On 2011-04-26, at 10:06 AM, Mark Hills wrote: > > We've often found it inconvenient that reads from Lustre mounts do not > > correctly increment IO counts (/proc//io). > > > > Below is a patch which aims to fix this functionality. > > > > The symptom is that writes are accounted for, but not reads. It seems that > > the accounting it normally done in the kernels page writeback and > > readahead functionality. Therefore as Lustre implements its own readahead, > > it must also maintain its own accounting on reads (but not writes). > > > > Is anyone able to confirm this rationale for the location of this > > particular call, and consider the following patch? Thanks. > > Mark, > It looks like this functionality was added to the kernel in Git commit > hash 7c3ab7381e79dfc7db14a67c6f4f3285664e1ec2, which was between 2.6.19 > and 2.6.20. This call needs to be made conditional so that it doesn't > break the client build on older kernels (it looks like it would break > RHEL5 and SLES10 clients, which don't have this functionality AFAICS). > > It looks like it may be possible to create a compatibility macro in > lustre/include/linux/lustre_compat25.h so that it doesn't break for > clients running on older kernels: > > #ifdef CONFIG_TASK_IO_ACCOUNTING > #include > #else > #define task_io_accounting_read(bytes) do {} while (0) > #endif Newer kernels always define task_io_account_read() as a function, even if it's a no-op. So to avoid redefining it it should check the kernel version, not the CONFIG_ flag. A revised patch is below, which I tested with kernel 2.6.32.28. Unfortunately I odn't have a pre-2.6.20 platform for testing. Thanks Andreas, -- Mark --- lustre/include/linux/lustre_compat25.h | 6 ++++++ lustre/llite/rw.c | 2 ++ 2 files changed, 8 insertions(+), 0 deletions(-) diff --git a/lustre/include/linux/lustre_compat25.h b/lustre/include/linux/lustre_compat25.h index 81865d0..666374b 100644 --- a/lustre/include/linux/lustre_compat25.h +++ b/lustre/include/linux/lustre_compat25.h @@ -143,6 +143,12 @@ void groups_free(struct group_info *ginfo); #endif /* LINUX_VERSION_CODE < KERNEL_VERSION(2,6,4) */ +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,20) +#define task_io_account_read(bytes) do {} while (0) +#else +#include +#endif + #ifndef page_private #define page_private(page) ((page)->private) #define set_page_private(page, v) ((page)->private = (v)) diff --git a/lustre/llite/rw.c b/lustre/llite/rw.c index ccd6a42..1fddd5b 100644 --- a/lustre/llite/rw.c +++ b/lustre/llite/rw.c @@ -1311,6 +1311,8 @@ static int ll_issue_page_read(struct obd_export *exp, if (rc) { LL_CDEBUG_PAGE(D_ERROR, page, "read queue failed: rc %d\n", rc); page_cache_release(page); + } else { + task_io_account_read(CFS_PAGE_SIZE); } RETURN(rc); } -- 1.7.1.1 From cap at nsc.liu.se Wed Apr 27 15:56:19 2011 From: cap at nsc.liu.se (Peter =?iso-8859-1?q?Kjellstr=F6m?=) Date: Wed, 27 Apr 2011 17:56:19 +0200 Subject: [Lustre-devel] [PATCH] Fix task IO accounting on reads In-Reply-To: References: <5870C1C1-3775-4CC8-B619-C79B6799D6E5@whamcloud.com> Message-ID: <201104271756.24080.cap@nsc.liu.se> On Wednesday, April 27, 2011 01:23:57 PM Mark Hills wrote: > On Tue, 26 Apr 2011, Andreas Dilger wrote: ... > > Mark, > > It looks like this functionality was added to the kernel in Git commit > > hash 7c3ab7381e79dfc7db14a67c6f4f3285664e1ec2, which was between 2.6.19 > > and 2.6.20. This call needs to be made conditional so that it doesn't > > break the client build on older kernels (it looks like it would break > > RHEL5 and SLES10 clients, which don't have this functionality AFAICS). > > > > It looks like it may be possible to create a compatibility macro in > > lustre/include/linux/lustre_compat25.h so that it doesn't break for > > clients running on older kernels: > > > > #ifdef CONFIG_TASK_IO_ACCOUNTING > > #include > > #else > > #define task_io_accounting_read(bytes) do {} while (0) > > #endif > > Newer kernels always define task_io_account_read() as a function, even if > it's a no-op. So to avoid redefining it it should check the kernel > version, not the CONFIG_ flag. > > A revised patch is below, which I tested with kernel 2.6.32.28. > Unfortunately I odn't have a pre-2.6.20 platform for testing. I suspect a real world complication that you'd have to handle is that the RHEL-5u6 kernel (2.6.18-238) has task_io_accounting_ops.h/task_io_accounting_read (yet another Redhat backport...) /Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From adilger at whamcloud.com Wed Apr 27 16:43:45 2011 From: adilger at whamcloud.com (Andreas Dilger) Date: Wed, 27 Apr 2011 10:43:45 -0600 Subject: [Lustre-devel] [PATCH] Fix task IO accounting on reads In-Reply-To: References: <5870C1C1-3775-4CC8-B619-C79B6799D6E5@whamcloud.com> Message-ID: <3059315E-A6DA-4AC1-BEB9-B3AF7E48AC36@whamcloud.com> On 2011-04-27, at 5:23 AM, Mark Hills wrote: > On Tue, 26 Apr 2011, Andreas Dilger wrote: >> On 2011-04-26, at 10:06 AM, Mark Hills wrote: >>> We've often found it inconvenient that reads from Lustre mounts do not >>> correctly increment IO counts (/proc//io). >>> >>> Below is a patch which aims to fix this functionality. >>> >>> The symptom is that writes are accounted for, but not reads. It seems that >>> the accounting it normally done in the kernels page writeback and >>> readahead functionality. Therefore as Lustre implements its own readahead, >>> it must also maintain its own accounting on reads (but not writes). >>> >>> Is anyone able to confirm this rationale for the location of this >>> particular call, and consider the following patch? Thanks. >> >> Mark, >> It looks like this functionality was added to the kernel in Git commit >> hash 7c3ab7381e79dfc7db14a67c6f4f3285664e1ec2, which was between 2.6.19 >> and 2.6.20. This call needs to be made conditional so that it doesn't >> break the client build on older kernels (it looks like it would break >> RHEL5 and SLES10 clients, which don't have this functionality AFAICS). >> >> It looks like it may be possible to create a compatibility macro in >> lustre/include/linux/lustre_compat25.h so that it doesn't break for >> clients running on older kernels: >> >> #ifdef CONFIG_TASK_IO_ACCOUNTING >> #include >> #else >> #define task_io_accounting_read(bytes) do {} while (0) >> #endif > > Newer kernels always define task_io_account_read() as a function, even if > it's a no-op. So to avoid redefining it it should check the kernel > version, not the CONFIG_ flag. I don't think this is a real problem, because if CONFIG_TASK_IO_ACCOUNTING is not defined, then the compatibility functions in the task_io_accounting_ops.h file will not be included and there shouldn't be any problem. I don't see task_io_accounting_ops.h included by any other headers in the kernel, only by fs/buffer.c and fs/direct-io.c, so my original proposal should be safe. > A revised patch is below, which I tested with kernel 2.6.32.28. > Unfortunately I odn't have a pre-2.6.20 platform for testing. > > Thanks Andreas, > > -- > Mark > > --- > lustre/include/linux/lustre_compat25.h | 6 ++++++ > lustre/llite/rw.c | 2 ++ > 2 files changed, 8 insertions(+), 0 deletions(-) > > diff --git a/lustre/include/linux/lustre_compat25.h b/lustre/include/linux/lustre_compat25.h > index 81865d0..666374b 100644 > --- a/lustre/include/linux/lustre_compat25.h > +++ b/lustre/include/linux/lustre_compat25.h > @@ -143,6 +143,12 @@ void groups_free(struct group_info *ginfo); > > #endif /* LINUX_VERSION_CODE < KERNEL_VERSION(2,6,4) */ > > +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,20) > +#define task_io_account_read(bytes) do {} while (0) > +#else > +#include > +#endif > + > #ifndef page_private > #define page_private(page) ((page)->private) > #define set_page_private(page, v) ((page)->private = (v)) > diff --git a/lustre/llite/rw.c b/lustre/llite/rw.c > index ccd6a42..1fddd5b 100644 > --- a/lustre/llite/rw.c > +++ b/lustre/llite/rw.c > @@ -1311,6 +1311,8 @@ static int ll_issue_page_read(struct obd_export *exp, > if (rc) { > LL_CDEBUG_PAGE(D_ERROR, page, "read queue failed: rc %d\n", rc); > page_cache_release(page); > + } else { > + task_io_account_read(CFS_PAGE_SIZE); > } > RETURN(rc); > } > -- > 1.7.1.1 > Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. From Mark.Hills at framestore.com Wed Apr 27 17:14:20 2011 From: Mark.Hills at framestore.com (Mark Hills) Date: Wed, 27 Apr 2011 18:14:20 +0100 (BST) Subject: [Lustre-devel] [PATCH] Fix task IO accounting on reads In-Reply-To: <3059315E-A6DA-4AC1-BEB9-B3AF7E48AC36@whamcloud.com> References: <5870C1C1-3775-4CC8-B619-C79B6799D6E5@whamcloud.com> <3059315E-A6DA-4AC1-BEB9-B3AF7E48AC36@whamcloud.com> Message-ID: On Wed, 27 Apr 2011, Andreas Dilger wrote: > On 2011-04-27, at 5:23 AM, Mark Hills wrote: > > On Tue, 26 Apr 2011, Andreas Dilger wrote: > >> On 2011-04-26, at 10:06 AM, Mark Hills wrote: > >>> We've often found it inconvenient that reads from Lustre mounts do not > >>> correctly increment IO counts (/proc//io). > >>> > >>> Below is a patch which aims to fix this functionality. > >>> > >>> The symptom is that writes are accounted for, but not reads. It seems that > >>> the accounting it normally done in the kernels page writeback and > >>> readahead functionality. Therefore as Lustre implements its own readahead, > >>> it must also maintain its own accounting on reads (but not writes). > >>> > >>> Is anyone able to confirm this rationale for the location of this > >>> particular call, and consider the following patch? Thanks. > >> > >> Mark, > >> It looks like this functionality was added to the kernel in Git commit > >> hash 7c3ab7381e79dfc7db14a67c6f4f3285664e1ec2, which was between 2.6.19 > >> and 2.6.20. This call needs to be made conditional so that it doesn't > >> break the client build on older kernels (it looks like it would break > >> RHEL5 and SLES10 clients, which don't have this functionality AFAICS). > >> > >> It looks like it may be possible to create a compatibility macro in > >> lustre/include/linux/lustre_compat25.h so that it doesn't break for > >> clients running on older kernels: > >> > >> #ifdef CONFIG_TASK_IO_ACCOUNTING > >> #include > >> #else > >> #define task_io_accounting_read(bytes) do {} while (0) > >> #endif > > > > Newer kernels always define task_io_account_read() as a function, even if > > it's a no-op. So to avoid redefining it it should check the kernel > > version, not the CONFIG_ flag. > > I don't think this is a real problem, because if > CONFIG_TASK_IO_ACCOUNTING is not defined, then the compatibility > functions in the task_io_accounting_ops.h file will not be included and > there shouldn't be any problem. I don't see task_io_accounting_ops.h > included by any other headers in the kernel, only by fs/buffer.c and > fs/direct-io.c, so my original proposal should be safe. Ah yes, of course you are correct. If you're happy the _ops.h header won't get included (indirectly or otherwise) in future. I guess I don't need to re-send with that modification, but let me know if you'd prefer that. Thanks for your help. [...] > > --- > > lustre/include/linux/lustre_compat25.h | 6 ++++++ > > lustre/llite/rw.c | 2 ++ > > 2 files changed, 8 insertions(+), 0 deletions(-) > > > > diff --git a/lustre/include/linux/lustre_compat25.h b/lustre/include/linux/lustre_compat25.h > > index 81865d0..666374b 100644 > > --- a/lustre/include/linux/lustre_compat25.h > > +++ b/lustre/include/linux/lustre_compat25.h > > @@ -143,6 +143,12 @@ void groups_free(struct group_info *ginfo); > > > > #endif /* LINUX_VERSION_CODE < KERNEL_VERSION(2,6,4) */ > > > > +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,20) > > +#define task_io_account_read(bytes) do {} while (0) > > +#else > > +#include > > +#endif > > + > > #ifndef page_private > > #define page_private(page) ((page)->private) > > #define set_page_private(page, v) ((page)->private = (v)) > > diff --git a/lustre/llite/rw.c b/lustre/llite/rw.c > > index ccd6a42..1fddd5b 100644 > > --- a/lustre/llite/rw.c > > +++ b/lustre/llite/rw.c > > @@ -1311,6 +1311,8 @@ static int ll_issue_page_read(struct obd_export *exp, > > if (rc) { > > LL_CDEBUG_PAGE(D_ERROR, page, "read queue failed: rc %d\n", rc); > > page_cache_release(page); > > + } else { > > + task_io_account_read(CFS_PAGE_SIZE); > > } > > RETURN(rc); > > } > > -- > > 1.7.1.1 > > > -- Mark From adilger at whamcloud.com Wed Apr 27 21:15:23 2011 From: adilger at whamcloud.com (Andreas Dilger) Date: Wed, 27 Apr 2011 15:15:23 -0600 Subject: [Lustre-devel] [PATCH] Fix task IO accounting on reads In-Reply-To: References: <5870C1C1-3775-4CC8-B619-C79B6799D6E5@whamcloud.com> <3059315E-A6DA-4AC1-BEB9-B3AF7E48AC36@whamcloud.com> Message-ID: <001F682B-CF29-4835-8A0B-5BBFA4021715@whamcloud.com> On 2011-04-27, at 11:14 AM, Mark Hills wrote: > On Wed, 27 Apr 2011, Andreas Dilger wrote: >> On 2011-04-27, at 5:23 AM, Mark Hills wrote: >>> On Tue, 26 Apr 2011, Andreas Dilger wrote: >>>> It looks like this functionality was added to the kernel in Git commit >>>> hash 7c3ab7381e79dfc7db14a67c6f4f3285664e1ec2, which was between 2.6.19 >>>> and 2.6.20. This call needs to be made conditional so that it doesn't >>>> break the client build on older kernels (it looks like it would break >>>> RHEL5 and SLES10 clients, which don't have this functionality AFAICS). >>>> >>>> It looks like it may be possible to create a compatibility macro in >>>> lustre/include/linux/lustre_compat25.h so that it doesn't break for >>>> clients running on older kernels: >>>> >>>> #ifdef CONFIG_TASK_IO_ACCOUNTING >>>> #include >>>> #else >>>> #define task_io_accounting_read(bytes) do {} while (0) >>>> #endif >>> >>> Newer kernels always define task_io_account_read() as a function, even if >>> it's a no-op. So to avoid redefining it it should check the kernel >>> version, not the CONFIG_ flag. >> >> I don't think this is a real problem, because if >> CONFIG_TASK_IO_ACCOUNTING is not defined, then the compatibility >> functions in the task_io_accounting_ops.h file will not be included and >> there shouldn't be any problem. I don't see task_io_accounting_ops.h >> included by any other headers in the kernel, only by fs/buffer.c and >> fs/direct-io.c, so my original proposal should be safe. > > Ah yes, of course you are correct. > > If you're happy the _ops.h header won't get included (indirectly or > otherwise) in future. > > I guess I don't need to re-send with that modification, but let me know if > you'd prefer that. > > Thanks for your help. Mark, for inclusion into the next 1.8 release the updated patch should be attached to a new bug at bugzilla.lustre.org, and a review requested on the patch (I can be one of the reviewers). For inclusion into the 2.1 release, please create a new bug at jira.whamcloud.com with the updated patch. The patch should include the standard kernel "Signed-off-by: {your name} " line. Sorry for the complexity in submitting patches while development transitions away from Oracle for newer releases. > [...] >>> --- >>> lustre/include/linux/lustre_compat25.h | 6 ++++++ >>> lustre/llite/rw.c | 2 ++ >>> 2 files changed, 8 insertions(+), 0 deletions(-) >>> >>> diff --git a/lustre/include/linux/lustre_compat25.h b/lustre/include/linux/lustre_compat25.h >>> index 81865d0..666374b 100644 >>> --- a/lustre/include/linux/lustre_compat25.h >>> +++ b/lustre/include/linux/lustre_compat25.h >>> @@ -143,6 +143,12 @@ void groups_free(struct group_info *ginfo); >>> >>> #endif /* LINUX_VERSION_CODE < KERNEL_VERSION(2,6,4) */ >>> >>> +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,20) >>> +#define task_io_account_read(bytes) do {} while (0) >>> +#else >>> +#include >>> +#endif >>> + >>> #ifndef page_private >>> #define page_private(page) ((page)->private) >>> #define set_page_private(page, v) ((page)->private = (v)) >>> diff --git a/lustre/llite/rw.c b/lustre/llite/rw.c >>> index ccd6a42..1fddd5b 100644 >>> --- a/lustre/llite/rw.c >>> +++ b/lustre/llite/rw.c >>> @@ -1311,6 +1311,8 @@ static int ll_issue_page_read(struct obd_export *exp, >>> if (rc) { >>> LL_CDEBUG_PAGE(D_ERROR, page, "read queue failed: rc %d\n", rc); >>> page_cache_release(page); >>> + } else { >>> + task_io_account_read(CFS_PAGE_SIZE); >>> } >>> RETURN(rc); >>> } >>> -- >>> 1.7.1.1 >>> >> > > -- > Mark Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. From pjones at whamcloud.com Wed Apr 27 21:19:51 2011 From: pjones at whamcloud.com (Peter Jones) Date: Wed, 27 Apr 2011 14:19:51 -0700 Subject: [Lustre-devel] [PATCH] Fix task IO accounting on reads In-Reply-To: <001F682B-CF29-4835-8A0B-5BBFA4021715@whamcloud.com> References: <5870C1C1-3775-4CC8-B619-C79B6799D6E5@whamcloud.com> <3059315E-A6DA-4AC1-BEB9-B3AF7E48AC36@whamcloud.com> <001F682B-CF29-4835-8A0B-5BBFA4021715@whamcloud.com> Message-ID: <4DB88877.6000000@whamcloud.com> Oracle will also require that a contributor agreement is submitted - see http://wiki.lustre.org/index.php/Contribution_Policy On 11-04-27 2:15 PM, Andreas Dilger wrote: > > for inclusion into the next 1.8 release the updated patch should be attached to a new bug at bugzilla.lustre.org, and a review requested on the patch (I can be one of the reviewers). From eeb at whamcloud.com Thu Apr 28 23:04:37 2011 From: eeb at whamcloud.com (Eric Barton) Date: Fri, 29 Apr 2011 00:04:37 +0100 Subject: [Lustre-devel] F/S stats visualisation Message-ID: <008a01cc05f8$a6b26840$f41738c0$@com> Hi, I've made some histograms that visualise the F/S stats people have so kindly sent in. There are a couple of surprises which seem to do with how the filesystems were set up (e.g. huge #s of inodes reserved but not used on OSTs) and used (different ratios of ops). The attached PDF has a summary for each filesystem containing... * Filesystem name * Ratio of MDS 1K blocks to MDS inodes Ratio of OST 1K blocks to MDS inodes Ratio of OST inodes to MDS inodes * Histogram of operation counts * Histogram of blocks total (blue) and used (red) on the MDS and all OSTs * Histogram of inodes total (blue) and used (red) on the MDS and all OSTs ...hope you find it useful. Cheers, Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: Lustre stats.pdf Type: application/pdf Size: 619235 bytes Desc: not available URL: From Nathan_Rutman at xyratex.com Fri Apr 29 17:35:22 2011 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Fri, 29 Apr 2011 10:35:22 -0700 Subject: [Lustre-devel] F/S stats visualisation In-Reply-To: <008a01cc05f8$a6b26840$f41738c0$@com> References: <008a01cc05f8$a6b26840$f41738c0$@com> Message-ID: <73AED5C780AE05478241DB067651A921023784B9@XYUS-EX22.xyus.xyratex.com> That's interesting - why do you think opens >> closes? Are we not tracking closes correctly? On Apr 28, 2011, at 4:04 PM, Eric Barton wrote: > Hi, > > I've made some histograms that visualise the F/S stats people have > so kindly sent in. There are a couple of surprises which seem to > do with how the filesystems were set up (e.g. huge #s of inodes > reserved but not used on OSTs) and used (different ratios of ops). > > The attached PDF has a summary for each filesystem containing... > > * Filesystem name > > * Ratio of MDS 1K blocks to MDS inodes > Ratio of OST 1K blocks to MDS inodes > Ratio of OST inodes to MDS inodes > > * Histogram of operation counts > > * Histogram of blocks total (blue) and used (red) on the MDS > and all OSTs > > * Histogram of inodes total (blue) and used (red) on the MDS > and all OSTs > > ...hope you find it useful. > > Cheers, > Eric > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________ From adilger at whamcloud.com Fri Apr 29 17:41:05 2011 From: adilger at whamcloud.com (Andreas Dilger) Date: Fri, 29 Apr 2011 11:41:05 -0600 Subject: [Lustre-devel] F/S stats visualisation In-Reply-To: <73AED5C780AE05478241DB067651A921023784B9@XYUS-EX22.xyus.xyratex.com> References: <008a01cc05f8$a6b26840$f41738c0$@com> <73AED5C780AE05478241DB067651A921023784B9@XYUS-EX22.xyus.xyratex.com> Message-ID: <662791CA-879E-4DB1-9C94-B93D7FA85384@whamcloud.com> On Apr 29, 2011, at 11:35, Nathan Rutman wrote: > That's interesting - why do you think opens >> closes? > Are we not tracking closes correctly? I was wondering the same, and/or we are double-counting opens? Clients won't send close RPCs if they are evicted (MDS does the close locally on eviction), but I'd hope that clients aren't evicted _that_ often. > On Apr 28, 2011, at 4:04 PM, Eric Barton wrote: > >> Hi, >> >> I've made some histograms that visualise the F/S stats people have >> so kindly sent in. There are a couple of surprises which seem to >> do with how the filesystems were set up (e.g. huge #s of inodes >> reserved but not used on OSTs) and used (different ratios of ops). >> >> The attached PDF has a summary for each filesystem containing... >> >> * Filesystem name >> >> * Ratio of MDS 1K blocks to MDS inodes >> Ratio of OST 1K blocks to MDS inodes >> Ratio of OST inodes to MDS inodes >> >> * Histogram of operation counts >> >> * Histogram of blocks total (blue) and used (red) on the MDS >> and all OSTs >> >> * Histogram of inodes total (blue) and used (red) on the MDS >> and all OSTs >> >> ...hope you find it useful. >> >> Cheers, >> Eric >> >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel > ______________________________________________________________________ > This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. > > Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. > > Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. > > The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. > ______________________________________________________________________ > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. From johann at whamcloud.com Fri Apr 29 18:01:04 2011 From: johann at whamcloud.com (Johann Lombardi) Date: Fri, 29 Apr 2011 20:01:04 +0200 Subject: [Lustre-devel] F/S stats visualisation In-Reply-To: <662791CA-879E-4DB1-9C94-B93D7FA85384@whamcloud.com> References: <008a01cc05f8$a6b26840$f41738c0$@com> <73AED5C780AE05478241DB067651A921023784B9@XYUS-EX22.xyus.xyratex.com> <662791CA-879E-4DB1-9C94-B93D7FA85384@whamcloud.com> Message-ID: <20110429180104.GA5771@neo> On Fri, Apr 29, 2011 at 11:41:05AM -0600, Andreas Dilger wrote: > On Apr 29, 2011, at 11:35, Nathan Rutman wrote: > > That's interesting - why do you think opens >> closes? > > Are we not tracking closes correctly? > > I was wondering the same, and/or we are double-counting opens? Clients won't send close RPCs if they are evicted (MDS does the close locally on eviction), but I'd hope that clients aren't evicted _that_ often. It seems that we increment the open stat counter on resent (for reply reconstruction), but not for close. Maybe that's the reason? It might also just be that open failed so no close is needed. Johann From apittman at ddn.com Fri Apr 29 07:06:06 2011 From: apittman at ddn.com (Ashley Pittman) Date: Fri, 29 Apr 2011 00:06:06 -0700 Subject: [Lustre-devel] F/S stats visualisation In-Reply-To: <008a01cc05f8$a6b26840$f41738c0$@com> References: <008a01cc05f8$a6b26840$f41738c0$@com> Message-ID: <84D90924-025F-4ABA-89BD-58C43C8FCDD9@ddn.com> On 29 Apr 2011, at 00:04, Eric Barton wrote: > > * Histogram of inodes total (blue) and used (red) on the MDS > and all OSTs What do you think of the blue lines showing total number of inodes? It appears to me as though df -i and lfs df -i are reporting different numbers for this value which I'm not able to easily explain. Ashley.