From james.vanns at framestore.com Mon May 13 13:19:38 2013 From: james.vanns at framestore.com (James Vanns) Date: Mon, 13 May 2013 14:19:38 +0100 (BST) Subject: [Lustre-devel] Export over NFS sets rsize to 1MB? In-Reply-To: <1212911910.19338063.1368451002112.JavaMail.root@framestore.com> Message-ID: <1975652007.19339235.1368451178426.JavaMail.root@framestore.com> Hello dev list. Apologies for a post to perhaps the wrong group but I'm having a bit of difficulty locating any document or wiki describing how and/or where the preferred read and write block size for NFS exports of a Lustre filesystem are set to 1MB? Basically we have two Lustre filesystems exported over NFSv3. Our lustre block size is 4k and the max r/w size is 1MB. Without any special rsize/wsize options set for the export the default one suggested to clients (MOUNT->FSINFO RPC) as the preferred size is set to 1MB. How does Lustre figure this out? Other non-Lustre exports are generally much less; 4, 8, 16 or 32 kilobytes. Any hints would be appreciated. Documentation or code paths welcome as are annotated /proc locations. Thanks, Jim PS. This is Lustre 1.8.8 on Linux 2.6.32 -- Jim Vanns Senior Software Developer Framestore From twanjari at andrew.cmu.edu Fri May 10 22:03:27 2013 From: twanjari at andrew.cmu.edu (twanjari at andrew.cmu.edu) Date: Fri, 10 May 2013 18:03:27 -0400 Subject: [Lustre-devel] Interested in contributing to Lustre Message-ID: <5a77f40fe7860376e5bcdfaeb8e6cb7a.squirrel@webmail.andrew.cmu.edu> Hi all, I am a grad student at Carnegie Mellon University. I had my course work in advanced storage systems in previous semester, and I am interested to work on Lustre. I prefer to take up a project that could be completed in a duration of a month or two. Since I am a novice w.r.t. my familiarity with Lustre code base, I seek your opinion to choose a project from the list: http://wiki.lustre.org/index.php/Lustre_Project_List My preferences (in order) are: 1. fallocate() API - https://bugzilla.lustre.org/show_bug.cgi?id=15064 2. kernel patch removal - https://bugzilla.lustre.org/show_bug.cgi?id=21524 3. ioctl() number cleanups - https://bugzilla.lustre.org/show_bug.cgi?id=20731 I am also open to other suggestion. Any suggestions regarding code walkthrough or documentation to help ramping up are also welcome. Thanks, Tejas Wanjari From surya1 at llnl.gov Mon May 13 21:34:54 2013 From: surya1 at llnl.gov (Prakash Surya) Date: Mon, 13 May 2013 14:34:54 -0700 Subject: [Lustre-devel] Interested in contributing to Lustre In-Reply-To: <5a77f40fe7860376e5bcdfaeb8e6cb7a.squirrel@webmail.andrew.cmu.edu> References: <5a77f40fe7860376e5bcdfaeb8e6cb7a.squirrel@webmail.andrew.cmu.edu> Message-ID: <20130513213454.GF7378@llnl.gov> I wish you luck in your endeavour! Here are some links which might prove useful to you: * https://jira.hpdd.intel.com - New issues are opened here * http://review.whamcloud.com - New patches are reviewed here * https://wiki.hpdd.intel.com/display/PUB/Submitting+Changes - Some info on the development cycle used by the Lustre project. Unfortunately, there isn't nearly enough documentation as there should be in the Lustre tree (especially for new comers), so feel free to ask questions. As far as potential projects, in my opinion, the project list you linked to is a bit dated. There are *many* open issues which need to be fixed, so I would suggest finding something that interests you on the JIRA bug tracker that I linked to above, since that is a more up to date source of information. If you are completely new, a good place to start is just downloading the git tree, building, and running the unit tests locally to get familiar with the project. Then you can start to modify the source and explore some of the areas that interest you. -- Cheers, Prakash On Fri, May 10, 2013 at 06:03:27PM -0400, twanjari at andrew.cmu.edu wrote: > Hi all, > > I am a grad student at Carnegie Mellon University. I had my course work in > advanced storage systems in previous semester, and I am interested to work > on Lustre. I prefer to take up a project that could be completed in a > duration of a month or two. > > Since I am a novice w.r.t. my familiarity with Lustre code base, I seek > your opinion to choose a project from the list: > http://wiki.lustre.org/index.php/Lustre_Project_List > > My preferences (in order) are: > 1. fallocate() API - https://bugzilla.lustre.org/show_bug.cgi?id=15064 > 2. kernel patch removal - https://bugzilla.lustre.org/show_bug.cgi?id=21524 > 3. ioctl() number cleanups - > https://bugzilla.lustre.org/show_bug.cgi?id=20731 > I am also open to other suggestion. > > Any suggestions regarding code walkthrough or documentation to help > ramping up are also welcome. > > Thanks, > Tejas Wanjari > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From andreas.dilger at intel.com Mon May 13 21:58:44 2013 From: andreas.dilger at intel.com (Dilger, Andreas) Date: Mon, 13 May 2013 21:58:44 +0000 Subject: [Lustre-devel] Export over NFS sets rsize to 1MB? In-Reply-To: <1975652007.19339235.1368451178426.JavaMail.root@framestore.com> Message-ID: On 2013/13/05 7:19 AM, "James Vanns" wrote: >Hello dev list. Apologies for a post to perhaps the wrong group but I'm >having a >bit of difficulty locating any document or wiki describing how and/or >where the >preferred read and write block size for NFS exports of a Lustre >filesystem are >set to 1MB? 1MB is the RPC size and "optimal IO size" for Lustre. This would normally be exported to applications via the stat(2) "st_blksize" field, though it is typically 2MB (2x the RPC size in order to allow some pipelining). I suspect this is where NFS is getting the value, since it is not passed up via the statfs(2) call. >Basically we have two Lustre filesystems exported over NFSv3. Our lustre >block size >is 4k and the max r/w size is 1MB. Without any special rsize/wsize >options set for >the export the default one suggested to clients (MOUNT->FSINFO RPC) as >the preferred >size is set to 1MB. How does Lustre figure this out? Other non-Lustre >exports are generally much less; 4, 8, 16 or 32 kilobytes. Taking a quick look at the code, it looks like NFS TCP connections all have a maximum max_payload of 1MB, but this is limited in a number of places in the code by the actual read size, and other maxima (for which I can't easily find the source value). >Any hints would be appreciated. Documentation or code paths welcome as >are annotated /proc locations. To clarify from your question - is this large blocksize causing a performance problem? I recall some applications having problems with stdio "fread()" and friends reading too much data into their buffers if they are doing random IO. Ideally stdio shouldn't be reading more than it needs when doing random IO. At one time in the past, we derived the st_blksize from the file stripe_size, but this caused problems with the NFS "Connectathon" or similar. It is currently limited by LL_MAX_BLKSIZE_BITS for all files, but I wouldn't recommend reducing this directly, since it would also affect "cp" and others that also depend on st_blksize for the "optimal IO size". It would be possible to reintroduce the per-file tunable in ll_update_inode() I think. Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division From andreas.dilger at intel.com Tue May 14 00:55:05 2013 From: andreas.dilger at intel.com (Dilger, Andreas) Date: Tue, 14 May 2013 00:55:05 +0000 Subject: [Lustre-devel] Interested in contributing to Lustre In-Reply-To: <5a77f40fe7860376e5bcdfaeb8e6cb7a.squirrel@webmail.andrew.cmu.edu> References: <5a77f40fe7860376e5bcdfaeb8e6cb7a.squirrel@webmail.andrew.cmu.edu> Message-ID: On 2013-05-13, at 12:44, "twanjari at andrew.cmu.edu" > wrote: I am a grad student at Carnegie Mellon University. I had my course work in advanced storage systems in previous semester, and I am interested to work on Lustre. I prefer to take up a project that could be completed in a duration of a month or two. Since I am a novice w.r.t. my familiarity with Lustre code base, I seek your opinion to choose a project from the list: http://wiki.lustre.org/index.php/Lustre_Project_List A more up-to-date list is at: https://wiki.hpdd.intel.com/display/PUB/Project+Ideas My preferences (in order) are: 1. fallocate() API - https://bugzilla.lustre.org/show_bug.cgi?id=15064 This one is a bit tricky, but benefits an increasing number of users and would be the most "filesystem-y" of the listed projects. Bonus marks if you tie this in with the Lustre MPIIO-ADIO driver. 2. kernel patch removal - https://bugzilla.lustre.org/show_bug.cgi?id=21524 The main project here is to replace the "dev-read-only" patch with hooks into the "dm-flakey" driver or similar, that accepts reads, but does not write anything to disk. It is partly Lustre and partly in the kernel. 3. ioctl() number cleanups - https://bugzilla.lustre.org/show_bug.cgi?id=20731 This would help to get Lustre into the upstream kernel, but is not necessarily going to get you distributed filesystem experience. I am also open to other suggestion. Any suggestions regarding code walkthrough or documentation to help ramping up are also welcome. Thanks, Tejas Wanjari _______________________________________________ Lustre-devel mailing list Lustre-devel at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-devel From spitzcor at cray.com Tue May 14 14:07:32 2013 From: spitzcor at cray.com (Cory Spitz) Date: Tue, 14 May 2013 14:07:32 +0000 Subject: [Lustre-devel] Interested in contributing to Lustre In-Reply-To: <5a77f40fe7860376e5bcdfaeb8e6cb7a.squirrel@webmail.andrew.cmu.edu> Message-ID: Tejas, In addition to the Lustre_Project_List at wiki.lustre.org, there is another project list at https://wiki.hpdd.intel.com/display/PUB/Project+Ideas. Thanks, -Cory On 5/10/13 5:03 PM, "twanjari at andrew.cmu.edu" wrote: >Hi all, > >I am a grad student at Carnegie Mellon University. I had my course work in >advanced storage systems in previous semester, and I am interested to work >on Lustre. I prefer to take up a project that could be completed in a >duration of a month or two. > >Since I am a novice w.r.t. my familiarity with Lustre code base, I seek >your opinion to choose a project from the list: >http://wiki.lustre.org/index.php/Lustre_Project_List > >My preferences (in order) are: >1. fallocate() API - https://bugzilla.lustre.org/show_bug.cgi?id=15064 >2. kernel patch removal - >https://bugzilla.lustre.org/show_bug.cgi?id=21524 >3. ioctl() number cleanups - >https://bugzilla.lustre.org/show_bug.cgi?id=20731 >I am also open to other suggestion. > >Any suggestions regarding code walkthrough or documentation to help >ramping up are also welcome. > >Thanks, >Tejas Wanjari > >_______________________________________________ >Lustre-devel mailing list >Lustre-devel at lists.lustre.org >http://lists.lustre.org/mailman/listinfo/lustre-devel From oleg.drokin at intel.com Tue May 14 06:49:58 2013 From: oleg.drokin at intel.com (Drokin, Oleg) Date: Tue, 14 May 2013 06:49:58 +0000 Subject: [Lustre-devel] Interested in contributing to Lustre In-Reply-To: <20130513213454.GF7378@llnl.gov> References: <5a77f40fe7860376e5bcdfaeb8e6cb7a.squirrel@webmail.andrew.cmu.edu> <20130513213454.GF7378@llnl.gov> Message-ID: Additionally there's this pretty good (though stale in many parts now, but still some other parts are ok and high level is certainly ok) Lustre internals book: users.nccs.gov/~fwang2/papers/lustre_report.pdf‎ On May 13, 2013, at 5:34 PM, Prakash Surya wrote: > I wish you luck in your endeavour! > > Here are some links which might prove useful to you: > > * https://jira.hpdd.intel.com - New issues are opened here > > * http://review.whamcloud.com - New patches are reviewed here > > * https://wiki.hpdd.intel.com/display/PUB/Submitting+Changes - Some > info on the development cycle used by the Lustre project. > > Unfortunately, there isn't nearly enough documentation as there should > be in the Lustre tree (especially for new comers), so feel free to ask > questions. > > As far as potential projects, in my opinion, the project list you > linked to is a bit dated. There are *many* open issues which need to be > fixed, so I would suggest finding something that interests you on the > JIRA bug tracker that I linked to above, since that is a more up to date > source of information. > > If you are completely new, a good place to start is just downloading the > git tree, building, and running the unit tests locally to get familiar > with the project. Then you can start to modify the source and explore > some of the areas that interest you. > > -- > Cheers, Prakash > > On Fri, May 10, 2013 at 06:03:27PM -0400, twanjari at andrew.cmu.edu wrote: >> Hi all, >> >> I am a grad student at Carnegie Mellon University. I had my course work in >> advanced storage systems in previous semester, and I am interested to work >> on Lustre. I prefer to take up a project that could be completed in a >> duration of a month or two. >> >> Since I am a novice w.r.t. my familiarity with Lustre code base, I seek >> your opinion to choose a project from the list: >> http://wiki.lustre.org/index.php/Lustre_Project_List >> >> My preferences (in order) are: >> 1. fallocate() API - https://bugzilla.lustre.org/show_bug.cgi?id=15064 >> 2. kernel patch removal - https://bugzilla.lustre.org/show_bug.cgi?id=21524 >> 3. ioctl() number cleanups - >> https://bugzilla.lustre.org/show_bug.cgi?id=20731 >> I am also open to other suggestion. >> >> Any suggestions regarding code walkthrough or documentation to help >> ramping up are also welcome. >> >> Thanks, >> Tejas Wanjari >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From spitzcor at cray.com Tue May 14 14:58:58 2013 From: spitzcor at cray.com (Cory Spitz) Date: Tue, 14 May 2013 14:58:58 +0000 Subject: [Lustre-devel] Interested in contributing to Lustre In-Reply-To: Message-ID: Same disclaimer applies for http://wiki.lustre.org/lid/agi/agi.html. The OpenSFS CDWG has discussed bringing these documents up-to-date. Nothing concrete yet though. Thanks, -Cory On 5/14/13 1:49 AM, "Drokin, Oleg" wrote: >Additionally there's this pretty good (though stale in many parts now, >but still some other parts are ok and high level is certainly ok) >Lustre internals book: users.nccs.gov/~fwang2/papers/lustre_report.pdf? > >On May 13, 2013, at 5:34 PM, Prakash Surya wrote: > >> I wish you luck in your endeavour! >> >> Here are some links which might prove useful to you: >> >> * https://jira.hpdd.intel.com - New issues are opened here >> >> * http://review.whamcloud.com - New patches are reviewed here >> >> * https://wiki.hpdd.intel.com/display/PUB/Submitting+Changes - Some >> info on the development cycle used by the Lustre project. >> >> Unfortunately, there isn't nearly enough documentation as there should >> be in the Lustre tree (especially for new comers), so feel free to ask >> questions. >> >> As far as potential projects, in my opinion, the project list you >> linked to is a bit dated. There are *many* open issues which need to be >> fixed, so I would suggest finding something that interests you on the >> JIRA bug tracker that I linked to above, since that is a more up to date >> source of information. >> >> If you are completely new, a good place to start is just downloading the >> git tree, building, and running the unit tests locally to get familiar >> with the project. Then you can start to modify the source and explore >> some of the areas that interest you. >> >> -- >> Cheers, Prakash >> >> On Fri, May 10, 2013 at 06:03:27PM -0400, twanjari at andrew.cmu.edu wrote: >>> Hi all, >>> >>> I am a grad student at Carnegie Mellon University. I had my course >>>work in >>> advanced storage systems in previous semester, and I am interested to >>>work >>> on Lustre. I prefer to take up a project that could be completed in a >>> duration of a month or two. >>> >>> Since I am a novice w.r.t. my familiarity with Lustre code base, I seek >>> your opinion to choose a project from the list: >>> http://wiki.lustre.org/index.php/Lustre_Project_List >>> >>> My preferences (in order) are: >>> 1. fallocate() API - https://bugzilla.lustre.org/show_bug.cgi?id=15064 >>> 2. kernel patch removal - >>>https://bugzilla.lustre.org/show_bug.cgi?id=21524 >>> 3. ioctl() number cleanups - >>> https://bugzilla.lustre.org/show_bug.cgi?id=20731 >>> I am also open to other suggestion. >>> >>> Any suggestions regarding code walkthrough or documentation to help >>> ramping up are also welcome. >>> >>> Thanks, >>> Tejas Wanjari >>> >>> _______________________________________________ >>> Lustre-devel mailing list >>> Lustre-devel at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-devel >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel > >_______________________________________________ >Lustre-devel mailing list >Lustre-devel at lists.lustre.org >http://lists.lustre.org/mailman/listinfo/lustre-devel From james.vanns at framestore.com Tue May 14 15:07:00 2013 From: james.vanns at framestore.com (James Vanns) Date: Tue, 14 May 2013 16:07:00 +0100 (BST) Subject: [Lustre-devel] Export over NFS sets rsize to 1MB? In-Reply-To: Message-ID: <1182911409.19669249.1368544019962.JavaMail.root@framestore.com> Thanks for replying Andreas... > On 2013/13/05 7:19 AM, "James Vanns" > wrote: > >Hello dev list. Apologies for a post to perhaps the wrong group but > >I'm > >having a > >bit of difficulty locating any document or wiki describing how > >and/or > >where the > >preferred read and write block size for NFS exports of a Lustre > >filesystem are > >set to 1MB? > > 1MB is the RPC size and "optimal IO size" for Lustre. This would > normally > be exported to applications via the stat(2) "st_blksize" field, > though it > is typically 2MB (2x the RPC size in order to allow some pipelining). > I > suspect this is where NFS is getting the value, since it is not > passed up > via the statfs(2) call. Hmm. OK. I've confirmed it isn't from any struct stat{} attribute (st_blksize is still just 4k) but yes, our RPC size is 1MB. It isn't coming from statfs() or statvfs() either. > >Basically we have two Lustre filesystems exported over NFSv3. Our > >lustre > >block size > >is 4k and the max r/w size is 1MB. Without any special rsize/wsize > >options set for > >the export the default one suggested to clients (MOUNT->FSINFO RPC) > >as > >the preferred > >size is set to 1MB. How does Lustre figure this out? Other > >non-Lustre > >exports are generally much less; 4, 8, 16 or 32 kilobytes. > > Taking a quick look at the code, it looks like NFS TCP connections > all > have a maximum max_payload of 1MB, but this is limited in a number of > places in the code by the actual read size, and other maxima (for > which I > can't easily find the source value). Yes it seems that 1MB is the maximum but also the optimal or preferred. > >Any hints would be appreciated. Documentation or code paths welcome > >as > >are annotated /proc locations. > > To clarify from your question - is this large blocksize causing a > performance problem? I recall some applications having problems with > stdio "fread()" and friends reading too much data into their buffers > if > they are doing random IO. Ideally stdio shouldn't be reading more > than it > needs when doing random IO. We're experiencing what appears to be (as of yet I have no hard evidence) contention due to connection 'hogging' for these large reads. We have a set of 4 NFS servers in a DNS round-robin all configured to serve up our Lustre filesystem across 64 knfsds (per host). It's possible that we simply don't have enough hosts (or knfsds) for the #clients because many of the clients will be reading large amounts of data (1MB at a time) and therefore preventing other queued clients from getting a look-in. Of course this appears to the user as just a very slow experience. At the moment, I'm just trying to understand where this 1MB is coming from! The RPC transport size (I forgot to confirm - yes, we're serving NFS over TCP) is 1MB for all other 'regular' NFS servers yet their r/wsize are quite different. Thanks for the feedback and sorry I can't be more accurate at the moment :\ Jim > At one time in the past, we derived the st_blksize from the file > stripe_size, but this caused problems with the NFS "Connectathon" or > similar. It is currently limited by LL_MAX_BLKSIZE_BITS for all > files, > but I wouldn't recommend reducing this directly, since it would also > affect "cp" and others that also depend on st_blksize for the > "optimal IO > size". It would be possible to reintroduce the per-file tunable in > ll_update_inode() I think. > > Cheers, Andreas > -- > Andreas Dilger > > Lustre Software Architect > Intel High Performance Data Division > > > -- Jim Vanns Senior Software Developer Framestore From nikitas_angelinas at xyratex.com Tue May 14 15:23:08 2013 From: nikitas_angelinas at xyratex.com (Nikitas Angelinas) Date: Tue, 14 May 2013 16:23:08 +0100 Subject: [Lustre-devel] Interested in contributing to Lustre In-Reply-To: References: Message-ID: In addition to the document Oleg has pointed out (which imho is probably the best resource for a newcomer developer to start with), and the link that Cory has provided which also includes documentation on some more recent additions to the source (and is also a good source of information for newcomers imo), some older HLDs can be found at http://wiki.lustre.org/index.php/Architecture_-_Lustre_HLDs, and some newer HLDs/architectural docs at https://wiki.hpdd.intel.com/display/PUB/Whamcloud+Designs. The Operations Manual is also quite useful i think, at https://wiki.hpdd.intel.com/display/PUB/Documentation. Added to all this, there are quite a few presentation slides and av recordings from previous LUG/LAD or other events around the web that you might find useful. As Oleg has hinted, some of the information you may come across might be a bit stale, as it applies to previous versions of the source, so please keep in mind that whole subsystems that are referenced in a document you are reading may have been completely replaced (by newer and better versions). But don't let this be an issue, please ask here if something does not make sense; people are usually quite keen to help. Cheers, Nikitas On 14 May 2013 15:58, Cory Spitz wrote: > Same disclaimer applies for http://wiki.lustre.org/lid/agi/agi.html. > > The OpenSFS CDWG has discussed bringing these documents up-to-date. > Nothing concrete yet though. > > Thanks, > -Cory > > > On 5/14/13 1:49 AM, "Drokin, Oleg" wrote: > > >Additionally there's this pretty good (though stale in many parts now, > >but still some other parts are ok and high level is certainly ok) > >Lustre internals book: users.nccs.gov/~fwang2/papers/lustre_report.pdf? > > > >On May 13, 2013, at 5:34 PM, Prakash Surya wrote: > > > >> I wish you luck in your endeavour! > >> > >> Here are some links which might prove useful to you: > >> > >> * https://jira.hpdd.intel.com - New issues are opened here > >> > >> * http://review.whamcloud.com - New patches are reviewed here > >> > >> * https://wiki.hpdd.intel.com/display/PUB/Submitting+Changes - Some > >> info on the development cycle used by the Lustre project. > >> > >> Unfortunately, there isn't nearly enough documentation as there should > >> be in the Lustre tree (especially for new comers), so feel free to ask > >> questions. > >> > >> As far as potential projects, in my opinion, the project list you > >> linked to is a bit dated. There are *many* open issues which need to be > >> fixed, so I would suggest finding something that interests you on the > >> JIRA bug tracker that I linked to above, since that is a more up to date > >> source of information. > >> > >> If you are completely new, a good place to start is just downloading the > >> git tree, building, and running the unit tests locally to get familiar > >> with the project. Then you can start to modify the source and explore > >> some of the areas that interest you. > >> > >> -- > >> Cheers, Prakash > >> > >> On Fri, May 10, 2013 at 06:03:27PM -0400, twanjari at andrew.cmu.eduwrote: > >>> Hi all, > >>> > >>> I am a grad student at Carnegie Mellon University. I had my course > >>>work in > >>> advanced storage systems in previous semester, and I am interested to > >>>work > >>> on Lustre. I prefer to take up a project that could be completed in a > >>> duration of a month or two. > >>> > >>> Since I am a novice w.r.t. my familiarity with Lustre code base, I seek > >>> your opinion to choose a project from the list: > >>> http://wiki.lustre.org/index.php/Lustre_Project_List > >>> > >>> My preferences (in order) are: > >>> 1. fallocate() API - https://bugzilla.lustre.org/show_bug.cgi?id=15064 > >>> 2. kernel patch removal - > >>>https://bugzilla.lustre.org/show_bug.cgi?id=21524 > >>> 3. ioctl() number cleanups - > >>> https://bugzilla.lustre.org/show_bug.cgi?id=20731 > >>> I am also open to other suggestion. > >>> > >>> Any suggestions regarding code walkthrough or documentation to help > >>> ramping up are also welcome. > >>> > >>> Thanks, > >>> Tejas Wanjari > >>> > >>> _______________________________________________ > >>> Lustre-devel mailing list > >>> Lustre-devel at lists.lustre.org > >>> http://lists.lustre.org/mailman/listinfo/lustre-devel > >> _______________________________________________ > >> Lustre-devel mailing list > >> Lustre-devel at lists.lustre.org > >> http://lists.lustre.org/mailman/listinfo/lustre-devel > > > >_______________________________________________ > >Lustre-devel mailing list > >Lustre-devel at lists.lustre.org > >http://lists.lustre.org/mailman/listinfo/lustre-devel > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > -- ------------------------------ For additional information including the registered office and the treatment of Xyratex confidential information please visit www.xyratex.com ------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.dilger at intel.com Tue May 14 22:06:08 2013 From: andreas.dilger at intel.com (Dilger, Andreas) Date: Tue, 14 May 2013 22:06:08 +0000 Subject: [Lustre-devel] Export over NFS sets rsize to 1MB? In-Reply-To: <1182911409.19669249.1368544019962.JavaMail.root@framestore.com> Message-ID: On 2013/14/05 9:07 AM, "James Vanns" wrote: >> On 2013/13/05 7:19 AM, "James Vanns" >> wrote: >> >Hello dev list. Apologies for a post to perhaps the wrong group but >> >I'm having a bit of difficulty locating any document or wiki describing >> >how and/or where the preferred read and write block size for NFS >> >exports of a Lustre filesystem are set to 1MB? >> >> 1MB is the RPC size and "optimal IO size" for Lustre. This would >> normally be exported to applications via the stat(2) "st_blksize" field, >> though it is typically 2MB (2x the RPC size in order to allow some >>pipelining). I suspect this is where NFS is getting the value, since >>it is not passed up via the statfs(2) call. > >Hmm. OK. I've confirmed it isn't from any struct stat{} attribute >(st_blksize >is still just 4k) but yes, our RPC size is 1MB. It isn't coming from >statfs() >or statvfs() either. I've CC'd the Linux NFS mailing list, since I don't know enough about the NFS client/server code to decide where this is coming from either. James, what kernel version do you have on the Lustre clients (NFS servers) and on the NFS clients? >> >Basically we have two Lustre filesystems exported over NFSv3. Our >> >lustre block size is 4k and the max r/w size is 1MB. Without any >> >special rsize/wsize options set for the export the default one >> >suggested to clients (MOUNT->FSINFO RPC) as the preferred >> >size is set to 1MB. How does Lustre figure this out? Other >> >non-Lustre exports are generally much less; 4, 8, 16 or 32 kilobytes. >> >> Taking a quick look at the code, it looks like NFS TCP connections >> all have a maximum max_payload of 1MB, but this is limited in a number >>of places in the code by the actual read size, and other maxima (for >> which I can't easily find the source value). > >Yes it seems that 1MB is the maximum but also the optimal or preferred. > >> >Any hints would be appreciated. Documentation or code paths welcome >> >as are annotated /proc locations. >> >> To clarify from your question - is this large blocksize causing a >> performance problem? I recall some applications having problems with >> stdio "fread()" and friends reading too much data into their buffers >> if they are doing random IO. Ideally stdio shouldn't be reading more >> than it needs when doing random IO. > >We're experiencing what appears to be (as of yet I have no hard evidence) >contention due to connection 'hogging' for these large reads. We have a >set >of 4 NFS servers in a DNS round-robin all configured to serve up our >Lustre >filesystem across 64 knfsds (per host). It's possible that we simply don't >have enough hosts (or knfsds) for the #clients because many of the clients >will be reading large amounts of data (1MB at a time) and therefore >preventing >other queued clients from getting a look-in. Of course this appears to >the user >as just a very slow experience. > >At the moment, I'm just trying to understand where this 1MB is coming >from! >The RPC transport size (I forgot to confirm - yes, we're serving NFS over >TCP) is 1MB for all other 'regular' NFS servers yet their r/wsize are >quite different. > >Thanks for the feedback and sorry I can't be more accurate at the moment >:\ It should also be possible to explicitly mount the clients with rsize=65536 and wsize=65536, but it would be better to understand the cause of this. >> At one time in the past, we derived the st_blksize from the file >> stripe_size, but this caused problems with the NFS "Connectathon" or >> similar because the block size would change from when the file was >>first opened. It is currently limited by LL_MAX_BLKSIZE_BITS for all >> files, but I wouldn't recommend reducing this directly, since it would >> also affect "cp" and others that also depend on st_blksize for the >> "optimal IO size". It would be possible to reintroduce the per-file >> tunable in ll_update_inode() I think. Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division From james.vanns at framestore.com Wed May 15 09:23:17 2013 From: james.vanns at framestore.com (James Vanns) Date: Wed, 15 May 2013 10:23:17 +0100 (BST) Subject: [Lustre-devel] Export over NFS sets rsize to 1MB? In-Reply-To: Message-ID: <2121037782.19846759.1368609797839.JavaMail.root@framestore.com> Thanks for the help Andreas - I too asked the NFS developers (on the list) and they came back with /proc/fs/nfsd/max_block_size and nfsd_get_default_max_blksize() in fs/nfsd/nfssvc.c. For what it's worth, this is kernel 2.6.32 and Lustre 1.8.8. Reckon I've got what I asked for now ;) Jim ----- Original Message ----- From: "Andreas Dilger" To: "james vanns" Cc: lustre-devel at lists.lustre.org, linux-nfs at vger.kernel.org Sent: Tuesday, 14 May, 2013 11:06:08 PM Subject: Re: [Lustre-devel] Export over NFS sets rsize to 1MB? On 2013/14/05 9:07 AM, "James Vanns" wrote: >> On 2013/13/05 7:19 AM, "James Vanns" >> wrote: >> >Hello dev list. Apologies for a post to perhaps the wrong group but >> >I'm having a bit of difficulty locating any document or wiki describing >> >how and/or where the preferred read and write block size for NFS >> >exports of a Lustre filesystem are set to 1MB? >> >> 1MB is the RPC size and "optimal IO size" for Lustre. This would >> normally be exported to applications via the stat(2) "st_blksize" field, >> though it is typically 2MB (2x the RPC size in order to allow some >>pipelining). I suspect this is where NFS is getting the value, since >>it is not passed up via the statfs(2) call. > >Hmm. OK. I've confirmed it isn't from any struct stat{} attribute >(st_blksize >is still just 4k) but yes, our RPC size is 1MB. It isn't coming from >statfs() >or statvfs() either. I've CC'd the Linux NFS mailing list, since I don't know enough about the NFS client/server code to decide where this is coming from either. James, what kernel version do you have on the Lustre clients (NFS servers) and on the NFS clients? >> >Basically we have two Lustre filesystems exported over NFSv3. Our >> >lustre block size is 4k and the max r/w size is 1MB. Without any >> >special rsize/wsize options set for the export the default one >> >suggested to clients (MOUNT->FSINFO RPC) as the preferred >> >size is set to 1MB. How does Lustre figure this out? Other >> >non-Lustre exports are generally much less; 4, 8, 16 or 32 kilobytes. >> >> Taking a quick look at the code, it looks like NFS TCP connections >> all have a maximum max_payload of 1MB, but this is limited in a number >>of places in the code by the actual read size, and other maxima (for >> which I can't easily find the source value). > >Yes it seems that 1MB is the maximum but also the optimal or preferred. > >> >Any hints would be appreciated. Documentation or code paths welcome >> >as are annotated /proc locations. >> >> To clarify from your question - is this large blocksize causing a >> performance problem? I recall some applications having problems with >> stdio "fread()" and friends reading too much data into their buffers >> if they are doing random IO. Ideally stdio shouldn't be reading more >> than it needs when doing random IO. > >We're experiencing what appears to be (as of yet I have no hard evidence) >contention due to connection 'hogging' for these large reads. We have a >set >of 4 NFS servers in a DNS round-robin all configured to serve up our >Lustre >filesystem across 64 knfsds (per host). It's possible that we simply don't >have enough hosts (or knfsds) for the #clients because many of the clients >will be reading large amounts of data (1MB at a time) and therefore >preventing >other queued clients from getting a look-in. Of course this appears to >the user >as just a very slow experience. > >At the moment, I'm just trying to understand where this 1MB is coming >from! >The RPC transport size (I forgot to confirm - yes, we're serving NFS over >TCP) is 1MB for all other 'regular' NFS servers yet their r/wsize are >quite different. > >Thanks for the feedback and sorry I can't be more accurate at the moment >:\ It should also be possible to explicitly mount the clients with rsize=65536 and wsize=65536, but it would be better to understand the cause of this. >> At one time in the past, we derived the st_blksize from the file >> stripe_size, but this caused problems with the NFS "Connectathon" or >> similar because the block size would change from when the file was >>first opened. It is currently limited by LL_MAX_BLKSIZE_BITS for all >> files, but I wouldn't recommend reducing this directly, since it would >> also affect "cp" and others that also depend on st_blksize for the >> "optimal IO size". It would be possible to reintroduce the per-file >> tunable in ll_update_inode() I think. Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Jim Vanns Senior Software Developer Framestore From ronc at lanl.gov Tue May 21 14:08:52 2013 From: ronc at lanl.gov (Ron Croonenberg) Date: Tue, 21 May 2013 08:08:52 -0600 Subject: [Lustre-devel] building the 'lustre kernel' In-Reply-To: References: <5a77f40fe7860376e5bcdfaeb8e6cb7a.squirrel@webmail.andrew.cmu.edu> <20130513213454.GF7378@llnl.gov> Message-ID: <519B7FF4.3040500@lanl.gov> I am looking into some Lustre development but want to build everything from scratch. I looked around to see how the linux kernel is configured but don't seem to find anything. any ideas? thanks, Ron From ronc at lanl.gov Tue May 21 14:11:37 2013 From: ronc at lanl.gov (Ron Croonenberg) Date: Tue, 21 May 2013 08:11:37 -0600 Subject: [Lustre-devel] lustre kernel configuration In-Reply-To: References: <5a77f40fe7860376e5bcdfaeb8e6cb7a.squirrel@webmail.andrew.cmu.edu> <20130513213454.GF7378@llnl.gov> Message-ID: <519B8099.7010803@lanl.gov> hello, I am looking into some Lustre development but want to build everything from scratch. I looked around to see how the linux kernel is configured but don't seem to find anything. any ideas? thanks, Ron From morrone2 at llnl.gov Tue May 21 18:16:10 2013 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Tue, 21 May 2013 11:16:10 -0700 Subject: [Lustre-devel] lustre kernel configuration In-Reply-To: <519B8099.7010803@lanl.gov> References: <5a77f40fe7860376e5bcdfaeb8e6cb7a.squirrel@webmail.andrew.cmu.edu> <20130513213454.GF7378@llnl.gov> <519B8099.7010803@lanl.gov> Message-ID: <519BB9EA.70304@llnl.gov> If you use the zfs osd for your servers, there is no need to modify your Linux kernel. That is probably a much easier way to get started with Lustre development. If you are set on using the ldiskfs osd, Intel has some instructions on their wiki here: https://wiki.hpdd.intel.com/display/PUB/Building+Lustre+from+Source Chris On 05/21/2013 07:11 AM, Ron Croonenberg wrote: > hello, > > I am looking into some Lustre development but want to build everything > from scratch. I looked around to see how the linux kernel is configured > but don't seem to find anything. > > any ideas? > > thanks, > > Ron > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From ronc at lanl.gov Tue May 21 19:04:58 2013 From: ronc at lanl.gov (Ron Croonenberg) Date: Tue, 21 May 2013 13:04:58 -0600 Subject: [Lustre-devel] lustre kernel configuration In-Reply-To: <519BB9EA.70304@llnl.gov> References: <5a77f40fe7860376e5bcdfaeb8e6cb7a.squirrel@webmail.andrew.cmu.edu> <20130513213454.GF7378@llnl.gov> <519B8099.7010803@lanl.gov> <519BB9EA.70304@llnl.gov> Message-ID: <519BC55A.4040903@lanl.gov> Hi Chris, thanks for the link. What I was looking for was the Linux kernel config but couldn't find any documentation foor (would have been nice to find the ones that are actuallu used in the binaries). Basically what I did is take my RHEL6 config and configured the kernel wit and compiling that seemed to work. (I did compile the rpms, just a regular make. That seemed to work for 1.8.7. I haven't installed anything yet., so we'll see on that part. thanks, Ron On 05/21/2013 12:16 PM, Christopher J. Morrone wrote: > If you use the zfs osd for your servers, there is no need to modify your > Linux kernel. That is probably a much easier way to get started with > Lustre development. > > If you are set on using the ldiskfs osd, Intel has some instructions on > their wiki here: > > https://wiki.hpdd.intel.com/display/PUB/Building+Lustre+from+Source > > Chris > > On 05/21/2013 07:11 AM, Ron Croonenberg wrote: >> hello, >> >> I am looking into some Lustre development but want to build everything >> from scratch. I looked around to see how the linux kernel is configured >> but don't seem to find anything. >> >> any ideas? >> >> thanks, >> >> Ron >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From spitzcor at cray.com Tue May 21 19:24:29 2013 From: spitzcor at cray.com (Cory Spitz) Date: Tue, 21 May 2013 19:24:29 +0000 Subject: [Lustre-devel] lustre kernel configuration In-Reply-To: <519BC55A.4040903@lanl.gov> Message-ID: Kernel configs are kept in the tree. You can find master's set at http://git.whamcloud.com/?p=fs/lustre-release.git;a=tree;f=lustre/kernel_pa tches/kernel_configs;h=88ca1484fa6597a7d786f3f09f03612f6d039db2;hb=HEAD -Cory On 5/21/13 2:04 PM, "Ron Croonenberg" wrote: >Hi Chris, > >thanks for the link. What I was looking for was the Linux kernel config >but couldn't find any documentation foor (would have been nice to find >the ones that are actuallu used in the binaries). > >Basically what I did is take my RHEL6 config and configured the kernel >wit and compiling that seemed to work. (I did compile the rpms, just a >regular make. That seemed to work for 1.8.7. >I haven't installed anything yet., so we'll see on that part. > >thanks, > >Ron > >On 05/21/2013 12:16 PM, Christopher J. Morrone wrote: >> If you use the zfs osd for your servers, there is no need to modify your >> Linux kernel. That is probably a much easier way to get started with >> Lustre development. >> >> If you are set on using the ldiskfs osd, Intel has some instructions on >> their wiki here: >> >> https://wiki.hpdd.intel.com/display/PUB/Building+Lustre+from+Source >> >> Chris >> >> On 05/21/2013 07:11 AM, Ron Croonenberg wrote: >>> hello, >>> >>> I am looking into some Lustre development but want to build everything >>> from scratch. I looked around to see how the linux kernel is configured >>> but don't seem to find anything. >>> >>> any ideas? >>> >>> thanks, >>> >>> Ron >>> _______________________________________________ >>> Lustre-devel mailing list >>> Lustre-devel at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>> >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> >_______________________________________________ >Lustre-devel mailing list >Lustre-devel at lists.lustre.org >http://lists.lustre.org/mailman/listinfo/lustre-devel From ronc at lanl.gov Tue May 21 19:36:40 2013 From: ronc at lanl.gov (Ron Croonenberg) Date: Tue, 21 May 2013 13:36:40 -0600 Subject: [Lustre-devel] lustre kernel configuration In-Reply-To: References: Message-ID: <519BCCC8.70506@lanl.gov> ok, cool thanks.. that probably will work better. (there were still a bunch of config options that were 'NEW' but a lot less Ron On 05/21/2013 01:24 PM, Cory Spitz wrote: > Kernel configs are kept in the tree. You can find master's set at > http://git.whamcloud.com/?p=fs/lustre-release.git;a=tree;f=lustre/kernel_pa > tches/kernel_configs;h=88ca1484fa6597a7d786f3f09f03612f6d039db2;hb=HEAD > > -Cory > > On 5/21/13 2:04 PM, "Ron Croonenberg" wrote: > >> Hi Chris, >> >> thanks for the link. What I was looking for was the Linux kernel config >> but couldn't find any documentation foor (would have been nice to find >> the ones that are actuallu used in the binaries). >> >> Basically what I did is take my RHEL6 config and configured the kernel >> wit and compiling that seemed to work. (I did compile the rpms, just a >> regular make. That seemed to work for 1.8.7. >> I haven't installed anything yet., so we'll see on that part. >> >> thanks, >> >> Ron >> >> On 05/21/2013 12:16 PM, Christopher J. Morrone wrote: >>> If you use the zfs osd for your servers, there is no need to modify your >>> Linux kernel. That is probably a much easier way to get started with >>> Lustre development. >>> >>> If you are set on using the ldiskfs osd, Intel has some instructions on >>> their wiki here: >>> >>> https://wiki.hpdd.intel.com/display/PUB/Building+Lustre+from+Source >>> >>> Chris >>> >>> On 05/21/2013 07:11 AM, Ron Croonenberg wrote: >>>> hello, >>>> >>>> I am looking into some Lustre development but want to build everything >>>> from scratch. I looked around to see how the linux kernel is configured >>>> but don't seem to find anything. >>>> >>>> any ideas? >>>> >>>> thanks, >>>> >>>> Ron >>>> _______________________________________________ >>>> Lustre-devel mailing list >>>> Lustre-devel at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>>> >>> >>> _______________________________________________ >>> Lustre-devel mailing list >>> Lustre-devel at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >