From kwestneat at datadirectnet.com Thu Dec 6 21:04:45 2007 From: kwestneat at datadirectnet.com (Kit Westneat) Date: Thu, 06 Dec 2007 16:04:45 -0500 Subject: [Lustre-devel] Problems with make rpms Message-ID: <475863ED.30908@datadirectnet.com> Hello, I'm trying to compile the b1_6 CVS branch, but I'm running into problems with 'make rpms'. 'make' works fine, but when I do 'make rpms' I get: make[4]: *** No rule to make target `/usr/src/redhat/BUILD/lustre-1.6.4.50/ldiskfs/Module.symvers', needed by `Module.symvers'. Stop. Any thoughts? Thanks, Kit From Alexey.Lyashkov at Sun.COM Fri Dec 7 04:54:31 2007 From: Alexey.Lyashkov at Sun.COM (Alex Lyashkov) Date: Fri, 07 Dec 2007 06:54:31 +0200 Subject: [Lustre-devel] Problems with make rpms In-Reply-To: <475863ED.30908@datadirectnet.com> References: <475863ED.30908@datadirectnet.com> Message-ID: <1197003271.3550.26.camel@bear.shadowland> Hello Kit, What kernel you used? is you build new kernel before build lustre? I'm ask about it, because some times ago in kernel series was added new patch for correctly create Module.symvers, if kernel sources not have this patch - this will be produce similar error. On Thu, 2007-12-06 at 16:04 -0500, Kit Westneat wrote: > Hello, > > I'm trying to compile the b1_6 CVS branch, but I'm running into problems > with 'make rpms'. 'make' works fine, but when I do 'make rpms' I get: > > make[4]: *** No rule to make target > `/usr/src/redhat/BUILD/lustre-1.6.4.50/ldiskfs/Module.symvers', needed > by `Module.symvers'. Stop. > > Any thoughts? > > Thanks, > Kit > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-devel From kwestneat at datadirectnet.com Fri Dec 7 22:32:56 2007 From: kwestneat at datadirectnet.com (Kit Westneat) Date: Fri, 07 Dec 2007 17:32:56 -0500 Subject: [Lustre-devel] Problems with make rpms In-Reply-To: <1197003271.3550.26.camel@bear.shadowland> References: <475863ED.30908@datadirectnet.com> <1197003271.3550.26.camel@bear.shadowland> Message-ID: <4759CA18.6040406@datadirectnet.com> Hi Alex, This is using 2.6.9-55.0.9.EL_lustre.1.6.3smp, on a Centos4 machine. I used the kernel-source rpm off of lustre.org, and then did make oldconfig and make, after changing the EXTRAVERSION to say smp instead of custom. I've attached a capture of make rpms after doing a make clean. Let me know if there is anything else that would be interesting. Thanks, Kit Alex Lyashkov wrote: > Hello Kit, > > What kernel you used? is you build new kernel before build lustre? > I'm ask about it, because some times ago in kernel series was added new > patch for correctly create Module.symvers, if kernel sources not have > this patch - this will be produce similar error. > > On Thu, 2007-12-06 at 16:04 -0500, Kit Westneat wrote: > >> Hello, >> >> I'm trying to compile the b1_6 CVS branch, but I'm running into problems >> with 'make rpms'. 'make' works fine, but when I do 'make rpms' I get: >> >> make[4]: *** No rule to make target >> `/usr/src/redhat/BUILD/lustre-1.6.4.50/ldiskfs/Module.symvers', needed >> by `Module.symvers'. Stop. >> >> Any thoughts? >> >> Thanks, >> Kit >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-devel >> > > -------------- next part -------------- A non-text attachment was scrubbed... Name: attachments.zip Type: application/octet-stream Size: 8473 bytes Desc: not available URL: From Alexey.Lyashkov at Sun.COM Sat Dec 8 05:45:31 2007 From: Alexey.Lyashkov at Sun.COM (Alex Lyashkov) Date: Sat, 08 Dec 2007 07:45:31 +0200 Subject: [Lustre-devel] Problems with make rpms In-Reply-To: <4759CA18.6040406@datadirectnet.com> References: <475863ED.30908@datadirectnet.com> <1197003271.3550.26.camel@bear.shadowland> <4759CA18.6040406@datadirectnet.com> Message-ID: <1197092731.3550.35.camel@bear.shadowland> On Fri, 2007-12-07 at 17:32 -0500, Kit Westneat wrote: > Hi Alex, > > This is using 2.6.9-55.0.9.EL_lustre.1.6.3smp, on a Centos4 machine. I > used the kernel-source rpm off of lustre.org, and then did make > oldconfig and make, after changing the EXTRAVERSION to say smp instead > of custom. > > I've attached a capture of make rpms after doing a make clean. Let me > know if there is anything else that would be interesting. In this rpm not included patch which added to kernel series after 1.6.3 released and new kernels for 1.6.4 is not released. If you want build b1_6, you can have two cases: 1) install quilt and apply rhel4 CFS kernel series into stock redhat kernel. 2) or need go into lustre/kernel_patches/patches and apply modpost_external_module_updates_rhel4.patch to you kernel sources by hand. after it kernel need to rebuild (for second case not need install because changes only on build rules and modpost utility) > > Thanks, > Kit > > Alex Lyashkov wrote: > > Hello Kit, > > > > What kernel you used? is you build new kernel before build lustre? > > I'm ask about it, because some times ago in kernel series was added new > > patch for correctly create Module.symvers, if kernel sources not have > > this patch - this will be produce similar error. > > > > On Thu, 2007-12-06 at 16:04 -0500, Kit Westneat wrote: > > > >> Hello, > >> > >> I'm trying to compile the b1_6 CVS branch, but I'm running into problems > >> with 'make rpms'. 'make' works fine, but when I do 'make rpms' I get: > >> > >> make[4]: *** No rule to make target > >> `/usr/src/redhat/BUILD/lustre-1.6.4.50/ldiskfs/Module.symvers', needed > >> by `Module.symvers'. Stop. > >> > >> Any thoughts? > >> > >> Thanks, > >> Kit > >> > >> _______________________________________________ > >> Lustre-devel mailing list > >> Lustre-devel at clusterfs.com > >> https://mail.clusterfs.com/mailman/listinfo/lustre-devel > >> > > > > > From Peter.Braam at Sun.COM Tue Dec 11 18:25:39 2007 From: Peter.Braam at Sun.COM (Peter Braam) Date: Tue, 11 Dec 2007 11:25:39 -0700 Subject: [Lustre-devel] read ahead Message-ID: <475ED623.6090204@sun.com> Can anyone tell me if read ahead in Lustre includes "early return" features. I mean that if I read 4K and readahead decides to fetch 1M will my request get serviced when the first 4K arrives? Is this important? - Peter - From Nikita.Danilov at Sun.COM Tue Dec 11 18:42:13 2007 From: Nikita.Danilov at Sun.COM (Nikita Danilov) Date: Tue, 11 Dec 2007 21:42:13 +0300 Subject: [Lustre-devel] read ahead In-Reply-To: <475ED623.6090204@sun.com> References: <475ED623.6090204@sun.com> Message-ID: <18270.55813.185376.299242@gargle.gargle.HOWL> Peter Braam writes: > Can anyone tell me if read ahead in Lustre includes "early return" > features. I mean that if I read 4K and readahead decides to fetch 1M > will my request get serviced when the first 4K arrives? Is this important? Currently read system call will proceed when the first RPC (including first 4K page and some number of read-ahead pages) is serviced: generic_file_read() waits on a page lock, and lock is released by completion routine (ll_ap_completion()). The problem with the "early return" (if I understood it correctly) is that large read-ahead window is an indication of sequential access, and in that case read is going to wait on the next page anyway. > > - Peter - Nikita. From Oleg.Drokin at Sun.COM Tue Dec 11 18:43:42 2007 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Tue, 11 Dec 2007 13:43:42 -0500 Subject: [Lustre-devel] read ahead In-Reply-To: <475ED623.6090204@sun.com> References: <475ED623.6090204@sun.com> Message-ID: <88917D9B-1A3F-4C20-8764-FD9A4A2FB973@sun.com> Hello! On Dec 11, 2007, at 1:25 PM, Peter Braam wrote: > Can anyone tell me if read ahead in Lustre includes "early return" > features. I mean that if I read 4K and readahead decides to fetch > 1M will my request get serviced when the first 4K arrives? Is this > important? I think this is impossible to implement with current architecture. We have one bulk RPC (1M in size) that until received completely, won't issue any callbacks. So only when that entire 1M is received your 4k request would return. On the other hand if your example is 4k and 2M, then we will return after 1M that contains requested 4k is received (but there is no guarantee at the moment we won't receive second 2M first, I believe). Bye, Oleg From Peter.Braam at Sun.COM Tue Dec 11 18:59:13 2007 From: Peter.Braam at Sun.COM (Peter Braam) Date: Tue, 11 Dec 2007 11:59:13 -0700 Subject: [Lustre-devel] read ahead In-Reply-To: <88917D9B-1A3F-4C20-8764-FD9A4A2FB973@sun.com> References: <475ED623.6090204@sun.com> <88917D9B-1A3F-4C20-8764-FD9A4A2FB973@sun.com> Message-ID: <475EDE01.2070408@sun.com> This might be quite damaging in some situations - for example, if the server has the 4K data cached in RAM it should refuse to do a disk read probably, but in order to do so it would need to know that part of the request is optional, while the 4K is mandatory. Can we give hints to the OSC about what part of I/O is requested by applications and what is requested for read-ahead? If so, could we use a more interesting IOV to do this faster? - Peter - Oleg Drokin wrote: > Hello! > > On Dec 11, 2007, at 1:25 PM, Peter Braam wrote: > >> Can anyone tell me if read ahead in Lustre includes "early return" >> features. I mean that if I read 4K and readahead decides to fetch 1M >> will my request get serviced when the first 4K arrives? Is this >> important? > > I think this is impossible to implement with current architecture. > We have one bulk RPC (1M in size) that until received completely, > won't issue any callbacks. > So only when that entire 1M is received your 4k request would return. > On the other hand if your example is 4k and 2M, then we will return > after 1M that contains requested 4k is received (but there is no > guarantee at the moment we won't receive second 2M first, I believe). > > Bye, > Oleg From Oleg.Drokin at Sun.COM Tue Dec 11 19:16:15 2007 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Tue, 11 Dec 2007 14:16:15 -0500 Subject: [Lustre-devel] read ahead In-Reply-To: <475EDE01.2070408@sun.com> References: <475ED623.6090204@sun.com> <88917D9B-1A3F-4C20-8764-FD9A4A2FB973@sun.com> <475EDE01.2070408@sun.com> Message-ID: <7CF747DE-23AE-4806-8AC9-3C79B8117BDC@Sun.COM> Hello! Unfortunately, currently osc has no idea about what was original read request. Original request size is only known in ll_file_read, that only gets proper lock. Then we jump into generic_file_read that calls ll_readpage for every page that needs to be read. ll_readpage has no idea how many more pages are going to be read in this request if any more, so we just try to stuff as much as we can into RPC (within our redahead window). Actually, now that I look into it, there is special readahead structure filled that tells how big this read reqest is, so ll_readahed can adjust the window size for the entire read request to fit in. So it seems it is possible to see what pages are readahead and what are from original request at ll_readahead level and we can pass that info down to osc as some sort of flag if needed. But we do not (yet?) have any caching on OST aside from device cache and we have no way to know what's in device cache too. I am not sure what do you mean by more interesting iov. Bye, Oleg On Dec 11, 2007, at 1:59 PM, Peter Braam wrote: > This might be quite damaging in some situations - for example, if > the server has the 4K data cached in RAM it should refuse to do a > disk read probably, but in order to do so it would need to know that > part of the request is optional, while the 4K is mandatory. > > Can we give hints to the OSC about what part of I/O is requested by > applications and what is requested for read-ahead? If so, could we > use a more interesting IOV to do this faster? > > - Peter - > > Oleg Drokin wrote: >> Hello! >> >> On Dec 11, 2007, at 1:25 PM, Peter Braam wrote: >> >>> Can anyone tell me if read ahead in Lustre includes "early return" >>> features. I mean that if I read 4K and readahead decides to fetch >>> 1M will my request get serviced when the first 4K arrives? Is >>> this important? >> >> I think this is impossible to implement with current architecture. >> We have one bulk RPC (1M in size) that until received completely, >> won't issue any callbacks. >> So only when that entire 1M is received your 4k request would return. >> On the other hand if your example is 4k and 2M, then we will return >> after 1M that contains requested 4k is received (but there is no >> guarantee at the moment we won't receive second 2M first, I believe). >> >> Bye, >> Oleg From adilger at sun.com Tue Dec 11 23:47:08 2007 From: adilger at sun.com (Andreas Dilger) Date: Tue, 11 Dec 2007 16:47:08 -0700 Subject: [Lustre-devel] read ahead In-Reply-To: <18270.55813.185376.299242@gargle.gargle.HOWL> References: <475ED623.6090204@sun.com> <18270.55813.185376.299242@gargle.gargle.HOWL> Message-ID: <20071211234708.GT3214@webber.adilger.int> On Dec 11, 2007 21:42 +0300, Nikita Danilov wrote: > Peter Braam writes: > > Can anyone tell me if read ahead in Lustre includes "early return" > > features. I mean that if I read 4K and readahead decides to fetch 1M > > will my request get serviced when the first 4K arrives? Is this important? > > Currently read system call will proceed when the first RPC (including > first 4K page and some number of read-ahead pages) is serviced: > generic_file_read() waits on a page lock, and lock is released by > completion routine (ll_ap_completion()). Another thing worth mentioning here is that if this is the FIRST 4kB read from the file, then only that 4kB will be returned in the RPC, because readahead hasn't done linear vs. random IO detection yet. If it is the second read (and linear) then the client will get the _rest_ of the 1MB and will have to wait for that second RPC to complete. For subsequent reads the readahead will of course prefetch the pages. For random reads the code does understand the difference between e.g. reads of 16 sequential pages (64kB generally) read at non-consecutive offsets and 16 sequential 4kB page reads. The former will NOT start readahead, while the latter does. Two areas where our readahead is lacking are: - strided reads (may turn the above 16 x 4kB reads into a situation where the client will prefetch pages instead of "random" IO, depending on access pattern, and will avoid prefetch of data the client is not expecting to use) - limiting the readahead to the rate that the client is actually consuming it (currently once we detect sequential reads the readahead window grows eventually to the maximum even if this is far more than what the client needs) Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Mark.Seger at hp.com Wed Dec 12 00:57:57 2007 From: Mark.Seger at hp.com (Mark Seger) Date: Tue, 11 Dec 2007 19:57:57 -0500 Subject: [Lustre-devel] read ahead In-Reply-To: <20071211234708.GT3214@webber.adilger.int> References: <475ED623.6090204@sun.com> <18270.55813.185376.299242@gargle.gargle.HOWL> <20071211234708.GT3214@webber.adilger.int> Message-ID: <475F3215.40602@hp.com> I thought it was the 3rd read that triggers readahead. When I track network and lustre I/O while doing 4, 8 and 12K random reads look at the network traffic: #<-----------Network----------><-------Lustre Client------> #netKBi pkt-in netKBo pkt-out Reads KBRead Writes KBWrite 0 9 1 15 1 4 0 0 18 149 16 165 0 0 0 0 57 626 56 639 0 0 0 0 34 382 34 392 2 8 0 0 12 30 5 40 0 0 0 0 0 10 1 23 0 0 0 0 0 8 1 24 0 0 0 0 1 20 1 23 3 12 0 0 1087 758 32 422 0 0 0 0 Since this is a shared network, you're seeing 'noise' on the link on the order of about 10-50MB/sec, but the spike of over 1M is clearly a result due to the readahead. I had also done earlier byte level tests in which reading 8192 bytes didn't do readahead while 8193 did. If I do a 12K and then a 16K random reads and add readahead readhead stats to the output look at the following you can again see 1MB network traffic associated with the 12KB random read, but now we also see 3 lustre cache misses since the readahead occurs on the 3rd page and nothing is in the cache yet. #<-----------Network----------><-------------Lustre Client--------------> #netKBi pkt-in netKBo pkt-out Reads KBRead Writes KBWrite Hits Misses 0 8 1 16 3 12 0 0 0 3 1086 757 31 408 0 0 0 0 0 0 0 7 1 22 0 0 0 0 0 0 0 9 1 26 0 0 0 0 0 0 0 10 2 29 0 0 0 0 0 0 0 8 1 20 0 0 0 0 0 0 0 10 1 21 4 16 0 0 1 3 2159 1478 56 781 0 0 0 0 0 0 By my question is why are we seeing a 2MB readahead (network traffic) when I'm only reading 16KB? Is it that lustre does a 1MB readahead when the 3th page is read and another 1MB when the forth page is read? That doesn't sound right to me. Further, looking at the hits/misses you can also see the first 3 pages are read over the network and the fourth comes out of cache because of the readahead on the 3rd. So again, where is the 2MB coming from? If anyone is interested, these stats come from collectl, which I've mentioned in the past: http://collectl.sourceforge.net/ There is an even more detailed format for readahead stats but I don't think anything else is relevant to this particular situation: # LUSTRE CLIENT SUMMARY: READAHEAD # Reads ReadKB Writes WriteKB Pend Hits Misses NotCon MisWin LckFal Discrd ZFile ZerWin RA2Eof HitMax 4 16 0 0 0 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -mark Andreas Dilger wrote: > On Dec 11, 2007 21:42 +0300, Nikita Danilov wrote: > >> Peter Braam writes: >> > Can anyone tell me if read ahead in Lustre includes "early return" >> > features. I mean that if I read 4K and readahead decides to fetch 1M >> > will my request get serviced when the first 4K arrives? Is this important? >> >> Currently read system call will proceed when the first RPC (including >> first 4K page and some number of read-ahead pages) is serviced: >> generic_file_read() waits on a page lock, and lock is released by >> completion routine (ll_ap_completion()). >> > > Another thing worth mentioning here is that if this is the FIRST 4kB read > from the file, then only that 4kB will be returned in the RPC, because > readahead hasn't done linear vs. random IO detection yet. If it is the > second read (and linear) then the client will get the _rest_ of the 1MB > and will have to wait for that second RPC to complete. For subsequent > reads the readahead will of course prefetch the pages. > > For random reads the code does understand the difference between e.g. > reads of 16 sequential pages (64kB generally) read at non-consecutive > offsets and 16 sequential 4kB page reads. The former will NOT start > readahead, while the latter does. > > Two areas where our readahead is lacking are: > - strided reads (may turn the above 16 x 4kB reads into a situation > where the client will prefetch pages instead of "random" IO, depending > on access pattern, and will avoid prefetch of data the client is not > expecting to use) > - limiting the readahead to the rate that the client is actually consuming > it (currently once we detect sequential reads the readahead window grows > eventually to the maximum even if this is far more than what the client > needs) > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-devel > From Nikita.Danilov at Sun.COM Wed Dec 12 05:53:59 2007 From: Nikita.Danilov at Sun.COM (Nikita Danilov) Date: Wed, 12 Dec 2007 08:53:59 +0300 Subject: [Lustre-devel] read ahead In-Reply-To: <20071211234708.GT3214@webber.adilger.int> References: <475ED623.6090204@sun.com> <18270.55813.185376.299242@gargle.gargle.HOWL> <20071211234708.GT3214@webber.adilger.int> Message-ID: <18271.30583.338003.357861@gargle.gargle.HOWL> Andreas Dilger writes: [...] > > Two areas where our readahead is lacking are: > - strided reads (may turn the above 16 x 4kB reads into a situation > where the client will prefetch pages instead of "random" IO, depending > on access pattern, and will avoid prefetch of data the client is not > expecting to use) > - limiting the readahead to the rate that the client is actually consuming > it (currently once we detect sequential reads the readahead window grows > eventually to the maximum even if this is far more than what the client > needs) I wonder how useful can inter-file read-ahead be. For example, starting an executable almost always incurs a sequence of reads of the shared libraries, compilation re-reads header files in the same order over and over again, etc. > > Cheers, Andreas Nikita. From chas at cmf.nrl.navy.mil Wed Dec 12 15:52:21 2007 From: chas at cmf.nrl.navy.mil (chas williams - CONTRACTOR) Date: Wed, 12 Dec 2007 10:52:21 -0500 Subject: [Lustre-devel] read ahead In-Reply-To: <20071211234708.GT3214@webber.adilger.int> Message-ID: <200712121552.lBCFqLxr003560@cmf.nrl.navy.mil> In message <20071211234708.GT3214 at webber.adilger.int>,Andreas Dilger writes: >For random reads the code does understand the difference between e.g. >reads of 16 sequential pages (64kB generally) read at non-consecutive >offsets and 16 sequential 4kB page reads. The former will NOT start >readahead, while the latter does. what about direct i/o? it looks like doing direct i/o to a file will never trigger readahead. is this intentional? From Oleg.Drokin at Sun.COM Wed Dec 12 16:41:18 2007 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Wed, 12 Dec 2007 11:41:18 -0500 Subject: [Lustre-devel] read ahead In-Reply-To: <200712121552.lBCFqLxr003560@cmf.nrl.navy.mil> References: <200712121552.lBCFqLxr003560@cmf.nrl.navy.mil> Message-ID: <6C95C14E-9F4E-4E3E-A435-5010C2EFC999@sun.com> Hello! On Dec 12, 2007, at 10:52 AM, chas williams - CONTRACTOR wrote: > In message <20071211234708.GT3214 at webber.adilger.int>,Andreas Dilger > writes: >> For random reads the code does understand the difference between e.g. >> reads of 16 sequential pages (64kB generally) read at non-consecutive >> offsets and 16 sequential 4kB page reads. The former will NOT start >> readahead, while the latter does. > what about direct i/o? it looks like doing direct i/o to a file > will never trigger readahead. is this intentional? Yes it is. DIRECT IO by it's nature is "direct", i.e. it gets straight into application buffers, we are not to put any more data there than application said it can accept. And we are not allowed to cache any of that data (or to use any cache to get the data during reads) too. Bye, Oleg From chas at cmf.nrl.navy.mil Thu Dec 13 15:15:53 2007 From: chas at cmf.nrl.navy.mil (chas williams - CONTRACTOR) Date: Thu, 13 Dec 2007 10:15:53 -0500 Subject: [Lustre-devel] read ahead In-Reply-To: <6C95C14E-9F4E-4E3E-A435-5010C2EFC999@sun.com> Message-ID: <200712131515.lBDFFrCa014152@cmf.nrl.navy.mil> In message <6C95C14E-9F4E-4E3E-A435-5010C2EFC999 at sun.com>,Oleg Drokin writes: >Yes it is. >DIRECT IO by it's nature is "direct", i.e. it gets straight into >application buffers, >we are not to put any more data there than application said it can >accept. i agree with this interpretation of O_DIRECT. >And we are not allowed to cache any of that data (or to use any cache >to get >the data during reads) too. this seems a bit stricter than i would expect. while i would expect O_DIRECT to bypass the kernel readahead mechanism, i think that lustre's readahead is outside of the scope of O_DIRECT. From Oleg.Drokin at Sun.COM Thu Dec 13 15:23:22 2007 From: Oleg.Drokin at Sun.COM (Oleg Drokin) Date: Thu, 13 Dec 2007 10:23:22 -0500 Subject: [Lustre-devel] read ahead In-Reply-To: <200712131515.lBDFFrCa014152@cmf.nrl.navy.mil> References: <200712131515.lBDFFrCa014152@cmf.nrl.navy.mil> Message-ID: <6AD2FC07-6589-4194-B92E-9844847AECD3@Sun.COM> Hello! On Dec 13, 2007, at 10:15 AM, chas williams - CONTRACTOR wrote: >> Yes it is. >> DIRECT IO by it's nature is "direct", i.e. it gets straight into >> application buffers, >> we are not to put any more data there than application said it can >> accept. > i agree with this interpretation of O_DIRECT. > >> And we are not allowed to cache any of that data (or to use any cache >> to get >> the data during reads) too. > this seems a bit stricter than i would expect. while i would expect > O_DIRECT to bypass the kernel readahead mechanism, i think that > lustre's > readahead is outside of the scope of O_DIRECT. While technically nothing prevents us from reading some more stuff into client cache (we cannot do any readahead into applications buffers for obvious reasons) with readahead after genuine directio request, we cannot use the cache for directio requests, so if next read request comes in directio mode, we MUST bypass cache and the whole purpose of readahead is therefore defeated. Bye, Oleg From Peter.Braam at Sun.COM Thu Dec 13 19:09:28 2007 From: Peter.Braam at Sun.COM (Peter Braam) Date: Thu, 13 Dec 2007 12:09:28 -0700 Subject: [Lustre-devel] read ahead In-Reply-To: <4760B247.4030505@sun.com> References: <475ED623.6090204@sun.com> <18270.55813.185376.299242@gargle.gargle.HOWL> <20071211234708.GT3214@webber.adilger.int> <475F7759.2080103@sun.com> <47603E32.6070203@sun.com> <4760B247.4030505@sun.com> Message-ID: <47618368.7000202@sun.com> Tom.Wang wrote: > Hi, Peter > > I just talked with Matt about his pCIFS test. In his test, there are 2 > clients, 2 OST, (stripe_count =2, stripe_size=1M). > Each client run 1 thread to read a shared file. Each thread(client) > will only read data of the shared file from 1 OST, so > read request is discontiguous (1M step stride I/O) for each of thread, > which will not trigger read-ahead, because > read-ahead window will be reset once it met discontinuous read > request. So kept read-ahead in the same step > of stride step ((stripe_count -1) * stripe_size) instead of > contiguous could just fix this problem, and it is what we are > working now. > Finally! This makes a lot of sense. > Btw: we also need this in lustre collective read ADIO driver, > since we will also need reorganize the data as 1client-> 1OST > model like pCIFS does. I now see why this stripe behavior with read-ahead is so very important for Lustre in some cases. It would also be important to know that in basic cases we really get as good read performance as write. - Peter - > Another alternative way maybe move read-ahead mechanism to > osc layer, but maybe a bad idea. > Thanks > WangDi > From adilger at sun.com Fri Dec 14 00:21:09 2007 From: adilger at sun.com (Andreas Dilger) Date: Thu, 13 Dec 2007 17:21:09 -0700 Subject: [Lustre-devel] read ahead In-Reply-To: <18271.30583.338003.357861@gargle.gargle.HOWL> References: <475ED623.6090204@sun.com> <18270.55813.185376.299242@gargle.gargle.HOWL> <20071211234708.GT3214@webber.adilger.int> <18271.30583.338003.357861@gargle.gargle.HOWL> Message-ID: <20071214002109.GO3214@webber.adilger.int> On Dec 12, 2007 08:53 +0300, Nikita Danilov wrote: > Andreas Dilger writes: > > Two areas where our readahead is lacking are: > > - strided reads (may turn the above 16 x 4kB reads into a situation > > where the client will prefetch pages instead of "random" IO, depending > > on access pattern, and will avoid prefetch of data the client is not > > expecting to use) > > - limiting the readahead to the rate that the client is actually consuming > > it (currently once we detect sequential reads the readahead window grows > > eventually to the maximum even if this is far more than what the client > > needs) > > I wonder how useful can inter-file read-ahead be. For example, starting > an executable almost always incurs a sequence of reads of the shared > libraries, compilation re-reads header files in the same order over and > over again, etc. Well, we already have a beginning of this kind of operation on the client with client-side metadata statahead. That detects readdir->stat operation and prefetches the MDS attribute data. The next step would be OST statahead, which could be started asynchronously as soon as the LOV EA is returned from the MDS instead of waiting for the userspace process to get to that entry and force the OST stat. OST statahead will not be needed on 1.8 in many cases when size-on-MDS is available (if file is closed) but would still be useful for 1.6 and the case of "impatient user running 'ls -l' in the job output directory while files are being written". The logical extension would be to detect readdir + read (for e.g. updatedb, find ... | xargs grep, etc) type loads and prefetch the file data if it is not too big, or at least just the first block for "file" or similar. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Alexey.Lyashkov at Sun.COM Thu Dec 20 10:20:31 2007 From: Alexey.Lyashkov at Sun.COM (Alex Lyashkov) Date: Thu, 20 Dec 2007 12:20:31 +0200 Subject: [Lustre-devel] read ahead In-Reply-To: <20071214002109.GO3214@webber.adilger.int> References: <475ED623.6090204@sun.com> <18270.55813.185376.299242@gargle.gargle.HOWL> <20071211234708.GT3214@webber.adilger.int> <18271.30583.338003.357861@gargle.gargle.HOWL> <20071214002109.GO3214@webber.adilger.int> Message-ID: <1198146031.3528.24.camel@bear.shadowland> On Thu, 2007-12-13 at 17:21 -0700, Andreas Dilger wrote: > On Dec 12, 2007 08:53 +0300, Nikita Danilov wrote: > > Andreas Dilger writes: > > > Two areas where our readahead is lacking are: > > > - strided reads (may turn the above 16 x 4kB reads into a situation > > > where the client will prefetch pages instead of "random" IO, depending > > > on access pattern, and will avoid prefetch of data the client is not > > > expecting to use) > > > - limiting the readahead to the rate that the client is actually consuming > > > it (currently once we detect sequential reads the readahead window grows > > > eventually to the maximum even if this is far more than what the client > > > needs) > > > > I wonder how useful can inter-file read-ahead be. For example, starting > > an executable almost always incurs a sequence of reads of the shared > > libraries, compilation re-reads header files in the same order over and > > over again, etc. > > Well, we already have a beginning of this kind of operation on the client > with client-side metadata statahead. That detects readdir->stat operation > and prefetches the MDS attribute data. I'm not sure this was good idea. stat readahead is slow for single cpu box, and slow in testing with local devices (very low network latency) - and can add noticeable speedup only with high latency network links, because can send many stats requests when main thread block in waiting one answer. -- Alex Lyashkov Lustre Group, Sun Microsystems From Peter.Braam at Sun.COM Thu Dec 20 20:36:14 2007 From: Peter.Braam at Sun.COM (Peter Braam) Date: Thu, 20 Dec 2007 13:36:14 -0700 Subject: [Lustre-devel] another use of MD request scheduling Message-ID: <476AD23E.2080107@sun.com> The metadata request scheduler could pre-fetch data from disk that will be required for executing the operation. This has been found to be very beneficial in many other applications. - peter - From adilger at sun.com Thu Dec 20 20:44:06 2007 From: adilger at sun.com (Andreas Dilger) Date: Thu, 20 Dec 2007 13:44:06 -0700 Subject: [Lustre-devel] read ahead In-Reply-To: <1198146031.3528.24.camel@bear.shadowland> References: <475ED623.6090204@sun.com> <18270.55813.185376.299242@gargle.gargle.HOWL> <20071211234708.GT3214@webber.adilger.int> <18271.30583.338003.357861@gargle.gargle.HOWL> <20071214002109.GO3214@webber.adilger.int> <1198146031.3528.24.camel@bear.shadowland> Message-ID: <20071220204406.GY3214@webber.adilger.int> On Dec 20, 2007 12:20 +0200, Alex Lyashkov wrote: > On Thu, 2007-12-13 at 17:21 -0700, Andreas Dilger wrote: > > Well, we already have a beginning of this kind of operation on the client > > with client-side metadata statahead. That detects readdir->stat operation > > and prefetches the MDS attribute data. > > I'm not sure this was good idea. stat readahead is slow for single cpu > box, and slow in testing with local devices (very low network latency) - > and can add noticeable speedup only with high latency network links, > because can send many stats requests when main thread block in waiting > one answer. This isn't the common configuration that we run in, however. Most Lustre setups today (and even laptops) have multiple CPUs, and clients are generally remote instead of local. In such testing the statahead is about 2x faster than without, and I think it could be even faster if we also had OST statahead. The current statahead is limited by the fact we are only hiding the client->MDS latency (1 of 2 synchronous RPCs) and not the client->OSTs latency (which happen concurrently for N stripes, but will be the slowest of all OST getattrs). So our maximum "ls -l" improvement is currently 2x until SOM (for closed files) or OST statahead is implemented. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From Peter.Braam at Sun.COM Thu Dec 20 20:52:28 2007 From: Peter.Braam at Sun.COM (Peter Braam) Date: Thu, 20 Dec 2007 13:52:28 -0700 Subject: [Lustre-devel] SNS and existing Lustre striping. In-Reply-To: <200712202333.57536.alexander.zarochentsev@sun.com> References: <200712202333.57536.alexander.zarochentsev@sun.com> Message-ID: <476AD60C.1080803@sun.com> I was hoping for 1. to keep things simple. - Peter - Alexander Zarochentsev wrote: > Hello, > > Coming SNS feature somehow duplicates and extents existing Lustre > striping functionality. How will they (SNS and striping) coexist > in the future versions of Lustre? > > I see 3 possible variants: > > 1. SNS replaces striping everywhere. > 2. one file can be SNS striped object or lustre striping object. > 3. one file can be SNS on top of Lustre-striping or Lustre-striping on > SNS and, probably, one part of file can be SNS and another part is > Lustre-striped. > > Does #3 makes any sense? > > I would prefer to assume that correct answer is (2) -- there are SNS > files and Lustre-striped files, because Lustre-striped files probably > have less overhead. > > Thanks, > Zam. > From adilger at sun.com Thu Dec 20 22:46:09 2007 From: adilger at sun.com (Andreas Dilger) Date: Thu, 20 Dec 2007 15:46:09 -0700 Subject: [Lustre-devel] SNS and existing Lustre striping. In-Reply-To: <476AD60C.1080803@sun.com> References: <200712202333.57536.alexander.zarochentsev@sun.com> <476AD60C.1080803@sun.com> Message-ID: <20071220224609.GD3214@webber.adilger.int> On Dec 20, 2007 13:52 -0700, Peter Braam wrote: > I was hoping for 1. to keep things simple. Presumably, an SNS layer would replace (or just extend?) the existing LOV functionality on the clients. We would definitely want to allow existing RAID 0 striped files to be usable by having the SNS-aware client understand the old LOV EA format, even if all new files are created with an enhanced SNS EA format. One feature that is quite desirable with SNS is to be able to add and remove full mirrors of the file (as opposed to restriping to change the layout in more complex ways) in an efficient manner. Since this functionality is already required to be able to reconstruct a missing mirror copy of a file it seems relatively straightforward to create a "ghost" mirror copy and then have the normal resync mechanism copy it. More complex layout changes (e.g. RAID0 striping to RAID5/6) can be done via the planned normal file migration/restripe mechanisms, and it should be up to the administrator to do this, as for some cases (e.g. scratch filesystem) it might not be worth the extra IO to restripe all the files and instead leave it up to normal file aging/purge to catch most of the files, and then have a later pass later to handle files that have a longer lifetimes. > Alexander Zarochentsev wrote: > > Coming SNS feature somehow duplicates and extents existing Lustre > > striping functionality. How will they (SNS and striping) coexist > > in the future versions of Lustre? > > > > I see 3 possible variants: > > > > 1. SNS replaces striping everywhere. > > 2. one file can be SNS striped object or lustre striping object. > > 3. one file can be SNS on top of Lustre-striping or Lustre-striping on > > SNS and, probably, one part of file can be SNS and another part is > > Lustre-striped. > > > > Does #3 makes any sense? > > > > I would prefer to assume that correct answer is (2) -- there are SNS > > files and Lustre-striped files, because Lustre-striped files probably > > have less overhead. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.