<br><font size=2 face="sans-serif">Andreas - Here is a snipet of the strace
output. </font>
<br>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
2097152) = 2097152</font>
<br><font size=2 face="Courier New">_llseek(3, 0, [0], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
2097152) = 2097152</font>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
2097152) = 2097152</font>
<br><font size=2 face="Courier New">_llseek(3, 0, [0], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
2097152) = 2097152</font>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
2097152) = 2097152</font>
<br><font size=2 face="Courier New">_llseek(3, 0, [0], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
2097152) = 2097152</font>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
2097152) = 2097152</font>
<br><font size=2 face="Courier New">_llseek(3, 0, [0], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
2097152) = 2097152</font>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
2097152) = 2097152</font>
<br><font size=2 face="Courier New">_llseek(3, 0, [0], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
2097152) = 2097152</font>
<br><font size=2 face="Courier New">_llseek(3, 2097152, [2097152], SEEK_SET)
= 0</font>
<br><font size=2 face="Courier New">read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
2097152) = 2097152</font>
<br><font size=2 face="Courier New">_llseek(3, 0, [0], SEEK_SET)
= 0</font>
<br><font size=2 face="sans-serif"><br>
Thanks again</font>
<br>
<br><font size=2 face="sans-serif">Rocky <br>
<br>
</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">From:</font>
<td><font size=1 face="sans-serif">Andreas Dilger <andreas.dilger@oracle.com></font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">To:</font>
<td><font size=1 face="sans-serif">Ronald K Long <rklong@usgs.gov></font>
<tr>
<td valign=top><font size=1 color=#5f5f5f face="sans-serif">Cc:</font>
<td><font size=1 face="sans-serif">lustre-discuss@lists.lustre.org</font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Date:</font>
<td><font size=1 face="sans-serif">04/13/2010 11:50 PM</font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Subject:</font>
<td><font size=1 face="sans-serif">Re: [Lustre-discuss] fseeks on lustre</font></table>
<br>
<hr noshade>
<br>
<br>
<br><tt><font size=2>On 2010-04-13, at 07:59, Ronald K Long wrote:<br>
> We are doing SEEK_SET<br>
><br>
> fseek(fp,offset[i],SEEK_SET<br>
><br>
> We were running into this same issue on our san file system until
we <br>
> set the dma_cache_read_ahead to match our buffer size of 256k. Just
<br>
> wondering if there is away to set that within lustre. We are
<br>
> running 1.8 on the MDS and OSS and the clients running the fseek are
<br>
> are running 1.6.<br>
<br>
Sorry, I didn't read enough into your question. When you said <br>
"opening a large file and doing fseek()" I thought that was the
only <br>
thing you are doing, but really you are doing IOs after the fseek() <br>
that is presumably what is taking a long time.<br>
<br>
It's true that if you are doing random reads it may cause sub-optimal <br>
performance because the client cannot do readahead to mitigate the <br>
network/disk latency.<br>
<br>
One problem that we saw some time ago at another customer was that <br>
their application doing random IO was getting the "IO blocksize"
from <br>
the file via {f,}stat() and reading from the file in chunks of <br>
st_blksize. Since Lustre returns st_blksize = 2MB, then the <br>
application wanting 4kB chunks of random data from the file was <br>
seeking and reading 2MB of extra data for each seek.<br>
<br>
It would be worthwhile to strace your application to see if it is <br>
doing the same thing.<br>
<br>
> Andreas Dilger <andreas.dilger@oracle.com> wrote:<br>
>> On 2010-04-07, at 14:09, Ronald K Long wrote:<br>
>> > I am having an issue with our lustre file system. In
our current<br>
>> > environment on a san file system opening a large file and
doing<br>
>> > fseeks completes in under 2 seconds. Running that same
routine on<br>
>> > our lustre file system the routine actually never finishes.<br>
>><br>
>> Doing fseek() itself is only a client-side operation, so it should<br>
>> have no performance impact, UNLESS you are doing SEEK_END, which<br>
>> requires that the actual file size be computed on the client.
That<br>
>> causes lock revocation from all of the clients and is an expensive<br>
>> operation. Using SEEK_CUR or SEEK_SET has no cost at all.<br>
>><br>
>> > Are there any tunable parameter in lustre that can alleviate
this<br>
>> > problem?<br>
>><br>
>> It depends on what the problem really is.<br>
<br>
<br>
Cheers, Andreas<br>
--<br>
Andreas Dilger<br>
Principal Engineer, Lustre Group<br>
Oracle Corporation Canada Inc.<br>
<br>
</font></tt>
<br>
<br>