<br><font size=2 face="sans-serif">After doing some more digging it looks
as though a bug was reported on this in 2007. </font>
<br>
<br><a href="https://bugzilla.lustre.org/show_bug.cgi?id=12739"><font size=2 face="sans-serif">https://bugzilla.lustre.org/show_bug.cgi?id=12739</font></a>
<br>
<br><font size=2 face="sans-serif">We have loaded the patch for lustre
attached to this bug, however when running the set_param command I am getting
the following error. </font>
<br>
<br><font size=2 face="sans-serif">lctl set_param llite*.*.stat_blksize=4096</font>
<br><font size=2 face="sans-serif">error: set_param: /proc/{fs,sys}/{lnet,lustre}/llite/lustre*/stat_blksize:
No such process</font>
<br>
<br><font size=2 face="sans-serif">Is this patch still valid for 2.6.9-78.0.22.EL_lustre.1.6.7.2smp</font>
<br>
<br><font size=2 face="sans-serif">Thanks again</font>
<br><font size=2 face="sans-serif"><br>
Rocky<br>
</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">From:</font>
<td><font size=1 face="sans-serif">Andreas Dilger <andreas.dilger@oracle.com></font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">To:</font>
<td><font size=1 face="sans-serif">Ronald K Long <rklong@usgs.gov></font>
<tr>
<td valign=top><font size=1 color=#5f5f5f face="sans-serif">Cc:</font>
<td><font size=1 face="sans-serif">"Brian J. Murrell" <Brian.Murrell@Sun.COM>,
lustre-discuss@lists.lustre.org, lustre-discuss-bounces@lists.lustre.org</font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Date:</font>
<td><font size=1 face="sans-serif">04/14/2010 02:13 PM</font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Subject:</font>
<td><font size=1 face="sans-serif">Re: [Lustre-discuss] fseeks on lustre</font></table>
<br>
<hr noshade>
<br>
<br>
<br><tt><font size=2>On 2010-04-14, at 11:08, Ronald K Long wrote:<br>
> We've narrowed down the problem quite a bit.<br>
><br>
> The problematic code snippet is not actually doing any reads or <br>
> writes;<br>
> it's just doing a massive number of fseek() operations within a couple<br>
> of nested loops. (Note: The production code is doing some I/O,
but <br>
> this<br>
> snippet was narrowed down to the bare minimum example that exhibited
<br>
> the<br>
> problem, which was how we discovered that fseek was the culprit.)<br>
><br>
> The issue appears to be the behavior of the glibc implementation of<br>
> fseek(). Apparently, a call to fseek() on a buffered file stream
<br>
> causes<br>
> glibc to flush the stream (regardless of whether a flush is actually<br>
> needed). If we modify the snippet to call setvbuf() and disable<br>
> buffering on the file stream before any of the fseek() calls, then
it<br>
> finishes more or less instantly, as you would expect.<br>
<br>
I'd encourage you to file a bug (preferably with a patch) against <br>
glibc to fix this. I've had reasonable success in getting problems
<br>
like this fixed upstream.<br>
<br>
> The problem is that this offending code is actually buried deep <br>
> within a<br>
> COTS library that we're using to do image processing (the Hierarchical<br>
> Data Format (HDF) library). While we do have access to the source
<br>
> code<br>
> for this library and could conceivably modify it, this is a large
and<br>
> complex library, and a change of this nature would require us to do
a<br>
> large amount of regression testing to ensure that nothing was broken.<br>
><br>
> So at the end of the day this is really not a "Lustre problem"
per se,<br>
> though we would still be interested in any suggestions as to how we
<br>
> can<br>
> minimize the effects of this glibc "flush penalty". This
penalty is <br>
> not<br>
> particularly onerous when reading and writing to local disk, but is<br>
> obviously more of an issue with a distributed filesystem.<br>
<br>
Similarly, HDF + Lustre usage is very common, and I expect that the <br>
HDF developers would be interested to fix this if possible.<br>
<br>
> On Wed, 2010-04-14 at 07:08 -0500, Ronald K Long wrote:<br>
> ><br>
> > Andreas - Here is a snipet of the strace output.<br>
> ><br>
> > read(3, <br>
> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0<br>
> > \0\0"..., 2097152) = 2097152<br>
><br>
> As Andreas suspected, your application is doing 2MB reads every time.<br>
> Does it really need 2MB of data on each read? If not, can you
fix <br>
> your<br>
> application to only read as much data as it actually wants?<br>
<br>
<br>
Cheers, Andreas<br>
--<br>
Andreas Dilger<br>
Principal Engineer, Lustre Group<br>
Oracle Corporation Canada Inc.<br>
<br>
</font></tt>
<br>
<br>