Its an issue with ext[34]. Take a look at <a href="http://jira.whamcloud.com/browse/LU-15">http://jira.whamcloud.com/browse/LU-15</a> for more details.<br><br>Jeremy<br><br><div class="gmail_quote">On Wed, Apr 20, 2011 at 8:28 AM, James Rose <span dir="ltr"><<a href="mailto:James.Rose@framestore.com">James.Rose@framestore.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Hi Kevin,<br>
<br>
Thanks for the suggestion. I will try this out.<br>
<br>
For the moment it seems that it may be disk space related. I have<br>
removed some data from the file system. Performance returned to where I<br>
would expect it to be as space freed up (currently at 83% full). Since<br>
free space I have seen two messages on an OSS where the number of<br>
threads is tuned to the amount of RAM in the host and six on an OSS that<br>
has the number of threads set higher than it should. This is a much<br>
better situation than the steady stream I was experiencing last night.<br>
Maybe disabling the read cache will remove the last few.<br>
<br>
I am still very curious what the rapid small reads seen when writing are<br>
as this showed up while mounted ldiskfs so not doing regular lustre<br>
operations at all.<br>
<br>
Thanks again for your help,<br>
<font color="#888888"><br>
James.<br>
</font><div><div></div><div class="h5"><br>
<br>
On Wed, 2011-04-20 at 08:48 -0300, Kevin Van Maren wrote:<br>
> First guess is the increased memory pressure caused by the Lustre 1.8<br>
> read cache. Many times "slow" messages are caused by memory<br>
> allocatons taking a long time.<br>
><br>
> You could try disabling the read cache and see if that clears up the<br>
> slow messages.<br>
><br>
> Kevin<br>
><br>
><br>
> On Apr 20, 2011, at 4:29 AM, James Rose <<a href="mailto:James.Rose@framestore.com">James.Rose@framestore.com</a>><br>
> wrote:<br>
><br>
> > Hi<br>
> ><br>
> > We have been experiencing degraded performance for a few days on a<br>
> > fresh install of lustre 1.8.5 (on RHEL5 using sun ext4 rpms). The<br>
> > initial bulk load of the data will be fine but once in use for a<br>
> > while writes become very slow to individual ost. This will block io<br>
> > for a few minutes and then carry on as normal. The slow writes will<br>
> > then move to another ost. This can be seen in iostat and many slow<br>
> > IO messages will be seen in the logs (example included)<br>
> ><br>
> > The osts are between 87 90 % full. Not ideal but has not caused any<br>
> > issues running 1.6.7.2 on the same hardware.<br>
> ><br>
> > The osts are RAID6 on external raid chassis (Infortrend). Each ost<br>
> > is 5.4T (small). The server is Dual AMD (4 cores). 16G Ram. Qlogic<br>
> > FC HBA.<br>
> ><br>
> > I mounted the osts as ldiskfs and tried a few write tests. These<br>
> > also show the same behaviour.<br>
> ><br>
> > While the write operation is blocked there will be hundreds of read<br>
> > tps and a very small kb/s read from the raid but now writes. As<br>
> > soon as this completes writes will go through at a more expected<br>
> > speed.<br>
> ><br>
> > Any idea what is going on?<br>
> ><br>
> > Many thanks<br>
> ><br>
> > James.<br>
> ><br>
> > Example error messages:<br>
> ><br>
> > Apr 20 04:53:04 oss5r-mgmt kernel: LustreError: dumping log to /tmp/<br>
> > lustre-log.1303271584.3935<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow quota<br>
> > init 286s due to heavy IO load<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow journal<br>
> > start 39s due to heavy IO load<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 39 previous<br>
> > similar messages<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow<br>
> > brw_start 39s due to heavy IO load<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 38 previous<br>
> > similar messages<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow journal<br>
> > start 133s due to heavy IO load<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 44 previous<br>
> > similar messages<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow<br>
> > brw_start 133s due to heavy IO load<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 44 previous<br>
> > similar messages<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow journal<br>
> > start 236s due to heavy IO load<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow i_mutex<br>
> > 40s due to heavy IO load<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 2 previous<br>
> > similar messages<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 6 previous<br>
> > similar messages<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow i_mutex<br>
> > 277s due to heavy IO load<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow<br>
> > direct_io 286s due to heavy IO load<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 3 previous<br>
> > similar messages<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow journal<br>
> > start 285s due to heavy IO load<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 1 previous<br>
> > similar message<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow<br>
> > commitrw commit 285s due to heavy IO load<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 1 previous<br>
> > similar message<br>
> > Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow parent<br>
> > lock 236s due to heavy IO load<br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Lustre-discuss mailing list<br>
> > <a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
> > <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
> _______________________________________________<br>
> Lustre-discuss mailing list<br>
> <a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br>
<br>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
</div></div></blockquote></div><br>