<div>I think we are in agreement here, but my point is adding a read cache to Lustre (which is feature I rely heavily on) isn't the fundmental problem, it just exposed it. The fundamental problem is with ext[34] performance on larger disks.</div>
<div></div>
<div>Jeremy<br><br></div>
<div class="gmail_quote">On Wed, Apr 20, 2011 at 8:57 AM, Kevin Van Maren <span dir="ltr"><<a href="mailto:kevin.van.maren@oracle.com">kevin.van.maren@oracle.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote">Yes, difficulty finding free disk space can also be a problem, but I<br>could not recall big changes in<br>
how that worked since 1.6, other than memory pressure from the read<br>cache pushing out the bitmaps.<br>See <a href="http://jira.whamcloud.com/browse/LU-15" target="_blank">http://jira.whamcloud.com/browse/LU-15</a><br><font color="#888888"><br>
Kevin<br></font>
<div>
<div></div>
<div class="h5"><br><br>James Rose wrote:<br>> Hi Kevin,<br>><br>> Thanks for the suggestion. I will try this out.<br>><br>> For the moment it seems that it may be disk space related. I have<br>> removed some data from the file system. Performance returned to where I<br>
> would expect it to be as space freed up (currently at 83% full). Since<br>> free space I have seen two messages on an OSS where the number of<br>> threads is tuned to the amount of RAM in the host and six on an OSS that<br>
> has the number of threads set higher than it should. This is a much<br>> better situation than the steady stream I was experiencing last night.<br>> Maybe disabling the read cache will remove the last few.<br>
><br>> I am still very curious what the rapid small reads seen when writing are<br>> as this showed up while mounted ldiskfs so not doing regular lustre<br>> operations at all.<br>><br>> Thanks again for your help,<br>
><br>> James.<br>><br>><br>> On Wed, 2011-04-20 at 08:48 -0300, Kevin Van Maren wrote:<br>><br>>> First guess is the increased memory pressure caused by the Lustre 1.8<br>>> read cache. Many times "slow" messages are caused by memory<br>
>> allocatons taking a long time.<br>>><br>>> You could try disabling the read cache and see if that clears up the<br>>> slow messages.<br>>><br>>> Kevin<br>>><br>>><br>>> On Apr 20, 2011, at 4:29 AM, James Rose <<a href="mailto:James.Rose@framestore.com">James.Rose@framestore.com</a>><br>
>> wrote:<br>>><br>>><br>>>> Hi<br>>>><br>>>> We have been experiencing degraded performance for a few days on a<br>>>> fresh install of lustre 1.8.5 (on RHEL5 using sun ext4 rpms). The<br>
>>> initial bulk load of the data will be fine but once in use for a<br>>>> while writes become very slow to individual ost. This will block io<br>>>> for a few minutes and then carry on as normal. The slow writes will<br>
>>> then move to another ost. This can be seen in iostat and many slow<br>>>> IO messages will be seen in the logs (example included)<br>>>><br>>>> The osts are between 87 90 % full. Not ideal but has not caused any<br>
>>> issues running 1.6.7.2 on the same hardware.<br>>>><br>>>> The osts are RAID6 on external raid chassis (Infortrend). Each ost<br>>>> is 5.4T (small). The server is Dual AMD (4 cores). 16G Ram. Qlogic<br>
>>> FC HBA.<br>>>><br>>>> I mounted the osts as ldiskfs and tried a few write tests. These<br>>>> also show the same behaviour.<br>>>><br>>>> While the write operation is blocked there will be hundreds of read<br>
>>> tps and a very small kb/s read from the raid but now writes. As<br>>>> soon as this completes writes will go through at a more expected<br>>>> speed.<br>>>><br>>>> Any idea what is going on?<br>
>>><br>>>> Many thanks<br>>>><br>>>> James.<br>>>><br>>>> Example error messages:<br>>>><br>>>> Apr 20 04:53:04 oss5r-mgmt kernel: LustreError: dumping log to /tmp/<br>
>>> lustre-log.1303271584.3935<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow quota<br>>>> init 286s due to heavy IO load<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow journal<br>
>>> start 39s due to heavy IO load<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 39 previous<br>>>> similar messages<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow<br>
>>> brw_start 39s due to heavy IO load<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 38 previous<br>>>> similar messages<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow journal<br>
>>> start 133s due to heavy IO load<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 44 previous<br>>>> similar messages<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow<br>
>>> brw_start 133s due to heavy IO load<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 44 previous<br>>>> similar messages<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow journal<br>
>>> start 236s due to heavy IO load<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow i_mutex<br>>>> 40s due to heavy IO load<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 2 previous<br>
>>> similar messages<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 6 previous<br>>>> similar messages<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow i_mutex<br>
>>> 277s due to heavy IO load<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow<br>>>> direct_io 286s due to heavy IO load<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 3 previous<br>
>>> similar messages<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow journal<br>>>> start 285s due to heavy IO load<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 1 previous<br>
>>> similar message<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow<br>>>> commitrw commit 285s due to heavy IO load<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: Skipped 1 previous<br>
>>> similar message<br>>>> Apr 20 04:53:40 oss5r-mgmt kernel: Lustre: rho-OST0012: slow parent<br>>>> lock 236s due to heavy IO load<br>>>><br>>>><br>>>> _______________________________________________<br>
>>> Lustre-discuss mailing list<br>>>> <a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>>>> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
>>><br>>> _______________________________________________<br>>> Lustre-discuss mailing list<br>>> <a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>>> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
>><br>><br>><br>><br><br>_______________________________________________<br>Lustre-discuss mailing list<br><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
</div></div></blockquote></div><br>