Is this filesystem nearly full? Fragmentation can decrease back end performance.<br><br>Also check the disks stats on the DDN, maybe you have a slow disk in one of your tiers.<br><br>Wojciech<br><br><div class="gmail_quote">
On 18 October 2010 18:49, Peter Kjellstrom <span dir="ltr"><<a href="mailto:cap@nsc.liu.se">cap@nsc.liu.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="im">On Monday 18 October 2010, John White wrote:<br>
> Hello Folks,<br>
> A while back (say 3 weeks ago) we started noticing extremely high loads<br>
> (load avg around 300 at times) on our OSSs when in production and serving<br>
> IO. This cluster was, at the time, on 1.8.2 (we have since upgraded to<br>
> 1.8.4 but the problem remains). The load increases fairly predictably as<br>
> clients generate IO but even 2 clients can produce a load avg above 5.00.<br>
<br>
</div>Does this impact performance or does it only show up as an unexpectedly high<br>
number on the OSSes?<br>
<font color="#888888"><br>
/Peter<br>
</font><div><div></div><div class="h5"><br>
> An identical file system of ours does not exhibit this behavior (sticks<br>
> below load avg 1.00 under even the heaviest IO load). I've looked around<br>
> bugzilla and haven't found anything. We've disabled heartbeat on the<br>
> off-chance that was generating the load (it's not), we've attempted using a<br>
> different client transport (o2ib->tcp), this did not solve the issue.<br>
> There doesn't appear to be any specific non-kernel thread causing the<br>
> high-load. The only info in dmesg/syslog pertains to sporadic client<br>
> evictions or sporadic slow setattr due to heavy IO load (we've since tuned<br>
> the number of OST threads). We're basically out of ideas to try.<br>
><br>
> As reference, this is a 1 MDS/4 OSS cluster backed by a DDN 9900 couplet<br>
> (15 tiers, 1:1 lun mapping) running the <a href="http://lustre.org" target="_blank">lustre.org</a> rpm build kernel for<br>
> 1.8.4. The MDS/OSSs are Dell R710s and the MDT is a Dell MD1000. Is this<br>
> a common problem or should a bug be filed? Any info available upon<br>
> request. Thanks for your time. ----------------<br>
> John White<br>
> High Performance Computing Services (HPCS)<br>
> (510) 486-7307<br>
> One Cyclotron Rd, MS: 50B-3209C<br>
> Lawrence Berkeley National Lab<br>
> Berkeley, CA 94720<br>
</div></div><br>_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>Wojciech Turek<br><br>Senior System Architect<br><br>High Performance Computing Service<br>University of Cambridge<br>Email: <a href="mailto:wjt27@cam.ac.uk" target="_blank">wjt27@cam.ac.uk</a><br>
Tel: (+)44 1223 763517 <br>