<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p>A guess for you to consider:<br>
</p>
<p><br>
</p>
<p>A very common cause of ksoftirqd load is a hypervisor putting memory pressure on a VM. At least VMWare, and I think KVM and others, use IRQs to implement some of their memory management and it can show up like this.<br>
</p>
<p><br>
</p>
<p>That would of course mean it's not really the ptlrpc module, I'm not sure how carefully you verified that it is causing this. (Obviously your 'remove it, check, add it, check' method is sound, but if you just checked once or twice, you may have been wrong
through bad luck or you could've been right at your limit of available memory.)<br>
</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Dilger, Andreas <andreas.dilger@intel.com><br>
<b>Sent:</b> Wednesday, September 27, 2017 11:50:03 AM<br>
<b>To:</b> Hans Henrik Happe<br>
<b>Cc:</b> Shehata, Amir; lustre-discuss; Olaf Weber<br>
<b>Subject:</b> Re: [lustre-discuss] 2.10.0 CentOS6.9 ksoftirqd CPU load</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">On Sep 26, 2017, at 01:10, Hans Henrik Happe <happe@nbi.dk> wrote:<br>
> <br>
> Hi,<br>
> <br>
> Did anyone else experience CPU load from ksoftirqd after 'modprobe<br>
> lustre'? On an otherwise idle node I see:<br>
> <br>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
> 9 root 20 0 0 0 0 S 28.5 0.0 2:05.58 ksoftirqd/1<br>
> <br>
> <br>
> 57 root 20 0 0 0 0 R 23.9 0.0 2:22.91 ksoftirqd/13<br>
> <br>
> The sum of those two is about 50% CPU.<br>
> <br>
> I have narrowed it down to the ptlrpc module. When I remove that, it stops.<br>
> <br>
> I also tested the 2.10.1-RC1, which is the same.<br>
<br>
If you can run "echo l > /proc/sysrq-trigger" it will report the processes<br>
that are currently running on the CPUs of your system to the console (and<br>
also /var/log/messages, if it can write everything in time).<br>
<br>
You might need to do this several times to get a representative sample of<br>
the ksoftirqd process stacks to see what they are doing that is consuming<br>
so much CPU.<br>
<br>
Alternately, "echo t > /proc/sysrq-trigger" will report the stacks of all<br>
processes to the console (and /v/l/m), but there will be a lot of them,<br>
and no better chance that it catches what ksoftirqd is doing 25% of the time.<br>
<br>
Cheers, Andreas<br>
--<br>
Andreas Dilger<br>
Lustre Principal Architect<br>
Intel Corporation<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</div>
</span></font>
</body>
</html>