<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Louis,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If you don’t have any particular monitoring on the servers (Prometheus, ganglia, etc..) you could also use sar (sysstat) or a similar tool to confirm the CPU waits for IO. Also the device saturation on sar or with iostat. For instance:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:Courier">avg-cpu: %user %nice %system %iowait %steal %idle<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier"> 0.19 0.00 6.09 0.10 0.06 93.55<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">sda 0.00 1.20 0.20 0.60 0.00 0.01 20.00 0.00 0.75 1.00 0.67 0.75 0.06<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">sdb 0.00 136.80 2.80 96.60 0.81 9.21 206.42 0.19 1.91 26.29 1.20 0.55 5.46<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Courier">sdc 0.00 144.20 58.80 128.00 2.34 16.82 210.08 0.24 1.31 2.68 0.68 0.66 12.40</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Then if you enable lustre job stats you can check on that specific device which job is doing most IO. Last but not least you could also parse which specific NID is doing the intensive IO on that OST (/proc/fs/lustre/obdfilter/<fs>-OST0007/exports/*/stats).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">Diego</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Louis Allen <louisallen@live.co.uk><br>
<b>Date: </b>Tuesday, 29 October 2019 at 17:43<br>
<b>To: </b>"Oral, H." <oralhs@ornl.gov>, "Carlson, Timothy S" <Timothy.Carlson@pnnl.gov>, "lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] [EXTERNAL] Re: Lustre Timeouts/Filesystem Hanging<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt;background:white"><span style="color:#212121">Thanks, will take a look.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;background:white"><span style="color:#212121"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;background:white"><span style="color:#212121">Any other areas i should be looking? Should i be applying any Lustre tuning? <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;background:white"><span style="color:#212121"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;background:white"><span style="color:#212121">Thanks<o:p></o:p></span></p>
<div id="ms-outlook-mobile-signature">
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt">Get <a href="https://aka.ms/ghei36">
Outlook for Android</a><o:p></o:p></p>
</div>
</div>
<div class="MsoNormal" align="center" style="margin-left:36.0pt;text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="color:black">From:</span></b><span style="color:black"> Oral, H. <oralhs@ornl.gov><br>
<b>Sent:</b> Monday, October 28, 2019 7:06:41 PM<br>
<b>To:</b> Louis Allen <louisallen@live.co.uk>; Carlson, Timothy S <Timothy.Carlson@pnnl.gov>; lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> Re: [EXTERNAL] Re: [lustre-discuss] Lustre Timeouts/Filesystem Hanging</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
For inspecting client side I/O, you can use Darshan. <br>
<br>
Thanks,<br>
<br>
Sarp<br>
<br>
-- <br>
Sarp Oral, PhD<br>
<br>
National Center for Computational Sciences<br>
Oak Ridge National Laboratory<br>
oralhs@ornl.gov<br>
865-574-2173 <br>
<br>
<br>
On 10/28/19, 1:58 PM, "lustre-discuss on behalf of Louis Allen" <lustre-discuss-bounces@lists.lustre.org on behalf of louisallen@live.co.uk> wrote:<br>
<br>
<br>
Thanks for the reply, Tim.<br>
<br>
<br>
Are there any tools I can use to see if that is the cause?<br>
<br>
<br>
Could any tuning possibly help the situation?<br>
<br>
<br>
Thanks<br>
<br>
<br>
<br>
<br>
<br>
________________________________________<br>
From: Carlson, Timothy S <Timothy.Carlson@pnnl.gov><br>
Sent: Monday, 28 October 2019, 17:24<br>
To: Louis Allen; lustre-discuss@lists.lustre.org<br>
Subject: RE: Lustre Timeouts/Filesystem Hanging<br>
<br>
<br>
In my experience, this is almost always related to some code doing really bad I/O. Let’s say you have a 1000 rank MPI code doing open/read 4k/close on a few specific files on that OST. That will make for a bad day.<br>
<br>
The other place you can see this, and this isn’t your case, is when ZFS refuses to give up on a disk that is failing and your overall I/O suffers from ZFS continuing to try to read from a disk that it should just kick out<br>
<br>
Tim<br>
<br>
<br>
From: lustre-discuss <lustre-discuss-bounces@lists.lustre.org><br>
On Behalf Of Louis Allen<br>
Sent: Monday, October 28, 2019 10:16 AM<br>
To: lustre-discuss@lists.lustre.org<br>
Subject: [lustre-discuss] Lustre Timeouts/Filesystem Hanging<br>
<br>
<br>
<br>
Hello,<br>
<br>
<br>
<br>
Lustre (2.12) seem to be hanging quite frequently (5+ times a day) for us and one of the OSS servers (out of 4) is reporting an extremely high load average (150+) but the CPU usage of that server<br>
is actually very low - so it must be related to something else - possibly CPU_IO_WAIT.<br>
<br>
<br>
<br>
The OSS server we are seeing the high load averages we can also see multiple LustreError messages in /var/log/messages:<br>
<br>
<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: LNet: Service thread pid 2403 was inactive for 200.08s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace<br>
for debugging purposes:<br>
Oct 28 11:22:23 pazlustreoss001 kernel: LNet: Skipped 4 previous similar messages<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: Pid: 2403, comm: ll_ost00_068 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Sun May 26 21:48:35 UTC 2019<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: Call Trace:<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc03747c5>] jbd2_log_wait_commit+0xc5/0x140 [jbd2]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0375e52>] jbd2_complete_transaction+0x52/0xa0 [jbd2]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0732da2>] ldiskfs_sync_file+0x2e2/0x320 [ldiskfs]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffa52760b0>] vfs_fsync_range+0x20/0x30<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0c8b651>] osd_object_sync+0xb1/0x160 [osd_ldiskfs]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0ab48a7>] tgt_sync+0xb7/0x270 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0dc3731>] ofd_sync_hdl+0x111/0x530 [ofd]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0aba1da>] tgt_request_handle+0xaea/0x1580 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0a5f80b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0a6313c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffa50c1c71>] kthread+0xd1/0xe0<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffa5775c37>] ret_from_fork_nospec_end+0x0/0x39<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffffffffff>] 0xffffffffffffffff<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: LustreError: dumping log to /tmp/lustre-log.1572261743.2403<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: Pid: 2292, comm: ll_ost03_043 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Sun May 26 21:48:35 UTC 2019<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: Call Trace:<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc03747c5>] jbd2_log_wait_commit+0xc5/0x140 [jbd2]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0375e52>] jbd2_complete_transaction+0x52/0xa0 [jbd2]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0732da2>] ldiskfs_sync_file+0x2e2/0x320 [ldiskfs]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffa52760b0>] vfs_fsync_range+0x20/0x30<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0c8b651>] osd_object_sync+0xb1/0x160 [osd_ldiskfs]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0ab48a7>] tgt_sync+0xb7/0x270 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0dc3731>] ofd_sync_hdl+0x111/0x530 [ofd]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0aba1da>] tgt_request_handle+0xaea/0x1580 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0a5f80b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: LNet: Service thread pid 2403 completed after 200.29s. This indicates the system was overloaded (too many service threads, or there were not enough hardware<br>
resources).<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: LNet: Skipped 48 previous similar messages<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0a6313c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffa50c1c71>] kthread+0xd1/0xe0<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffa5775c37>] ret_from_fork_nospec_end+0x0/0x39<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffffffffff>] 0xffffffffffffffff<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>