<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr" text-align="left">
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255); text-align: left;" dir="ltr">
Thanks, will take a look.</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255); text-align: left;" dir="ltr">
<br>
</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255); text-align: left;" dir="ltr">
Any other areas i should be looking? Should i be applying any Lustre tuning? </div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255); text-align: left;" dir="ltr">
<br>
</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255); text-align: left;" dir="ltr">
Thanks</div>
<div id="ms-outlook-mobile-signature">
<div><br>
</div>
Get <a href="https://aka.ms/ghei36">Outlook for Android</a></div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Oral, H. <oralhs@ornl.gov><br>
<b>Sent:</b> Monday, October 28, 2019 7:06:41 PM<br>
<b>To:</b> Louis Allen <louisallen@live.co.uk>; Carlson, Timothy S <Timothy.Carlson@pnnl.gov>; lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> Re: [EXTERNAL] Re: [lustre-discuss] Lustre Timeouts/Filesystem Hanging</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">For inspecting client side I/O, you can use Darshan. <br>
<br>
Thanks,<br>
<br>
Sarp<br>
<br>
-- <br>
Sarp Oral, PhD<br>
<br>
National Center for Computational Sciences<br>
Oak Ridge National Laboratory<br>
oralhs@ornl.gov<br>
865-574-2173 <br>
<br>
<br>
\ufeffOn 10/28/19, 1:58 PM, "lustre-discuss on behalf of Louis Allen" <lustre-discuss-bounces@lists.lustre.org on behalf of louisallen@live.co.uk> wrote:<br>
<br>
<br>
Thanks for the reply, Tim.<br>
<br>
<br>
Are there any tools I can use to see if that is the cause?<br>
<br>
<br>
Could any tuning possibly help the situation?<br>
<br>
<br>
Thanks<br>
<br>
<br>
<br>
<br>
<br>
________________________________________<br>
From: Carlson, Timothy S <Timothy.Carlson@pnnl.gov><br>
Sent: Monday, 28 October 2019, 17:24<br>
To: Louis Allen; lustre-discuss@lists.lustre.org<br>
Subject: RE: Lustre Timeouts/Filesystem Hanging<br>
<br>
<br>
In my experience, this is almost always related to some code doing really bad I/O. Let\u2019s say you have a 1000 rank MPI code doing open/read 4k/close on a few specific files on that OST. That will make for a bad day.<br>
<br>
The other place you can see this, and this isn\u2019t your case, is when ZFS refuses to give up on a disk that is failing and your overall I/O suffers from ZFS continuing to try to read from a disk that it should just kick out<br>
<br>
Tim<br>
<br>
<br>
From: lustre-discuss <lustre-discuss-bounces@lists.lustre.org><br>
On Behalf Of Louis Allen<br>
Sent: Monday, October 28, 2019 10:16 AM<br>
To: lustre-discuss@lists.lustre.org<br>
Subject: [lustre-discuss] Lustre Timeouts/Filesystem Hanging<br>
<br>
<br>
<br>
Hello,<br>
<br>
<br>
<br>
Lustre (2.12) seem to be hanging quite frequently (5+ times a day) for us and one of the OSS servers (out of 4) is reporting an extremely high load average (150+) but the CPU usage of that server<br>
is actually very low - so it must be related to something else - possibly CPU_IO_WAIT.<br>
<br>
<br>
<br>
The OSS server we are seeing the high load averages we can also see multiple LustreError messages in /var/log/messages:<br>
<br>
<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: LNet: Service thread pid 2403 was inactive for 200.08s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace<br>
for debugging purposes:<br>
Oct 28 11:22:23 pazlustreoss001 kernel: LNet: Skipped 4 previous similar messages<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: Pid: 2403, comm: ll_ost00_068 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Sun May 26 21:48:35 UTC 2019<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: Call Trace:<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc03747c5>] jbd2_log_wait_commit+0xc5/0x140 [jbd2]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0375e52>] jbd2_complete_transaction+0x52/0xa0 [jbd2]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0732da2>] ldiskfs_sync_file+0x2e2/0x320 [ldiskfs]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffa52760b0>] vfs_fsync_range+0x20/0x30<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0c8b651>] osd_object_sync+0xb1/0x160 [osd_ldiskfs]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0ab48a7>] tgt_sync+0xb7/0x270 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0dc3731>] ofd_sync_hdl+0x111/0x530 [ofd]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0aba1da>] tgt_request_handle+0xaea/0x1580 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0a5f80b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0a6313c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffa50c1c71>] kthread+0xd1/0xe0<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffa5775c37>] ret_from_fork_nospec_end+0x0/0x39<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffffffffff>] 0xffffffffffffffff<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: LustreError: dumping log to /tmp/lustre-log.1572261743.2403<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: Pid: 2292, comm: ll_ost03_043 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Sun May 26 21:48:35 UTC 2019<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: Call Trace:<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc03747c5>] jbd2_log_wait_commit+0xc5/0x140 [jbd2]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0375e52>] jbd2_complete_transaction+0x52/0xa0 [jbd2]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0732da2>] ldiskfs_sync_file+0x2e2/0x320 [ldiskfs]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffa52760b0>] vfs_fsync_range+0x20/0x30<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0c8b651>] osd_object_sync+0xb1/0x160 [osd_ldiskfs]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0ab48a7>] tgt_sync+0xb7/0x270 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0dc3731>] ofd_sync_hdl+0x111/0x530 [ofd]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0aba1da>] tgt_request_handle+0xaea/0x1580 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0a5f80b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: LNet: Service thread pid 2403 completed after 200.29s. This indicates the system was overloaded (too many service threads, or there were not enough hardware<br>
resources).<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: LNet: Skipped 48 previous similar messages<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffc0a6313c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffa50c1c71>] kthread+0xd1/0xe0<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffa5775c37>] ret_from_fork_nospec_end+0x0/0x39<br>
<br>
Oct 28 11:22:23 pazlustreoss001 kernel: [<ffffffffffffffff>] 0xffffffffffffffff<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</div>
</span></font></div>
</body>
</html>