<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
This is the trace up to the LBUG</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span>[5533797.889690] Lustre: Skipped 341 previous similar messages<br>
</span>
<div>[5533958.749284] LustreError: 105499:0:(tgt_grant.c:571:tgt_grant_incoming()) lustre19-OST002c: cli 901dcd33-cf45-dad4-a0c7-89b9a1fb91b6/ffff99656aa5a800 dirty 0 pend 0 grant -1310720<br>
</div>
<div>[5533958.754365] LustreError: 105499:0:(tgt_grant.c:573:tgt_grant_incoming()) LBUG<br>
</div>
<div>[5533958.756929] Pid: 105499, comm: ll_ost_io01_071 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Tue Apr 30 22:18:15 UTC 2019<br>
</div>
<div>[5533958.756931] Call Trace:<br>
</div>
<span>[5533958.756948] [<ffffffffc0bf57cc>] libcfs_call_trace+0x8c/0xc0 [libcfs]</span><br>
</div>
<div>
<div id="appendonsend"></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size: 11pt;" data-ogsc="" face="Calibri, sans-serif" color="#000000"><b>From:</b> Alex Zarochentsev <zamenator@gmail.com><br>
<b>Sent:</b> Wednesday, July 15, 2020 11:20 AM<br>
<b>To:</b> Kurt Strosahl <strosahl@jlab.org><br>
<b>Cc:</b> lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> [EXTERNAL] Re: [lustre-discuss] oss servers crashing</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Hello!<br>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Wed, Jul 15, 2020 at 5:28 PM Kurt Strosahl <<a href="mailto:strosahl@jlab.org">strosahl@jlab.org</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Good Morning,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Yesterday one of our lustre file servers rebooted several times. the crash dump showed:</div>
</div>
</blockquote>
<div><br>
</div>
<div>can you please provide a failed lustre assert message just above the kernel panic message ?</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div>Zam.</div>
<div><br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<div style="box-sizing:border-box; font-family:"Segoe UI",system-ui,"Apple Color Emoji","Segoe UI Emoji",sans-serif; font-size:14px">
[14333982.153989] Pid: 381367, comm: ll_ost_io01_076 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Tue Apr 30 22:18:15 UTC 2019<br>
[14333982.153989] Kernel panic - not syncing: LBUG<br>
[14333982.153990] Call Trace:<br>
[14333982.153993] CPU: 4 PID: 380760 Comm: ll_ost_io01_072 Kdump: loaded Tainted: P OE ------------ 3.10.0-957.10.1.el7_lustre.x86_64 #1<br>
[14333982.153994] Hardware name: Supermicro Super Server/X11DPL-i, BIOS 3.1 05/21/2019<br>
[14333982.153995] Call Trace:<br>
[14333982.154002] [<ffffffffbaf62e41>] dump_stack+0x19/0x1b<br>
[14333982.154006] [<ffffffffbaf5c550>] panic+0xe8/0x21f<br>
[14333982.154018] [<ffffffffc0ab87cc>] libcfs_call_trace+0x8c/0xc0 [libcfs]<br>
[14333982.154026] [<ffffffffc0ab88cb>] lbug_with_loc+0x9b/0xa0 [libcfs]<br>
[14333982.154036] [<ffffffffc0ab887c>] lbug_with_loc+0x4c/0xa0 [libcfs]<br>
[14333982.154096] [<ffffffffc12dfae0>] tgt_grant_incoming.isra.6+0x570/0x570 [ptlrpc]<br>
[14333982.154174] [<ffffffffc12dfae0>] tgt_grant_prepare_read+0x0/0x3b0 [ptlrpc]<br>
[14333982.154232] [<ffffffffc12dfbeb>] tgt_grant_prepare_read+0x10b/0x3b0 [ptlrpc]<br>
[14333982.154297] [<ffffffffc12dfbeb>] tgt_grant_prepare_read+0x10b/0x3b0 [ptlrpc]<br>
[14333982.154306] [<ffffffffc15e1ad0>] ofd_preprw+0x450/0x1160 [ofd]</div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
lustre versions:<span><br>
</span>
<div>lustre-resource-agents-2.12.1-1.el7.x86_64<br>
</div>
<div>lustre-2.12.1-1.el7.x86_64<br>
</div>
<div>kernel-devel-3.10.0-957.10.1.el7_lustre.x86_64<br>
</div>
<div>lustre-osd-zfs-mount-2.12.1-1.el7.x86_64<br>
</div>
<div>kernel-headers-3.10.0-957.10.1.el7_lustre.x86_64<br>
</div>
<div>kernel-3.10.0-957.10.1.el7_lustre.x86_64<br>
</div>
<span>lustre-zfs-dkms-2.12.1-1.el7.noarch</span><br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Could this be: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.whamcloud.com_browse_LU-2D12120&d=DwMFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=a1-ymUluZsecMceDMlAHsomwMJl4Iqg-UcfvwQZVldk&m=YJT9Uk1-l_VkuqWZ8LzbYCOgQgIB7NodKUdU04ZH2I8&s=UUumm95pXNO2HROIgZLcbfcrDYD98rOqJY3diW7U1i4&e=" id="x_gmail-m_6168877629920452931LPlnk282199" target="_blank">
https://jira.whamcloud.com/browse/LU-12120</a><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
w/r,<br>
</div>
<div id="x_gmail-m_6168877629920452931Signature">
<div>
<div id="x_gmail-m_6168877629920452931divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif;">
<p style="margin-top:0px; margin-bottom:0px"><span id="x_gmail-m_6168877629920452931ms-rterangepaste-start"></span><span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">Kurt J. Strosahl</span><br style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">
<span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">System Administrator: Lustre, HPC</span><br style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">
<span style="color: rgb(51, 51, 51); font-family: monospace; font-size: 14.16px;">Scientific Computing Group, Thomas Jefferson National Accelerator Facility</span><span id="x_gmail-m_6168877629920452931ms-rterangepaste-end"></span><br>
</p>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwMFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=a1-ymUluZsecMceDMlAHsomwMJl4Iqg-UcfvwQZVldk&m=YJT9Uk1-l_VkuqWZ8LzbYCOgQgIB7NodKUdU04ZH2I8&s=ImWiPMyWLoKXcmVRovEfUKFf5zp_d9wHSg1UfCKnzCU&e=" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>