<div dir="auto"><div>Hello </div><div dir="auto"><br></div><div dir="auto">I \U0001f914 it's a kernel panic that's why the nodes reboot.</div><div dir="auto"><br></div><div dir="auto">What kind of OS is installed on nodes?</div><div dir="auto"><br></div><div dir="auto">Thanks </div><div><br></div><div data-smartmail="gmail_signature">Tahari.Abdeslam</div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">Le ven. 25 sept. 2026, 12:14, Anil Mahadev Misal via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(51,51,51)">
Dear Team,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(51,51,51)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(51,51,51)">
Any update on below issue!</div>
<div id="m_-846357289642987727Signature">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(51,51,51)">
<br>
</div>
<p style="margin-top:1em;margin-bottom:1em"><span style="font-family:"Franklin Gothic Book","Avenir Next Condensed",sans-serif;font-size:11pt;color:rgb(78,89,89)"></span><span style="font-family:Tahoma,Geneva,sans-serif;font-size:11pt;color:rgb(0,0,0)"></span></p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-family:"Franklin Gothic Book","Avenir Next Condensed",sans-serif;font-size:11pt;color:rgb(78,89,89)">Thanks & Regards,<br>
Anil Mahadev Misal</span><span style="font-family:Tahoma,Geneva,sans-serif;font-size:11pt;color:rgb(0,0,0)"><br>
</span><span style="font-family:"Aptos Narrow",Aptos_EmbeddedFont,Aptos_MSFontService,sans-serif;font-size:11pt;color:rgb(0,0,0)">\U0001f4f1 +91-7045598174<br>
\U0001f4e7 <a href="mailto:supportmum@netwebindia.com" target="_blank" rel="noreferrer">supportmum@netwebindia.com</a></span></p>
<div style="font-family:Tahoma,Geneva,sans-serif;font-size:12pt;color:rgb(51,51,51)">
<br>
</div>
</div>
<div id="m_-846357289642987727appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_-846357289642987727divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of <a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss-request@lists.lustre.org</a> <<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss-request@lists.lustre.org</a>><br>
<b>Sent:</b> 15 September 2026 20:53<br>
<b>To:</b> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a> <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a>><br>
<b>Subject:</b> lustre-discuss Digest, Vol 3, Issue 7</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>Send lustre-discuss mailing list submissions to<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org" target="_blank" rel="noreferrer">
https://ind01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fmailman%2Flistinfo%2Flustre-discuss_lists.lustre.org&data=05%7C02%7Canil.misal%40netwebindia.com%7C560f9a491a754218843d08df133d5df9%7Ca7c8cd3d12ff4b4fa365fefc38041590%7C0%7C0%7C639250826418600663%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kx214SP6sSEFcXu9Uqu3J%2Fgwq9eBM6H9V%2BQkb9xj0YY%3D&reserved=0</a><br>
<br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss-request@lists.lustre.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:lustre-discuss-owner@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss-owner@lists.lustre.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of lustre-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Lustre Client cpu nodes reboot (Anil Mahadev Misal)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 15 Sep 2026 15:22:59 +0000<br>
From: Anil Mahadev Misal <<a href="mailto:Anil.misal@netwebindia.com" target="_blank" rel="noreferrer">Anil.misal@netwebindia.com</a>><br>
To: "<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a>"<br>
<<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a>><br>
Cc: Elaiya Rajan <<a href="mailto:Elaiyarajan.N@netwebindia.com" target="_blank" rel="noreferrer">Elaiyarajan.N@netwebindia.com</a>>, Tata Srinivas<br>
<<a href="mailto:srinivas.t@netwebindia.com" target="_blank" rel="noreferrer">srinivas.t@netwebindia.com</a>><br>
Subject: [lustre-discuss] Lustre Client cpu nodes reboot<br>
Message-ID:<br>
<<a href="mailto:MAUPR01MB12521E0953552241D471D667F8FBA2@MAUPR01MB12521.INDPRD01.PROD.OUTLOOK.COM" target="_blank" rel="noreferrer">MAUPR01MB12521E0953552241D471D667F8FBA2@MAUPR01MB12521.INDPRD01.PROD.OUTLOOK.COM</a>><br>
<br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Dear Team,<br>
We are facing an issue with all of the Lustre client nodes, where the nodes gets rebooted while running CPU-intensive jobs.<br>
We have observed that the nodes are working fine when running GPU jobs, and the reboot issue is occurring specifically during CPU job execution.<br>
<br>
<br>
lctl lustre_build_version<br>
Lustre version: 2.17.0<br>
<br>
pcs status<br>
Cluster name: lustre-ha<br>
Cluster Summary:<br>
* Stack: corosync (Pacemaker is running)<br>
* Current DC: ioserver1 (version 2.1.10-1.el9-5693eaeee) - partition with quorum<br>
* Last updated: Tue Sep 15 20:48:55 2026 on ioserver1<br>
* Last change: Sat Sep 12 14:20:53 2026 by root via root on ioserver1<br>
* 2 nodes configured<br>
* 24 resource instances configured<br>
<br>
Node List:<br>
* Online: [ ioserver1 ioserver2 ]<br>
<br>
Full List of Resources:<br>
* Resource Group: gr-lustre:<br>
* mgt (ocf:lustre:Lustre): Started ioserver1<br>
* mdt (ocf:lustre:Lustre): Started ioserver1<br>
* Resource Group: gr-storage1:<br>
* zfs-ostpool0 (ocf:heartbeat:ZFS): Started ioserver1<br>
* ost0 (ocf:lustre:Lustre): Started ioserver1<br>
* zfs-ostpool1 (ocf:heartbeat:ZFS): Started ioserver1<br>
* ost1 (ocf:lustre:Lustre): Started ioserver1<br>
* zfs-ostpool2 (ocf:heartbeat:ZFS): Started ioserver1<br>
* ost2 (ocf:lustre:Lustre): Started ioserver1<br>
* zfs-ostpool3 (ocf:heartbeat:ZFS): Started ioserver1<br>
* ost3 (ocf:lustre:Lustre): Started ioserver1<br>
* Resource Group: gr-storage2:<br>
* zfs-ostpool4 (ocf:heartbeat:ZFS): Started ioserver2<br>
* ost4 (ocf:lustre:Lustre): Started ioserver2<br>
* zfs-ostpool5 (ocf:heartbeat:ZFS): Started ioserver2<br>
* ost5 (ocf:lustre:Lustre): Started ioserver2<br>
* zfs-ostpool6 (ocf:heartbeat:ZFS): Started ioserver2<br>
* ost6 (ocf:lustre:Lustre): Started ioserver2<br>
* zfs-ostpool7 (ocf:heartbeat:ZFS): Started ioserver2<br>
* ost7 (ocf:lustre:Lustre): Started ioserver2<br>
* Clone Set: healthLNET-clone [healthLNET]:<br>
* Started: [ ioserver1 ioserver2 ]<br>
* Clone Set: healthLUSTRE-clone [healthLUSTRE]:<br>
* Started: [ ioserver1 ioserver2 ]<br>
* fence-ioserver1 (stonith:fence_ipmilan): Started ioserver1<br>
* fence-ioserver2 (stonith:fence_ipmilan): Started ioserver1<br>
<br>
Daemon Status:<br>
corosync: active/enabled<br>
pacemaker: active/enabled<br>
pcsd: active/enabled<br>
<br>
<br>
Crash dump<br>
sudo cat /var/crash/127.0.0.1-2026-09-15-03:52:08/vmcore-dmesg.txt | grep -E -C 10 "Kernel panic|BUG:|Oops|RIP:"<br>
[64730.820615] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant()) lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed 5111808<br>
[66108.385514] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant()) lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed 5111808<br>
[70006.232564] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant()) lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed 5111808<br>
[71695.499922] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant()) lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed 5111808<br>
[72383.549162] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant()) lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed 5111808<br>
[72950.579937] NMI watchdog: Watchdog detected hard LOCKUP on cpu 143<br>
[72950.579947] Modules linked in: ib_core mgc(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) sunrpc(E) bonding(E) tls(E) rfkill amd_atl(E) intel_rapl_msr(E) intel_rapl_common(E) amd64_edac(E)
edac_mce_amd(E) vfat(E) fat(E) kvm_amd(E) ast kvm(E) drm_client_lib(E) drm_shmem_helper(E) ipmi_ssif(E) acpi_ipmi(E) drm_kms_helper(E) rapl(E) ipmi_si(E) acpi_cpufreq(E) i2c_piix4(E) pcspkr(E) k10temp(E) ipmi_devintf(E) i2c_smbus(E) ipmi_msghandler(E) joydev(E)
fuse(E) drm(E) dm_mod(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ahci(E) crct10dif_pclmul(E) libahci(E) crc32_pclmul(E) igb(E) crc32c_intel(E) i2c_algo_bit(E) bnxt_en libata(E) ghash_clmulni_intel(E) ccp(E) dca(E) sp5100_tco(E) rndis_host cdc_ether usbnet mii(E)<br>
[72950.580009] CPU: 143 PID: 1445 Comm: kworker/143:1 Kdump: loaded Tainted: G OE ------ --- 5.14.0-611.24.1.el9_7.x86_64 #1<br>
[72950.580014] Hardware name: Tyrone Systems SDA200A2N-212/SDA200A2N-212, BIOS 3.9a 06/09/2026<br>
[72950.580016] Workqueue: events brw_queue_work [osc]<br>
[72950.580041] RIP: 0010:native_queued_spin_lock_slowpath+0x20d/0x2b0<br>
[72950.580052] Code: 85 c0 74 ec eb f5 41 8d 4d 01 41 c1 e4 10 c1 e1 12 44 09 e1 89 c8 c1 e8 10 66 87 45 02 89 c2 c1 e2 10 75 4a 31 d2 eb 02 f3 90 <8b> 45 00 66 85 c0 75 f6 89 c6 66 31 f6 39 ce 74 74 c6 45 00 01 48<br>
[72950.580053] RSP: 0018:ff59ebe75d71bbd8 EFLAGS: 00000002<br>
[72950.580055] RAX: 0000000002400101 RBX: ff32dd03fbff48c0 RCX: 0000000002400000<br>
[72950.580057] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ff32dc4900e91680<br>
[72950.580058] RBP: ff32dc4900e91680 R08: 00007f9b2a236450 R09: 0000000000000dcb<br>
[72950.580059] R10: ffef2ac044a275c0 R11: ffef2ac0470d0bc0 R12: 0000000000000000<br>
[72950.580060] R13: 000000000000008f R14: 0000000000000000 R15: 0000000000000dca<br>
[72950.580061] FS: 0000000000000000(0000) GS:ff32dd03fbfc0000(0000) knlGS:0000000000000000<br>
[72950.580063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br>
[72950.580064] CR2: 00007f72c89fb780 CR3: 000000c19fcbe001 CR4: 0000000000f71ef0<br>
--<br>
[72950.580309] ? dequeue_task_fair+0xaa/0x370<br>
[72950.580312] osc_io_unplug0+0x7c/0xa0 [osc]<br>
[72950.580323] brw_queue_work+0x57/0x110 [osc]<br>
[72950.580334] process_one_work+0x193/0x380<br>
[72950.580341] worker_thread+0x281/0x3a0<br>
[72950.580344] ? __pfx_worker_thread+0x10/0x10<br>
[72950.580347] kthread+0x101/0x110<br>
[72950.580352] ? __pfx_kthread+0x10/0x10<br>
[72950.580355] ret_from_fork+0x28/0x50<br>
[72950.580362] </TASK><br>
[72950.580365] Kernel panic - not syncing: Hard LOCKUP<br>
[72950.580367] CPU: 143 PID: 1445 Comm: kworker/143:1 Kdump: loaded Tainted: G OE ------ --- 5.14.0-611.24.1.el9_7.x86_64 #1<br>
[72950.580369] Hardware name: Tyrone Systems SDA200A2N-212/SDA200A2N-212, BIOS 3.9a 06/09/2026<br>
[72950.580370] Workqueue: events brw_queue_work [osc]<br>
[72950.580380] Call Trace:<br>
[72950.580382] <NMI><br>
[72950.580383] dump_stack_lvl+0x34/0x48<br>
[72950.580389] panic+0x107/0x2bb<br>
[72950.580395] nmi_panic.cold+0xc/0xc<br>
[72950.580398] watchdog_overflow_callback.cold+0x5c/0x70<br>
[72950.580401] __perf_event_overflow+0x11f/0x360<br>
--<br>
[72950.580418] ? ghes_copy_tofrom_phys+0x74/0x120<br>
[72950.580422] ? srso_alias_return_thunk+0x5/0xfbef5<br>
[72950.580423] ? __ghes_peek_estatus.isra.0+0x49/0xb0<br>
[72950.580426] ? srso_alias_return_thunk+0x5/0xfbef5<br>
[72950.580427] ? ghes_in_nmi_queue_one_entry.constprop.0+0x3b/0x2a0<br>
[72950.580431] perf_event_nmi_handler+0x28/0x50<br>
[72950.580433] nmi_handle.part.0+0x56/0x110<br>
[72950.580437] default_do_nmi+0x127/0x180<br>
[72950.580441] exc_nmi+0x100/0x180<br>
[72950.580444] end_repeat_nmi+0xf/0x5e<br>
[72950.580447] RIP: 0010:native_queued_spin_lock_slowpath+0x20d/0x2b0<br>
[72950.580450] Code: 85 c0 74 ec eb f5 41 8d 4d 01 41 c1 e4 10 c1 e1 12 44 09 e1 89 c8 c1 e8 10 66 87 45 02 89 c2 c1 e2 10 75 4a 31 d2 eb 02 f3 90 <8b> 45 00 66 85 c0 75 f6 89 c6 66 31 f6 39 ce 74 74 c6 45 00 01 48<br>
[72950.580452] RSP: 0018:ff59ebe75d71bbd8 EFLAGS: 00000002<br>
[72950.580453] RAX: 0000000002400101 RBX: ff32dd03fbff48c0 RCX: 0000000002400000<br>
[72950.580454] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ff32dc4900e91680<br>
[72950.580455] RBP: ff32dc4900e91680 R08: 00007f9b2a236450 R09: 0000000000000dcb<br>
[72950.580456] R10: ffef2ac044a275c0 R11: ffef2ac0470d0bc0 R12: 0000000000000000<br>
[72950.580457] R13: 000000000000008f R14: 0000000000000000 R15: 0000000000000dca<br>
[72950.580461] ? native_queued_spin_lock_slowpath+0x20d/0x2b0<br>
[72950.580464] ? native_queued_spin_lock_slowpath+0x20d/0x2b0<br>
[72950.580467] </NMI><br>
<br>
<br>
<br>
Thanks & Regards,<br>
Anil Mahadev Misal<br>
\U0001f4f1 +91-7045598174<br>
\U0001f4e7 <a href="mailto:supportmum@netwebindia.com" target="_blank" rel="noreferrer">supportmum@netwebindia.com</a><br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260915/4b9254d5/attachment.html" target="_blank" rel="noreferrer">https://ind01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fpipermail%2Flustre-discuss_lists.lustre.org%2Fattachments%2F20260915%2F4b9254d5%2Fattachment.html&data=05%7C02%7Canil.misal%40netwebindia.com%7C560f9a491a754218843d08df133d5df9%7Ca7c8cd3d12ff4b4fa365fefc38041590%7C0%7C0%7C639250826418630383%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=RXN3kC0q7E%2BcTtDWF7nKx%2FSqXSBpUPhl%2BNlh48F1rPY%3D&reserved=0</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org" target="_blank" rel="noreferrer">https://ind01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fmailman%2Flistinfo%2Flustre-discuss_lists.lustre.org&data=05%7C02%7Canil.misal%40netwebindia.com%7C560f9a491a754218843d08df133d5df9%7Ca7c8cd3d12ff4b4fa365fefc38041590%7C0%7C0%7C639250826418649694%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=EBbt1Q5PQOJHtaYkrXe1L%2FBqfWw19DQ9fM4UXbJOKQY%3D&reserved=0</a><br>
<br>
<br>
------------------------------<br>
<br>
End of lustre-discuss Digest, Vol 3, Issue 7<br>
********************************************<br>
</div>
</span></font></div>
</div>
-- <br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org" rel="noreferrer noreferrer" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org</a><br>
</blockquote></div>