[lustre-discuss] lustre-discuss Digest, Vol 3, Issue 7
Abdeslam Tahari
abeslam at gmail.com
Fri Sep 25 16:10:50 UTC 2026
If you like I suggest that solution for your issue
It may probably a cou failure due to temperature or lack of power supply
To check that out
Install stress-ng
And check by the command
stress-ng --cpu 0 --cpu-method matrixprod -t
It will tell what component crashs the kernel and restarts the machine
You can install lm- sensors to check the temperature of the cpus during
processing to see if there's no overheat
Tahari.Abdeslam
Le ven. 25 sept. 2026, 16:08, Anil Mahadev Misal <Anil.misal at netwebindia.com>
a écrit :
> Dear @Abdeslam Tah <abdeslam.tahari.aflou at gmail.com>,
>
> We facing issue with both kernel
>
> Thanks & Regards,
> Anil Mahadev Misal
>
> Get Outlook for Android <https://aka.ms/AAb9ysg>
> ------------------------------
> *From:* Abdeslam Tahari <abeslam at gmail.com>
> *Sent:* Friday, 25 September 2026 19:32:23
> *To:* Anil Mahadev Misal <Anil.misal at netwebindia.com>
> *Cc:* Abdeslam Tah <abdeslam.tahari.aflou at gmail.com>; lustre-discuss <
> lustre-discuss at lists.lustre.org>
> *Subject:* Re: [lustre-discuss] lustre-discuss Digest, Vol 3, Issue 7
>
> Can you post the grub menu or the kernels installed in the boot directory
> of a node
> Please
>
> Tahari.Abdeslam
>
> Le ven. 25 sept. 2026, 16:01, Abdeslam Tahari <abeslam at gmail.com> a
> écrit :
>
> I think it's a kernel issue
>
> Tahari.Abdeslam
>
> Le ven. 25 sept. 2026, 16:00, Abdeslam Tahari <abeslam at gmail.com> a
> écrit :
>
> Try to boot a node with an old kernel select it in the grub menu, or
> select an old kernel from the menu if you can , launch the job and see if
> it reboots
>
> Tahari.Abdeslam
>
> Le ven. 25 sept. 2026, 15:52, Anil Mahadev Misal <
> Anil.misal at netwebindia.com> a écrit :
>
> Dear @Abdeslam Tah <abdeslam.tahari.aflou at gmail.com>,
>
> There is no change in kernel from last 1.5 month. Last week we had update
> kernel of one CPU nodes, but issue remains same.
>
> It's strange that node reboot happened only running cpu job. GPU jobs are
> working fine
>
> Thanks & Regards,
> Anil Mahadev Misal
> ------------------------------
> *From:* Abdeslam Tahari <abeslam at gmail.com>
> *Sent:* Friday, 25 September 2026 19:17:09
> *To:* Anil Mahadev Misal <Anil.misal at netwebindia.com>
> *Cc:* Abdeslam Tah <abdeslam.tahari.aflou at gmail.com>; lustre-discuss <
> lustre-discuss at lists.lustre.org>
> *Subject:* Re: [lustre-discuss] lustre-discuss Digest, Vol 3, Issue 7
>
> You don't often get email from abeslam at gmail.com. Learn why this is
> important <https://aka.ms/LearnAboutSenderIdentification>
> It has been updated or upgraded recently?
>
> A new kernel version was installed recently?
>
> Tahari.Abdeslam
>
> Le ven. 25 sept. 2026, 15:40, Anil Mahadev Misal <
> Anil.misal at netwebindia.com> a écrit :
>
> Dear @Abdeslam Tah <abdeslam.tahari.aflou at gmail.com>,
>
> Thank you for your support.
>
> Os version Rocky linux 9.8
>
> Thanks & Regards,
> Anil Mahadev Misal
> ------------------------------
> *From:* Abdeslam Tah <abdeslam.tahari.aflou at gmail.com>
> *Sent:* Friday, 25 September 2026 19:07:01
> *To:* Abdeslam Tahari <abeslam at gmail.com>
> *Cc:* Anil Mahadev Misal <Anil.misal at netwebindia.com>; lustre-discuss <
> lustre-discuss at lists.lustre.org>
> *Subject:* Re: [lustre-discuss] lustre-discuss Digest, Vol 3, Issue 7
>
> You don't often get email from abdeslam.tahari.aflou at gmail.com. Learn why
> this is important <https://aka.ms/LearnAboutSenderIdentification>
> What is the installed os operating system
> I can help you out
>
> Le ven. 25 sept. 2026, 15:32, Abdeslam Tahari via lustre-discuss <
> lustre-discuss at lists.lustre.org> a écrit :
>
> Hello
>
> I 🤔 it's a kernel panic that's why the nodes reboot.
>
> What kind of OS is installed on nodes?
>
> Thanks
>
> Tahari.Abdeslam
>
> Le ven. 25 sept. 2026, 12:14, Anil Mahadev Misal via lustre-discuss <
> lustre-discuss at lists.lustre.org> a écrit :
>
> Dear Team,
>
> Any update on below issue!
>
> Thanks & Regards,
> Anil Mahadev Misal
> 📱 +91-7045598174
> 📧 supportmum at netwebindia.com
>
> ------------------------------
> *From:* lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on
> behalf of lustre-discuss-request at lists.lustre.org <
> lustre-discuss-request at lists.lustre.org>
> *Sent:* 15 September 2026 20:53
> *To:* lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
> *Subject:* lustre-discuss Digest, Vol 3, Issue 7
>
> Send lustre-discuss mailing list submissions to
> lustre-discuss at lists.lustre.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>
> https://ind01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fmailman%2Flistinfo%2Flustre-discuss_lists.lustre.org&data=05%7C02%7Canil.misal%40netwebindia.com%7C560f9a491a754218843d08df133d5df9%7Ca7c8cd3d12ff4b4fa365fefc38041590%7C0%7C0%7C639250826418600663%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kx214SP6sSEFcXu9Uqu3J%2Fgwq9eBM6H9V%2BQkb9xj0YY%3D&reserved=0
> <http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org>
>
> or, via email, send a message with subject or body 'help' to
> lustre-discuss-request at lists.lustre.org
>
> You can reach the person managing the list at
> lustre-discuss-owner at lists.lustre.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of lustre-discuss digest..."
>
>
> Today's Topics:
>
> 1. Lustre Client cpu nodes reboot (Anil Mahadev Misal)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 15 Sep 2026 15:22:59 +0000
> From: Anil Mahadev Misal <Anil.misal at netwebindia.com>
> To: "lustre-discuss at lists.lustre.org"
> <lustre-discuss at lists.lustre.org>
> Cc: Elaiya Rajan <Elaiyarajan.N at netwebindia.com>, Tata Srinivas
> <srinivas.t at netwebindia.com>
> Subject: [lustre-discuss] Lustre Client cpu nodes reboot
> Message-ID:
> <
> MAUPR01MB12521E0953552241D471D667F8FBA2 at MAUPR01MB12521.INDPRD01.PROD.OUTLOOK.COM
> >
>
> Content-Type: text/plain; charset="utf-8"
>
> Dear Team,
> We are facing an issue with all of the Lustre client nodes, where the
> nodes gets rebooted while running CPU-intensive jobs.
> We have observed that the nodes are working fine when running GPU jobs,
> and the reboot issue is occurring specifically during CPU job execution.
>
>
> lctl lustre_build_version
> Lustre version: 2.17.0
>
> pcs status
> Cluster name: lustre-ha
> Cluster Summary:
> * Stack: corosync (Pacemaker is running)
> * Current DC: ioserver1 (version 2.1.10-1.el9-5693eaeee) - partition
> with quorum
> * Last updated: Tue Sep 15 20:48:55 2026 on ioserver1
> * Last change: Sat Sep 12 14:20:53 2026 by root via root on ioserver1
> * 2 nodes configured
> * 24 resource instances configured
>
> Node List:
> * Online: [ ioserver1 ioserver2 ]
>
> Full List of Resources:
> * Resource Group: gr-lustre:
> * mgt (ocf:lustre:Lustre): Started ioserver1
> * mdt (ocf:lustre:Lustre): Started ioserver1
> * Resource Group: gr-storage1:
> * zfs-ostpool0 (ocf:heartbeat:ZFS): Started ioserver1
> * ost0 (ocf:lustre:Lustre): Started ioserver1
> * zfs-ostpool1 (ocf:heartbeat:ZFS): Started ioserver1
> * ost1 (ocf:lustre:Lustre): Started ioserver1
> * zfs-ostpool2 (ocf:heartbeat:ZFS): Started ioserver1
> * ost2 (ocf:lustre:Lustre): Started ioserver1
> * zfs-ostpool3 (ocf:heartbeat:ZFS): Started ioserver1
> * ost3 (ocf:lustre:Lustre): Started ioserver1
> * Resource Group: gr-storage2:
> * zfs-ostpool4 (ocf:heartbeat:ZFS): Started ioserver2
> * ost4 (ocf:lustre:Lustre): Started ioserver2
> * zfs-ostpool5 (ocf:heartbeat:ZFS): Started ioserver2
> * ost5 (ocf:lustre:Lustre): Started ioserver2
> * zfs-ostpool6 (ocf:heartbeat:ZFS): Started ioserver2
> * ost6 (ocf:lustre:Lustre): Started ioserver2
> * zfs-ostpool7 (ocf:heartbeat:ZFS): Started ioserver2
> * ost7 (ocf:lustre:Lustre): Started ioserver2
> * Clone Set: healthLNET-clone [healthLNET]:
> * Started: [ ioserver1 ioserver2 ]
> * Clone Set: healthLUSTRE-clone [healthLUSTRE]:
> * Started: [ ioserver1 ioserver2 ]
> * fence-ioserver1 (stonith:fence_ipmilan): Started ioserver1
> * fence-ioserver2 (stonith:fence_ipmilan): Started ioserver1
>
> Daemon Status:
> corosync: active/enabled
> pacemaker: active/enabled
> pcsd: active/enabled
>
>
> Crash dump
> sudo cat /var/crash/127.0.0.1-2026-09-15-03:52:08/vmcore-dmesg.txt | grep
> -E -C 10 "Kernel panic|BUG:|Oops|RIP:"
> [64730.820615] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant())
> lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed
> 5111808
> [66108.385514] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant())
> lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed
> 5111808
> [70006.232564] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant())
> lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed
> 5111808
> [71695.499922] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant())
> lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed
> 5111808
> [72383.549162] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant())
> lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed
> 5111808
> [72950.579937] NMI watchdog: Watchdog detected hard LOCKUP on cpu 143
> [72950.579947] Modules linked in: ib_core mgc(OE) lustre(OE) mdc(OE)
> lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE)
> obdclass(OE) lnet(OE) libcfs(OE) sunrpc(E) bonding(E) tls(E) rfkill
> amd_atl(E) intel_rapl_msr(E) intel_rapl_common(E) amd64_edac(E)
> edac_mce_amd(E) vfat(E) fat(E) kvm_amd(E) ast kvm(E) drm_client_lib(E)
> drm_shmem_helper(E) ipmi_ssif(E) acpi_ipmi(E) drm_kms_helper(E) rapl(E)
> ipmi_si(E) acpi_cpufreq(E) i2c_piix4(E) pcspkr(E) k10temp(E)
> ipmi_devintf(E) i2c_smbus(E) ipmi_msghandler(E) joydev(E) fuse(E) drm(E)
> dm_mod(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ahci(E) crct10dif_pclmul(E)
> libahci(E) crc32_pclmul(E) igb(E) crc32c_intel(E) i2c_algo_bit(E) bnxt_en
> libata(E) ghash_clmulni_intel(E) ccp(E) dca(E) sp5100_tco(E) rndis_host
> cdc_ether usbnet mii(E)
> [72950.580009] CPU: 143 PID: 1445 Comm: kworker/143:1 Kdump: loaded
> Tainted: G OE ------ --- 5.14.0-611.24.1.el9_7.x86_64 #1
> [72950.580014] Hardware name: Tyrone Systems SDA200A2N-212/SDA200A2N-212,
> BIOS 3.9a 06/09/2026
> [72950.580016] Workqueue: events brw_queue_work [osc]
> [72950.580041] RIP: 0010:native_queued_spin_lock_slowpath+0x20d/0x2b0
> [72950.580052] Code: 85 c0 74 ec eb f5 41 8d 4d 01 41 c1 e4 10 c1 e1 12 44
> 09 e1 89 c8 c1 e8 10 66 87 45 02 89 c2 c1 e2 10 75 4a 31 d2 eb 02 f3 90
> <8b> 45 00 66 85 c0 75 f6 89 c6 66 31 f6 39 ce 74 74 c6 45 00 01 48
> [72950.580053] RSP: 0018:ff59ebe75d71bbd8 EFLAGS: 00000002
> [72950.580055] RAX: 0000000002400101 RBX: ff32dd03fbff48c0 RCX:
> 0000000002400000
> [72950.580057] RDX: 0000000000000000 RSI: 0000000000000100 RDI:
> ff32dc4900e91680
> [72950.580058] RBP: ff32dc4900e91680 R08: 00007f9b2a236450 R09:
> 0000000000000dcb
> [72950.580059] R10: ffef2ac044a275c0 R11: ffef2ac0470d0bc0 R12:
> 0000000000000000
> [72950.580060] R13: 000000000000008f R14: 0000000000000000 R15:
> 0000000000000dca
> [72950.580061] FS: 0000000000000000(0000) GS:ff32dd03fbfc0000(0000)
> knlGS:0000000000000000
> [72950.580063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [72950.580064] CR2: 00007f72c89fb780 CR3: 000000c19fcbe001 CR4:
> 0000000000f71ef0
> --
> [72950.580309] ? dequeue_task_fair+0xaa/0x370
> [72950.580312] osc_io_unplug0+0x7c/0xa0 [osc]
> [72950.580323] brw_queue_work+0x57/0x110 [osc]
> [72950.580334] process_one_work+0x193/0x380
> [72950.580341] worker_thread+0x281/0x3a0
> [72950.580344] ? __pfx_worker_thread+0x10/0x10
> [72950.580347] kthread+0x101/0x110
> [72950.580352] ? __pfx_kthread+0x10/0x10
> [72950.580355] ret_from_fork+0x28/0x50
> [72950.580362] </TASK>
> [72950.580365] Kernel panic - not syncing: Hard LOCKUP
> [72950.580367] CPU: 143 PID: 1445 Comm: kworker/143:1 Kdump: loaded
> Tainted: G OE ------ --- 5.14.0-611.24.1.el9_7.x86_64 #1
> [72950.580369] Hardware name: Tyrone Systems SDA200A2N-212/SDA200A2N-212,
> BIOS 3.9a 06/09/2026
> [72950.580370] Workqueue: events brw_queue_work [osc]
> [72950.580380] Call Trace:
> [72950.580382] <NMI>
> [72950.580383] dump_stack_lvl+0x34/0x48
> [72950.580389] panic+0x107/0x2bb
> [72950.580395] nmi_panic.cold+0xc/0xc
> [72950.580398] watchdog_overflow_callback.cold+0x5c/0x70
> [72950.580401] __perf_event_overflow+0x11f/0x360
> --
> [72950.580418] ? ghes_copy_tofrom_phys+0x74/0x120
> [72950.580422] ? srso_alias_return_thunk+0x5/0xfbef5
> [72950.580423] ? __ghes_peek_estatus.isra.0+0x49/0xb0
> [72950.580426] ? srso_alias_return_thunk+0x5/0xfbef5
> [72950.580427] ? ghes_in_nmi_queue_one_entry.constprop.0+0x3b/0x2a0
> [72950.580431] perf_event_nmi_handler+0x28/0x50
> [72950.580433] nmi_handle.part.0+0x56/0x110
> [72950.580437] default_do_nmi+0x127/0x180
> [72950.580441] exc_nmi+0x100/0x180
> [72950.580444] end_repeat_nmi+0xf/0x5e
> [72950.580447] RIP: 0010:native_queued_spin_lock_slowpath+0x20d/0x2b0
> [72950.580450] Code: 85 c0 74 ec eb f5 41 8d 4d 01 41 c1 e4 10 c1 e1 12 44
> 09 e1 89 c8 c1 e8 10 66 87 45 02 89 c2 c1 e2 10 75 4a 31 d2 eb 02 f3 90
> <8b> 45 00 66 85 c0 75 f6 89 c6 66 31 f6 39 ce 74 74 c6 45 00 01 48
> [72950.580452] RSP: 0018:ff59ebe75d71bbd8 EFLAGS: 00000002
> [72950.580453] RAX: 0000000002400101 RBX: ff32dd03fbff48c0 RCX:
> 0000000002400000
> [72950.580454] RDX: 0000000000000000 RSI: 0000000000000100 RDI:
> ff32dc4900e91680
> [72950.580455] RBP: ff32dc4900e91680 R08: 00007f9b2a236450 R09:
> 0000000000000dcb
> [72950.580456] R10: ffef2ac044a275c0 R11: ffef2ac0470d0bc0 R12:
> 0000000000000000
> [72950.580457] R13: 000000000000008f R14: 0000000000000000 R15:
> 0000000000000dca
> [72950.580461] ? native_queued_spin_lock_slowpath+0x20d/0x2b0
> [72950.580464] ? native_queued_spin_lock_slowpath+0x20d/0x2b0
> [72950.580467] </NMI>
>
>
>
> Thanks & Regards,
> Anil Mahadev Misal
> 📱 +91-7045598174
> 📧 supportmum at netwebindia.com
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> https://ind01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fpipermail%2Flustre-discuss_lists.lustre.org%2Fattachments%2F20260915%2F4b9254d5%2Fattachment.html&data=05%7C02%7Canil.misal%40netwebindia.com%7C560f9a491a754218843d08df133d5df9%7Ca7c8cd3d12ff4b4fa365fefc38041590%7C0%7C0%7C639250826418630383%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=RXN3kC0q7E%2BcTtDWF7nKx%2FSqXSBpUPhl%2BNlh48F1rPY%3D&reserved=0
> <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260915/4b9254d5/attachment.html>
> >
>
> ------------------------------
>
> Subject: Digest Footer
>
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
>
> https://ind01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fmailman%2Flistinfo%2Flustre-discuss_lists.lustre.org&data=05%7C02%7Canil.misal%40netwebindia.com%7C560f9a491a754218843d08df133d5df9%7Ca7c8cd3d12ff4b4fa365fefc38041590%7C0%7C0%7C639250826418649694%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=EBbt1Q5PQOJHtaYkrXe1L%2FBqfWw19DQ9fM4UXbJOKQY%3D&reserved=0
> <http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org>
>
>
> ------------------------------
>
> End of lustre-discuss Digest, Vol 3, Issue 7
> ********************************************
> --
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org
>
> --
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260925/d457800a/attachment-0001.html>
More information about the lustre-discuss
mailing list