[lustre-discuss] lustre-discuss Digest, Vol 3, Issue 7
Abdeslam Tahari
abeslam at gmail.com
Fri Sep 25 13:49:27 UTC 2026
If there's a new kernel version, try to boot on the old one
Tahari.Abdeslam
Le ven. 25 sept. 2026, 15:47, Abdeslam Tahari <abeslam at gmail.com> a écrit :
> It has been updated or upgraded recently?
>
> A new kernel version was installed recently?
>
> Tahari.Abdeslam
>
> Le ven. 25 sept. 2026, 15:40, Anil Mahadev Misal <
> Anil.misal at netwebindia.com> a écrit :
>
>> Dear @Abdeslam Tah <abdeslam.tahari.aflou at gmail.com>,
>>
>> Thank you for your support.
>>
>> Os version Rocky linux 9.8
>>
>> Thanks & Regards,
>> Anil Mahadev Misal
>> ------------------------------
>> *From:* Abdeslam Tah <abdeslam.tahari.aflou at gmail.com>
>> *Sent:* Friday, 25 September 2026 19:07:01
>> *To:* Abdeslam Tahari <abeslam at gmail.com>
>> *Cc:* Anil Mahadev Misal <Anil.misal at netwebindia.com>; lustre-discuss <
>> lustre-discuss at lists.lustre.org>
>> *Subject:* Re: [lustre-discuss] lustre-discuss Digest, Vol 3, Issue 7
>>
>> You don't often get email from abdeslam.tahari.aflou at gmail.com. Learn
>> why this is important <https://aka.ms/LearnAboutSenderIdentification>
>> What is the installed os operating system
>> I can help you out
>>
>> Le ven. 25 sept. 2026, 15:32, Abdeslam Tahari via lustre-discuss <
>> lustre-discuss at lists.lustre.org> a écrit :
>>
>> Hello
>>
>> I 🤔 it's a kernel panic that's why the nodes reboot.
>>
>> What kind of OS is installed on nodes?
>>
>> Thanks
>>
>> Tahari.Abdeslam
>>
>> Le ven. 25 sept. 2026, 12:14, Anil Mahadev Misal via lustre-discuss <
>> lustre-discuss at lists.lustre.org> a écrit :
>>
>> Dear Team,
>>
>> Any update on below issue!
>>
>> Thanks & Regards,
>> Anil Mahadev Misal
>> 📱 +91-7045598174
>> 📧 supportmum at netwebindia.com
>>
>> ------------------------------
>> *From:* lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on
>> behalf of lustre-discuss-request at lists.lustre.org <
>> lustre-discuss-request at lists.lustre.org>
>> *Sent:* 15 September 2026 20:53
>> *To:* lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
>> *Subject:* lustre-discuss Digest, Vol 3, Issue 7
>>
>> Send lustre-discuss mailing list submissions to
>> lustre-discuss at lists.lustre.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>
>> https://ind01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fmailman%2Flistinfo%2Flustre-discuss_lists.lustre.org&data=05%7C02%7Canil.misal%40netwebindia.com%7C560f9a491a754218843d08df133d5df9%7Ca7c8cd3d12ff4b4fa365fefc38041590%7C0%7C0%7C639250826418600663%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kx214SP6sSEFcXu9Uqu3J%2Fgwq9eBM6H9V%2BQkb9xj0YY%3D&reserved=0
>> <http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org>
>>
>> or, via email, send a message with subject or body 'help' to
>> lustre-discuss-request at lists.lustre.org
>>
>> You can reach the person managing the list at
>> lustre-discuss-owner at lists.lustre.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of lustre-discuss digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Lustre Client cpu nodes reboot (Anil Mahadev Misal)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 15 Sep 2026 15:22:59 +0000
>> From: Anil Mahadev Misal <Anil.misal at netwebindia.com>
>> To: "lustre-discuss at lists.lustre.org"
>> <lustre-discuss at lists.lustre.org>
>> Cc: Elaiya Rajan <Elaiyarajan.N at netwebindia.com>, Tata Srinivas
>> <srinivas.t at netwebindia.com>
>> Subject: [lustre-discuss] Lustre Client cpu nodes reboot
>> Message-ID:
>> <
>> MAUPR01MB12521E0953552241D471D667F8FBA2 at MAUPR01MB12521.INDPRD01.PROD.OUTLOOK.COM
>> >
>>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Dear Team,
>> We are facing an issue with all of the Lustre client nodes, where the
>> nodes gets rebooted while running CPU-intensive jobs.
>> We have observed that the nodes are working fine when running GPU jobs,
>> and the reboot issue is occurring specifically during CPU job execution.
>>
>>
>> lctl lustre_build_version
>> Lustre version: 2.17.0
>>
>> pcs status
>> Cluster name: lustre-ha
>> Cluster Summary:
>> * Stack: corosync (Pacemaker is running)
>> * Current DC: ioserver1 (version 2.1.10-1.el9-5693eaeee) - partition
>> with quorum
>> * Last updated: Tue Sep 15 20:48:55 2026 on ioserver1
>> * Last change: Sat Sep 12 14:20:53 2026 by root via root on ioserver1
>> * 2 nodes configured
>> * 24 resource instances configured
>>
>> Node List:
>> * Online: [ ioserver1 ioserver2 ]
>>
>> Full List of Resources:
>> * Resource Group: gr-lustre:
>> * mgt (ocf:lustre:Lustre): Started ioserver1
>> * mdt (ocf:lustre:Lustre): Started ioserver1
>> * Resource Group: gr-storage1:
>> * zfs-ostpool0 (ocf:heartbeat:ZFS): Started ioserver1
>> * ost0 (ocf:lustre:Lustre): Started ioserver1
>> * zfs-ostpool1 (ocf:heartbeat:ZFS): Started ioserver1
>> * ost1 (ocf:lustre:Lustre): Started ioserver1
>> * zfs-ostpool2 (ocf:heartbeat:ZFS): Started ioserver1
>> * ost2 (ocf:lustre:Lustre): Started ioserver1
>> * zfs-ostpool3 (ocf:heartbeat:ZFS): Started ioserver1
>> * ost3 (ocf:lustre:Lustre): Started ioserver1
>> * Resource Group: gr-storage2:
>> * zfs-ostpool4 (ocf:heartbeat:ZFS): Started ioserver2
>> * ost4 (ocf:lustre:Lustre): Started ioserver2
>> * zfs-ostpool5 (ocf:heartbeat:ZFS): Started ioserver2
>> * ost5 (ocf:lustre:Lustre): Started ioserver2
>> * zfs-ostpool6 (ocf:heartbeat:ZFS): Started ioserver2
>> * ost6 (ocf:lustre:Lustre): Started ioserver2
>> * zfs-ostpool7 (ocf:heartbeat:ZFS): Started ioserver2
>> * ost7 (ocf:lustre:Lustre): Started ioserver2
>> * Clone Set: healthLNET-clone [healthLNET]:
>> * Started: [ ioserver1 ioserver2 ]
>> * Clone Set: healthLUSTRE-clone [healthLUSTRE]:
>> * Started: [ ioserver1 ioserver2 ]
>> * fence-ioserver1 (stonith:fence_ipmilan): Started ioserver1
>> * fence-ioserver2 (stonith:fence_ipmilan): Started ioserver1
>>
>> Daemon Status:
>> corosync: active/enabled
>> pacemaker: active/enabled
>> pcsd: active/enabled
>>
>>
>> Crash dump
>> sudo cat /var/crash/127.0.0.1-2026-09-15-03:52:08/vmcore-dmesg.txt | grep
>> -E -C 10 "Kernel panic|BUG:|Oops|RIP:"
>> [64730.820615] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant())
>> lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed
>> 5111808
>> [66108.385514] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant())
>> lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed
>> 5111808
>> [70006.232564] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant())
>> lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed
>> 5111808
>> [71695.499922] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant())
>> lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed
>> 5111808
>> [72383.549162] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant())
>> lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed
>> 5111808
>> [72950.579937] NMI watchdog: Watchdog detected hard LOCKUP on cpu 143
>> [72950.579947] Modules linked in: ib_core mgc(OE) lustre(OE) mdc(OE)
>> lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE)
>> obdclass(OE) lnet(OE) libcfs(OE) sunrpc(E) bonding(E) tls(E) rfkill
>> amd_atl(E) intel_rapl_msr(E) intel_rapl_common(E) amd64_edac(E)
>> edac_mce_amd(E) vfat(E) fat(E) kvm_amd(E) ast kvm(E) drm_client_lib(E)
>> drm_shmem_helper(E) ipmi_ssif(E) acpi_ipmi(E) drm_kms_helper(E) rapl(E)
>> ipmi_si(E) acpi_cpufreq(E) i2c_piix4(E) pcspkr(E) k10temp(E)
>> ipmi_devintf(E) i2c_smbus(E) ipmi_msghandler(E) joydev(E) fuse(E) drm(E)
>> dm_mod(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ahci(E) crct10dif_pclmul(E)
>> libahci(E) crc32_pclmul(E) igb(E) crc32c_intel(E) i2c_algo_bit(E) bnxt_en
>> libata(E) ghash_clmulni_intel(E) ccp(E) dca(E) sp5100_tco(E) rndis_host
>> cdc_ether usbnet mii(E)
>> [72950.580009] CPU: 143 PID: 1445 Comm: kworker/143:1 Kdump: loaded
>> Tainted: G OE ------ --- 5.14.0-611.24.1.el9_7.x86_64 #1
>> [72950.580014] Hardware name: Tyrone Systems SDA200A2N-212/SDA200A2N-212,
>> BIOS 3.9a 06/09/2026
>> [72950.580016] Workqueue: events brw_queue_work [osc]
>> [72950.580041] RIP: 0010:native_queued_spin_lock_slowpath+0x20d/0x2b0
>> [72950.580052] Code: 85 c0 74 ec eb f5 41 8d 4d 01 41 c1 e4 10 c1 e1 12
>> 44 09 e1 89 c8 c1 e8 10 66 87 45 02 89 c2 c1 e2 10 75 4a 31 d2 eb 02 f3 90
>> <8b> 45 00 66 85 c0 75 f6 89 c6 66 31 f6 39 ce 74 74 c6 45 00 01 48
>> [72950.580053] RSP: 0018:ff59ebe75d71bbd8 EFLAGS: 00000002
>> [72950.580055] RAX: 0000000002400101 RBX: ff32dd03fbff48c0 RCX:
>> 0000000002400000
>> [72950.580057] RDX: 0000000000000000 RSI: 0000000000000100 RDI:
>> ff32dc4900e91680
>> [72950.580058] RBP: ff32dc4900e91680 R08: 00007f9b2a236450 R09:
>> 0000000000000dcb
>> [72950.580059] R10: ffef2ac044a275c0 R11: ffef2ac0470d0bc0 R12:
>> 0000000000000000
>> [72950.580060] R13: 000000000000008f R14: 0000000000000000 R15:
>> 0000000000000dca
>> [72950.580061] FS: 0000000000000000(0000) GS:ff32dd03fbfc0000(0000)
>> knlGS:0000000000000000
>> [72950.580063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [72950.580064] CR2: 00007f72c89fb780 CR3: 000000c19fcbe001 CR4:
>> 0000000000f71ef0
>> --
>> [72950.580309] ? dequeue_task_fair+0xaa/0x370
>> [72950.580312] osc_io_unplug0+0x7c/0xa0 [osc]
>> [72950.580323] brw_queue_work+0x57/0x110 [osc]
>> [72950.580334] process_one_work+0x193/0x380
>> [72950.580341] worker_thread+0x281/0x3a0
>> [72950.580344] ? __pfx_worker_thread+0x10/0x10
>> [72950.580347] kthread+0x101/0x110
>> [72950.580352] ? __pfx_kthread+0x10/0x10
>> [72950.580355] ret_from_fork+0x28/0x50
>> [72950.580362] </TASK>
>> [72950.580365] Kernel panic - not syncing: Hard LOCKUP
>> [72950.580367] CPU: 143 PID: 1445 Comm: kworker/143:1 Kdump: loaded
>> Tainted: G OE ------ --- 5.14.0-611.24.1.el9_7.x86_64 #1
>> [72950.580369] Hardware name: Tyrone Systems SDA200A2N-212/SDA200A2N-212,
>> BIOS 3.9a 06/09/2026
>> [72950.580370] Workqueue: events brw_queue_work [osc]
>> [72950.580380] Call Trace:
>> [72950.580382] <NMI>
>> [72950.580383] dump_stack_lvl+0x34/0x48
>> [72950.580389] panic+0x107/0x2bb
>> [72950.580395] nmi_panic.cold+0xc/0xc
>> [72950.580398] watchdog_overflow_callback.cold+0x5c/0x70
>> [72950.580401] __perf_event_overflow+0x11f/0x360
>> --
>> [72950.580418] ? ghes_copy_tofrom_phys+0x74/0x120
>> [72950.580422] ? srso_alias_return_thunk+0x5/0xfbef5
>> [72950.580423] ? __ghes_peek_estatus.isra.0+0x49/0xb0
>> [72950.580426] ? srso_alias_return_thunk+0x5/0xfbef5
>> [72950.580427] ? ghes_in_nmi_queue_one_entry.constprop.0+0x3b/0x2a0
>> [72950.580431] perf_event_nmi_handler+0x28/0x50
>> [72950.580433] nmi_handle.part.0+0x56/0x110
>> [72950.580437] default_do_nmi+0x127/0x180
>> [72950.580441] exc_nmi+0x100/0x180
>> [72950.580444] end_repeat_nmi+0xf/0x5e
>> [72950.580447] RIP: 0010:native_queued_spin_lock_slowpath+0x20d/0x2b0
>> [72950.580450] Code: 85 c0 74 ec eb f5 41 8d 4d 01 41 c1 e4 10 c1 e1 12
>> 44 09 e1 89 c8 c1 e8 10 66 87 45 02 89 c2 c1 e2 10 75 4a 31 d2 eb 02 f3 90
>> <8b> 45 00 66 85 c0 75 f6 89 c6 66 31 f6 39 ce 74 74 c6 45 00 01 48
>> [72950.580452] RSP: 0018:ff59ebe75d71bbd8 EFLAGS: 00000002
>> [72950.580453] RAX: 0000000002400101 RBX: ff32dd03fbff48c0 RCX:
>> 0000000002400000
>> [72950.580454] RDX: 0000000000000000 RSI: 0000000000000100 RDI:
>> ff32dc4900e91680
>> [72950.580455] RBP: ff32dc4900e91680 R08: 00007f9b2a236450 R09:
>> 0000000000000dcb
>> [72950.580456] R10: ffef2ac044a275c0 R11: ffef2ac0470d0bc0 R12:
>> 0000000000000000
>> [72950.580457] R13: 000000000000008f R14: 0000000000000000 R15:
>> 0000000000000dca
>> [72950.580461] ? native_queued_spin_lock_slowpath+0x20d/0x2b0
>> [72950.580464] ? native_queued_spin_lock_slowpath+0x20d/0x2b0
>> [72950.580467] </NMI>
>>
>>
>>
>> Thanks & Regards,
>> Anil Mahadev Misal
>> 📱 +91-7045598174
>> 📧 supportmum at netwebindia.com
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> https://ind01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fpipermail%2Flustre-discuss_lists.lustre.org%2Fattachments%2F20260915%2F4b9254d5%2Fattachment.html&data=05%7C02%7Canil.misal%40netwebindia.com%7C560f9a491a754218843d08df133d5df9%7Ca7c8cd3d12ff4b4fa365fefc38041590%7C0%7C0%7C639250826418630383%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=RXN3kC0q7E%2BcTtDWF7nKx%2FSqXSBpUPhl%2BNlh48F1rPY%3D&reserved=0
>> <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260915/4b9254d5/attachment.html>
>> >
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>>
>> https://ind01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fmailman%2Flistinfo%2Flustre-discuss_lists.lustre.org&data=05%7C02%7Canil.misal%40netwebindia.com%7C560f9a491a754218843d08df133d5df9%7Ca7c8cd3d12ff4b4fa365fefc38041590%7C0%7C0%7C639250826418649694%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=EBbt1Q5PQOJHtaYkrXe1L%2FBqfWw19DQ9fM4UXbJOKQY%3D&reserved=0
>> <http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org>
>>
>>
>> ------------------------------
>>
>> End of lustre-discuss Digest, Vol 3, Issue 7
>> ********************************************
>> --
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org
>>
>> --
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260925/8715ce4d/attachment-0001.html>
More information about the lustre-discuss
mailing list