[lustre-discuss] lustre-discuss Digest, Vol 3, Issue 7

Fri Sep 25 13:40:50 UTC 2026

Dear @Abdeslam Tah<mailto:abdeslam.tahari.aflou at gmail.com>,

Thank you for your support.

Os version Rocky linux 9.8

Thanks & Regards,
Anil Mahadev Misal
________________________________
From: Abdeslam Tah <abdeslam.tahari.aflou at gmail.com>
Sent: Friday, 25 September 2026 19:07:01
To: Abdeslam Tahari <abeslam at gmail.com>
Cc: Anil Mahadev Misal <Anil.misal at netwebindia.com>; lustre-discuss <lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] lustre-discuss Digest, Vol 3, Issue 7

You don't often get email from abdeslam.tahari.aflou at gmail.com. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
What is the installed os operating system
I can help you out

Le ven. 25 sept. 2026, 15:32, Abdeslam Tahari via lustre-discuss <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>> a écrit :
Hello

I 🤔 it's a kernel panic that's why the nodes reboot.

What kind of OS is installed on nodes?

Thanks

Tahari.Abdeslam

Le ven. 25 sept. 2026, 12:14, Anil Mahadev Misal via lustre-discuss <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>> a écrit :
Dear Team,

Any update on below issue!

Thanks & Regards,
Anil Mahadev Misal
📱 +91-7045598174
📧 supportmum at netwebindia.com<mailto:supportmum at netwebindia.com>

________________________________
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org>> on behalf of lustre-discuss-request at lists.lustre.org<mailto:lustre-discuss-request at lists.lustre.org> <lustre-discuss-request at lists.lustre.org<mailto:lustre-discuss-request at lists.lustre.org>>
Sent: 15 September 2026 20:53
To: lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org> <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>>
Subject: lustre-discuss Digest, Vol 3, Issue 7

Send lustre-discuss mailing list submissions to
        lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>

To subscribe or unsubscribe via the World Wide Web, visit
        https://ind01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fmailman%2Flistinfo%2Flustre-discuss_lists.lustre.org&data=05%7C02%7Canil.misal%40netwebindia.com%7C560f9a491a754218843d08df133d5df9%7Ca7c8cd3d12ff4b4fa365fefc38041590%7C0%7C0%7C639250826418600663%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kx214SP6sSEFcXu9Uqu3J%2Fgwq9eBM6H9V%2BQkb9xj0YY%3D&reserved=0<http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org>

or, via email, send a message with subject or body 'help' to
        lustre-discuss-request at lists.lustre.org<mailto:lustre-discuss-request at lists.lustre.org>

You can reach the person managing the list at
        lustre-discuss-owner at lists.lustre.org<mailto:lustre-discuss-owner at lists.lustre.org>

When replying, please edit your Subject line so it is more specific
than "Re: Contents of lustre-discuss digest..."

Today's Topics:

   1. Lustre Client cpu nodes reboot (Anil Mahadev Misal)

----------------------------------------------------------------------

Message: 1
Date: Tue, 15 Sep 2026 15:22:59 +0000
From: Anil Mahadev Misal <Anil.misal at netwebindia.com<mailto:Anil.misal at netwebindia.com>>
To: "lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>"
        <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>>
Cc: Elaiya Rajan <Elaiyarajan.N at netwebindia.com<mailto:Elaiyarajan.N at netwebindia.com>>, Tata Srinivas
        <srinivas.t at netwebindia.com<mailto:srinivas.t at netwebindia.com>>
Subject: [lustre-discuss] Lustre Client cpu nodes reboot
Message-ID:
        <MAUPR01MB12521E0953552241D471D667F8FBA2 at MAUPR01MB12521.INDPRD01.PROD.OUTLOOK.COM<mailto:MAUPR01MB12521E0953552241D471D667F8FBA2 at MAUPR01MB12521.INDPRD01.PROD.OUTLOOK.COM>>

Content-Type: text/plain; charset="utf-8"

Dear Team,
We are facing an issue with all of the Lustre client nodes, where the nodes gets rebooted while running CPU-intensive jobs.
We have observed that the nodes are working fine when running GPU jobs, and the reboot issue is occurring specifically during CPU job execution.

lctl lustre_build_version
Lustre version: 2.17.0

pcs status
Cluster name: lustre-ha
Cluster Summary:
  * Stack: corosync (Pacemaker is running)
  * Current DC: ioserver1 (version 2.1.10-1.el9-5693eaeee) - partition with quorum
  * Last updated: Tue Sep 15 20:48:55 2026 on ioserver1
  * Last change:  Sat Sep 12 14:20:53 2026 by root via root on ioserver1
  * 2 nodes configured
  * 24 resource instances configured

Node List:
  * Online: [ ioserver1 ioserver2 ]

Full List of Resources:
  * Resource Group: gr-lustre:
    * mgt       (ocf:lustre:Lustre):     Started ioserver1
    * mdt       (ocf:lustre:Lustre):     Started ioserver1
  * Resource Group: gr-storage1:
    * zfs-ostpool0      (ocf:heartbeat:ZFS):     Started ioserver1
    * ost0      (ocf:lustre:Lustre):     Started ioserver1
    * zfs-ostpool1      (ocf:heartbeat:ZFS):     Started ioserver1
    * ost1      (ocf:lustre:Lustre):     Started ioserver1
    * zfs-ostpool2      (ocf:heartbeat:ZFS):     Started ioserver1
    * ost2      (ocf:lustre:Lustre):     Started ioserver1
    * zfs-ostpool3      (ocf:heartbeat:ZFS):     Started ioserver1
    * ost3      (ocf:lustre:Lustre):     Started ioserver1
  * Resource Group: gr-storage2:
    * zfs-ostpool4      (ocf:heartbeat:ZFS):     Started ioserver2
    * ost4      (ocf:lustre:Lustre):     Started ioserver2
    * zfs-ostpool5      (ocf:heartbeat:ZFS):     Started ioserver2
    * ost5      (ocf:lustre:Lustre):     Started ioserver2
    * zfs-ostpool6      (ocf:heartbeat:ZFS):     Started ioserver2
    * ost6      (ocf:lustre:Lustre):     Started ioserver2
    * zfs-ostpool7      (ocf:heartbeat:ZFS):     Started ioserver2
    * ost7      (ocf:lustre:Lustre):     Started ioserver2
  * Clone Set: healthLNET-clone [healthLNET]:
    * Started: [ ioserver1 ioserver2 ]
  * Clone Set: healthLUSTRE-clone [healthLUSTRE]:
    * Started: [ ioserver1 ioserver2 ]
  * fence-ioserver1     (stonith:fence_ipmilan):         Started ioserver1
  * fence-ioserver2     (stonith:fence_ipmilan):         Started ioserver1

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Crash dump
sudo cat /var/crash/127.0.0.1-2026-09-15-03:52:08/vmcore-dmesg.txt | grep -E -C 10 "Kernel panic|BUG:|Oops|RIP:"
[64730.820615] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant()) lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed 5111808
[66108.385514] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant()) lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed 5111808
[70006.232564] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant()) lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed 5111808
[71695.499922] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant()) lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed 5111808
[72383.549162] LustreError: 4161:0:(osc_request.c:1009:osc_init_grant()) lustrefs-OST0004-osc-ff32dc4364de0800: granted 3407872 but already consumed 5111808
[72950.579937] NMI watchdog: Watchdog detected hard LOCKUP on cpu 143
[72950.579947] Modules linked in: ib_core mgc(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) sunrpc(E) bonding(E) tls(E) rfkill amd_atl(E) intel_rapl_msr(E) intel_rapl_common(E) amd64_edac(E) edac_mce_amd(E) vfat(E) fat(E) kvm_amd(E) ast kvm(E) drm_client_lib(E) drm_shmem_helper(E) ipmi_ssif(E) acpi_ipmi(E) drm_kms_helper(E) rapl(E) ipmi_si(E) acpi_cpufreq(E) i2c_piix4(E) pcspkr(E) k10temp(E) ipmi_devintf(E) i2c_smbus(E) ipmi_msghandler(E) joydev(E) fuse(E) drm(E) dm_mod(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ahci(E) crct10dif_pclmul(E) libahci(E) crc32_pclmul(E) igb(E) crc32c_intel(E) i2c_algo_bit(E) bnxt_en libata(E) ghash_clmulni_intel(E) ccp(E) dca(E) sp5100_tco(E) rndis_host cdc_ether usbnet mii(E)
[72950.580009] CPU: 143 PID: 1445 Comm: kworker/143:1 Kdump: loaded Tainted: G           OE      ------  ---  5.14.0-611.24.1.el9_7.x86_64 #1
[72950.580014] Hardware name: Tyrone Systems SDA200A2N-212/SDA200A2N-212, BIOS 3.9a 06/09/2026
[72950.580016] Workqueue: events brw_queue_work [osc]
[72950.580041] RIP: 0010:native_queued_spin_lock_slowpath+0x20d/0x2b0
[72950.580052] Code: 85 c0 74 ec eb f5 41 8d 4d 01 41 c1 e4 10 c1 e1 12 44 09 e1 89 c8 c1 e8 10 66 87 45 02 89 c2 c1 e2 10 75 4a 31 d2 eb 02 f3 90 <8b> 45 00 66 85 c0 75 f6 89 c6 66 31 f6 39 ce 74 74 c6 45 00 01 48
[72950.580053] RSP: 0018:ff59ebe75d71bbd8 EFLAGS: 00000002
[72950.580055] RAX: 0000000002400101 RBX: ff32dd03fbff48c0 RCX: 0000000002400000
[72950.580057] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ff32dc4900e91680
[72950.580058] RBP: ff32dc4900e91680 R08: 00007f9b2a236450 R09: 0000000000000dcb
[72950.580059] R10: ffef2ac044a275c0 R11: ffef2ac0470d0bc0 R12: 0000000000000000
[72950.580060] R13: 000000000000008f R14: 0000000000000000 R15: 0000000000000dca
[72950.580061] FS:  0000000000000000(0000) GS:ff32dd03fbfc0000(0000) knlGS:0000000000000000
[72950.580063] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[72950.580064] CR2: 00007f72c89fb780 CR3: 000000c19fcbe001 CR4: 0000000000f71ef0
--
[72950.580309]  ? dequeue_task_fair+0xaa/0x370
[72950.580312]  osc_io_unplug0+0x7c/0xa0 [osc]
[72950.580323]  brw_queue_work+0x57/0x110 [osc]
[72950.580334]  process_one_work+0x193/0x380
[72950.580341]  worker_thread+0x281/0x3a0
[72950.580344]  ? __pfx_worker_thread+0x10/0x10
[72950.580347]  kthread+0x101/0x110
[72950.580352]  ? __pfx_kthread+0x10/0x10
[72950.580355]  ret_from_fork+0x28/0x50
[72950.580362]  </TASK>
[72950.580365] Kernel panic - not syncing: Hard LOCKUP
[72950.580367] CPU: 143 PID: 1445 Comm: kworker/143:1 Kdump: loaded Tainted: G           OE      ------  ---  5.14.0-611.24.1.el9_7.x86_64 #1
[72950.580369] Hardware name: Tyrone Systems SDA200A2N-212/SDA200A2N-212, BIOS 3.9a 06/09/2026
[72950.580370] Workqueue: events brw_queue_work [osc]
[72950.580380] Call Trace:
[72950.580382]  <NMI>
[72950.580383]  dump_stack_lvl+0x34/0x48
[72950.580389]  panic+0x107/0x2bb
[72950.580395]  nmi_panic.cold+0xc/0xc
[72950.580398]  watchdog_overflow_callback.cold+0x5c/0x70
[72950.580401]  __perf_event_overflow+0x11f/0x360
--
[72950.580418]  ? ghes_copy_tofrom_phys+0x74/0x120
[72950.580422]  ? srso_alias_return_thunk+0x5/0xfbef5
[72950.580423]  ? __ghes_peek_estatus.isra.0+0x49/0xb0
[72950.580426]  ? srso_alias_return_thunk+0x5/0xfbef5
[72950.580427]  ? ghes_in_nmi_queue_one_entry.constprop.0+0x3b/0x2a0
[72950.580431]  perf_event_nmi_handler+0x28/0x50
[72950.580433]  nmi_handle.part.0+0x56/0x110
[72950.580437]  default_do_nmi+0x127/0x180
[72950.580441]  exc_nmi+0x100/0x180
[72950.580444]  end_repeat_nmi+0xf/0x5e
[72950.580447] RIP: 0010:native_queued_spin_lock_slowpath+0x20d/0x2b0
[72950.580450] Code: 85 c0 74 ec eb f5 41 8d 4d 01 41 c1 e4 10 c1 e1 12 44 09 e1 89 c8 c1 e8 10 66 87 45 02 89 c2 c1 e2 10 75 4a 31 d2 eb 02 f3 90 <8b> 45 00 66 85 c0 75 f6 89 c6 66 31 f6 39 ce 74 74 c6 45 00 01 48
[72950.580452] RSP: 0018:ff59ebe75d71bbd8 EFLAGS: 00000002
[72950.580453] RAX: 0000000002400101 RBX: ff32dd03fbff48c0 RCX: 0000000002400000
[72950.580454] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ff32dc4900e91680
[72950.580455] RBP: ff32dc4900e91680 R08: 00007f9b2a236450 R09: 0000000000000dcb
[72950.580456] R10: ffef2ac044a275c0 R11: ffef2ac0470d0bc0 R12: 0000000000000000
[72950.580457] R13: 000000000000008f R14: 0000000000000000 R15: 0000000000000dca
[72950.580461]  ? native_queued_spin_lock_slowpath+0x20d/0x2b0
[72950.580464]  ? native_queued_spin_lock_slowpath+0x20d/0x2b0
[72950.580467]  </NMI>

Thanks & Regards,
Anil Mahadev Misal
📱 +91-7045598174
📧 supportmum at netwebindia.com<mailto:supportmum at netwebindia.com>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ind01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fpipermail%2Flustre-discuss_lists.lustre.org%2Fattachments%2F20260915%2F4b9254d5%2Fattachment.html&data=05%7C02%7Canil.misal%40netwebindia.com%7C560f9a491a754218843d08df133d5df9%7Ca7c8cd3d12ff4b4fa365fefc38041590%7C0%7C0%7C639250826418630383%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=RXN3kC0q7E%2BcTtDWF7nKx%2FSqXSBpUPhl%2BNlh48F1rPY%3D&reserved=0<http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260915/4b9254d5/attachment.html>>

------------------------------

Subject: Digest Footer

lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
https://ind01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Fmailman%2Flistinfo%2Flustre-discuss_lists.lustre.org&data=05%7C02%7Canil.misal%40netwebindia.com%7C560f9a491a754218843d08df133d5df9%7Ca7c8cd3d12ff4b4fa365fefc38041590%7C0%7C0%7C639250826418649694%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=EBbt1Q5PQOJHtaYkrXe1L%2FBqfWw19DQ9fM4UXbJOKQY%3D&reserved=0<http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org>

------------------------------

End of lustre-discuss Digest, Vol 3, Issue 7
********************************************
--
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org
--
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260925/72178d9e/attachment-0001.html>