[lustre-discuss] Kernel oops with lustre 2.17.0 on rocky 9.7 kernel 5.14.0-611.13.1
Nguyen Hoang Minh
minhnh at cmsinc.co.jp
Fri Sep 4 13:19:02 UTC 2026
Hi,
Due to a known bug LU-19121, we got some inaccessible files on our
Lustre 2.16.1.
Since this bug cannot be fixed in 2.16.1, we have recently upgrade our
cluster to Lustre 2.17.0 and ran e2fsck on all MDTs. e2fsck was be able
to fix inode corruptions on mdt1. However, if we mount mdt1, the host
server will immediately crashes with the following log:
=====================
[ 145.375972] Lustre: Lustre: Build Version: 2.17.0
[ 148.190895] Lustre: MGS: mounting server target with '-t lustre'
deprecated, use '-t lustre_tgt'
[ 149.386969] LDISKFS-fs (sdb): mounted filesystem
22a869df-1f5d-47b1-9d17-03380f3ab0b6 r/w with ordered data mode. Quota
mode: journalled.
[ 150.629175] SELinux: (dev lustre, type lustre) has no xattr support
[ 150.629235] SELinux: (dev lustre, type lustre) falling back to genfs
[ 3307.791077] LDISKFS-fs warning (device sdc):
ldiskfs_multi_mount_protect:334: MMP interval 42 higher than expected,
please wait.
[ 3350.684500] LDISKFS-fs (sdc): 9 truncates cleaned up
[ 3350.684557] LDISKFS-fs (sdc): recovery complete
[ 3350.687957] LDISKFS-fs (sdc): mounted filesystem
57aec60d-ce35-43e3-bb8c-ba329938b8f3 r/w with ordered data mode. Quota
mode: journalled.
[ 3352.248028] Lustre: lustreFS-MDT0000: Imperative Recovery enabled,
recovery window shrunk from 300-900 down to 150-900
[ 3352.253154] Lustre: lustreFS-MDT0000: in recovery but waiting for the
first client to connect
[ 3352.253230] SELinux: (dev lustre, type lustre) has no xattr support
[ 3352.253281] SELinux: (dev lustre, type lustre) falling back to genfs
[ 3353.617185] LNet: 1 local NIs in recovery (showing 1):
192.168.200.47 at o2ib10
[ 3374.119376] Lustre: lustreFS-MDT0000: Will be in recovery for at
least 2:30, or until 3 clients reconnect
[ 3387.880357] Lustre: lustreFS-MDT0001-osp-MDT0000: Connection restored
to 192.168.200.48 at o2ib10 (at 192.168.200.48 at o2ib10)
[ 3388.450341] Lustre: lustreFS-MDT0000: Recovery over after 0:15, of 3
clients 3 recovered and 0 were evicted.
[ 3388.482527] BUG: unable to handle page fault for address:
ffd317785f002008
[ 3388.482533] #PF: supervisor read access in kernel mode
[ 3388.482535] #PF: error_code(0x0000) - not-present page
[ 3388.482536] PGD 0
[ 3388.482541] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 3388.482546] CPU: 9 PID: 23517 Comm: qsd_reint_1.lus Kdump: loaded
Tainted: G OE ------ ---
5.14.0-611.13.1_lustre.el9.x86_64 #1
[ 3388.482551] Hardware name: Red Hat KVM/RHEL-AV, BIOS
1.16.0-4.module+el8.9.0+90052+d3bf71d8 04/01/2014
[ 3388.482552] RIP: 0010:kfree+0x6e/0x330
[ 3388.482585] Code: 80 48 01 d8 0f 82 d5 02 00 00 48 c7 c2 00 00 00 80
48 2b 15 fc eb 39 01 48 01 d0 48 c1 e8 0c 48 c1 e0 06 48 03 05 da eb 39
01 <48> 8b 50 08 49 89 c4 f6 c2 01 0f 85 d8 01 00 00 0f 1f 44 00 00 49
[ 3388.482586] RSP: 0018:ff40544f8d71fdd0 EFLAGS: 00010286
[ 3388.482591] RAX: ffd317785f002000 RBX: 0000000000080003 RCX:
0057088740207bf8
[ 3388.482592] RDX: 00e4b45740000000 RSI: ffffffffc19dacae RDI:
0000000000080003
[ 3388.482593] RBP: ff40544f8d71fe18 R08: 0000000000000000 R09:
0000000000000008
[ 3388.482594] R10: 0000000000000002 R11: 0000000000000001 R12:
ffffffffc1a3d660
[ 3388.482595] R13: ff1b4ba99ea17200 R14: ff1b4ba94be68c00 R15:
ff1b4ba99ea17230
[ 3388.482596] FS: 0000000000000000(0000) GS:ff1b4bc83fa40000(0000)
knlGS:0000000000000000
[ 3388.482598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3388.482599] CR2: ffd317785f002008 CR3: 00000001269e0003 CR4:
0000000000773ef0
[ 3388.482603] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 3388.482604] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 3388.482605] PKRU: 55555554
[ 3388.482606] Call Trace:
[ 3388.482607] <TASK>
[ 3388.482609] ? show_trace_log_lvl+0x1c4/0x2df
[ 3388.482635] ? show_trace_log_lvl+0x1c4/0x2df
[ 3388.482644] ? key_fini+0x4e/0x160 [obdclass]
[ 3388.482983] ? __die_body.cold+0x8/0xd
[ 3388.482987] ? page_fault_oops+0x132/0x170
[ 3388.483003] ? kernelmode_fixup_or_oops+0x83/0x110
[ 3388.483005] ? exc_page_fault+0xa7/0x150
[ 3388.483015] ? asm_exc_page_fault+0x22/0x30
[ 3388.483045] ? key_fini+0x4e/0x160 [obdclass]
[ 3388.483109] ? kfree+0x6e/0x330
[ 3388.483112] key_fini+0x4e/0x160 [obdclass]
[ 3388.483196] lu_context_fini+0x42/0x160 [obdclass]
[ 3388.483265] lu_env_fini+0x16/0x30 [obdclass]
[ 3388.483340] qsd_reint_main+0x2ec/0xb70 [lquota]
[ 3388.483427] ? __schedule+0x239/0x540
[ 3388.483433] ? _raw_spin_unlock_irqrestore+0xa/0x30
[ 3388.483439] ? __pfx_qsd_reint_main+0x10/0x10 [lquota]
[ 3388.483458] ? kthread+0x101/0x110
[ 3388.483468] kthread+0x101/0x110
[ 3388.483470] ? __pfx_kthread+0x10/0x10
[ 3388.483473] ret_from_fork+0x28/0x50
[ 3388.483486] </TASK>
[ 3388.483490] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE)
lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) mbcache
jbd2 lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE)
ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) rfkill
nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet
nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink
intel_rapl_msr intel_rapl_common intel_uncore_frequency_common nfit
libnvdimm snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg
snd_intel_sdw_acpi kvm_intel snd_hda_codec snd_hda_core snd_hwdep kvm
snd_seq snd_seq_device qxl drm_ttm_helper iTCO_wdt snd_pcm ttm
iTCO_vendor_support rapl snd_timer drm_exec drm_client_lib i2c_i801
drm_kms_helper snd mlx5_fwctl i2c_smbus lpc_ich virtio_balloon pcspkr
soundcore fwctl joydev drm fuse xfs libcrc32c mlx5_ib macsec mlx5_core
sd_mod ahci sg libahci libata crct10dif_pclmul crc32_pclmul crc32c_intel
mlxfw psample virtio_net tls
[ 3388.483681] virtio_console ghash_clmulni_intel virtio_scsi(E)
net_failover pci_hyperv_intf failover serio_raw dm_mirror dm_region_hash
dm_log dm_mod rpcrdma sunrpc rdma_ucm ib_uverbs ib_srpt ib_isert
iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi
ib_umad rdma_cm ib_ipoib iw_cm ib_cm ib_core
[ 3388.483758] CR2: ffd317785f002008
===================
We have tried to mount the other mds servers (which e2fsck found nothing
to fix) and those servers were not crashed. The mdt1 and mgt share the
same server in our setting.
Thank for any help that you can provide
Best regards,
NGUYEN Hoang Minh
HPC Researcher
More information about the lustre-discuss
mailing list