[lustre-discuss] OST failed to mount,massive duplicate FID

[姚秋玲]

Hi Lustre Community,


I'm asking for help with a severe Lustre OST issue after backend disk array failure.



The storage array failure caused 4 out of 6 OSTs offline.
After e2fsck repair, 5 OSTs recovered and work normally.
Only dybfs2-OST0003(device /dev/mapper/mpathd) fails to mount with error -17 File exists.

May 11 14:54:40 dybfs16 kernel: LustreError: 3663:0:(osd_oi.c:762:osd_oi_insert()) dm-2: the FID [0x200000003:0x3:0x0] is used by two objects: 3080193/2625393380 35842/449209919

May 11 14:54:40 dybfs16 kernel: LustreError: 3663:0:(obd_config.c:559:class_setup()) setup dybfs2-OST0003 failed (-17)

May 11 14:54:40 dybfs16 kernel: LustreError: 3663:0:(obd_config.c:1835:class_config_llog_handler()) MGC192.168.50.23 at tcp: cfg command failed: rc = -17

May 11 14:54:40 dybfs16 kernel: Lustre:    cmd=cf003 0:dybfs2-OST0003  1:dev  2:0  3:f  

May 11 14:54:40 dybfs16 kernel: LustreError: 15c-8: MGC192.168.50.23 at tcp: The configuration from log 'dybfs2-OST0003' failed (-17). This may be the result of communication errors between 

this node and the MGS, a bad configuration, or other errors. See the syslog for more information.

May 11 14:54:40 dybfs16 kernel: LustreError: 3526:0:(obd_mount_server.c:1397:server_start_targets()) failed to start server dybfs2-OST0003: -17

May 11 14:54:40 dybfs16 kernel: LustreError: 3526:0:(obd_mount_server.c:1992:server_fill_super()) Unable to start targets: -17

May 11 14:54:40 dybfs16 kernel: LustreError: 3526:0:(obd_config.c:610:class_cleanup()) Device 3 not setup

May 11 14:54:40 dybfs16 kernel: Lustre: server umount dybfs2-OST0003 complete

May 11 14:54:40 dybfs16 kernel: LustreError: 3526:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount /dev/mapper/mpathd (-17)



Sample kernel errors:

plaintext
May 11 12:23:16 dybfs16 kernel: LustreError: 4127:0:(osd_oi.c:762:osd_oi_insert()) dm-3: the FID [0x200000003:0x3:0x0] is used by two objects: 3080193/2625393380 86/3624058548
May 11 12:36:30 dybfs16 kernel: LustreError: 3847:0:(osd_oi.c:762:osd_oi_insert()) dm-3: the FID [0x200000003:0x3:0x0] is used by two objects: 3080193/2625393380 87/358952283
May 11 12:54:56 dybfs16 kernel: LustreError: 3479:0:(osd_oi.c:762:osd_oi_insert()) dm-2: the FID [0x200000003:0x3:0x0] is used by two objects: 3080193/2625393380 94/3669235547
May 11 13:09:27 dybfs16 kernel: LustreError: 3846:0:(osd_oi.c:762:osd_oi_insert()) dm-3: the FID [0x200000003:0x3:0x0] is used by two objects: 3080193/2625393380 33537/2387867276
May 11 13:22:03 dybfs16 kernel: LustreError: 4178:0:(osd_oi.c:762:osd_oi_insert()) dm-3: the FID [0x200000003:0x3:0x0] is used by two objects: 3080193/2625393380 33538/541222096
May 11 13:29:56 dybfs16 kernel: LustreError: 3479:0:(osd_oi.c:762:osd_oi_insert()) dm-3: the FID [0x200000003:0x3:0x0] is used by two objects: 3080193/2625393380 33666/126694764
May 11 13:45:06 dybfs16 kernel: LustreError: 4172:0:(osd_oi.c:762:osd_oi_insert()) dm-3: the FID [0x200000003:0x3:0x0] is used by two objects: 3080193/2625393380 33667/133493056
May 11 14:03:50 dybfs16 kernel: LustreError: 3556:0:(osd_oi.c:762:osd_oi_insert()) dm-3: the FID [0x200000003:0x3:0x0] is used by two objects: 3080193/2625393380 33668/417086394
May 11 14:14:43 dybfs16 kernel: LustreError: 3562:0:(osd_oi.c:762:osd_oi_insert()) dm-4: the FID [0x200000003:0x3:0x0] is used by two objects: 3080193/2625393380 33669/3676336670
May 11 14:36:10 dybfs16 kernel: LustreError: 4572:0:(osd_oi.c:762:osd_oi_insert()) dm-4: the FID [0x200000003:0x3:0x0] is used by two objects: 3080193/2625393380 35841/3333636550


May 11 14:54:40 dybfs16 kernel: LustreError: 3663:0:(osd_oi.c:762:osd_oi_insert()) dm-2: the FID [0x200000003:0x3:0x0] is used by two objects: 3080193/2625393380 35842/449209919

Used debugfs to locate conflicting inodes and unlink the duplicate objects(86,87~35842)
Ran e2fsck -fy /dev/mapper/mpathd multiple times,fix them to lost+found
Rebooted node and restarted Lustre stack


Current Lustre module status (lctl dl)
plaintext
  0 UP osd-ldiskfs dybfs2-OST0003-osd dybfs2-OST0003-osd_UUID 3
  1 UP osd-ldiskfs dybfs2-OST0000-osd dybfs2-OST0000-osd_UUID 4
  2 UP ost OSS OSS_uuid 2
  3 UP mgc MGC192.168.50.23 at tcp 530c1b9f-3835-9edb-6b97-f9b3b7edca49 4
  4 UP obdfilter dybfs2-OST0000 dybfs2-OST0000_UUID 140
  5 UP lwp dybfs2-MDT0000-lwp-OST0000 dybfs2-MDT0000-lwp-OST0000_UUID 4
  6 UP osd-ldiskfs dybfs2-OST0001-osd dybfs2-OST0001-osd_UUID 4
  7 UP obdfilter dybfs2-OST0001 dybfs2-OST0001_UUID 214
  8 UP lwp dybfs2-MDT0000-lwp-OST0001 dybfs2-MDT0000-lwp-OST0001_UUID 4
  9 UP osd-ldiskfs dybfs2-OST0002-osd dybfs2-OST0002-osd_UUID 4
 10 UP obdfilter dybfs2-OST0002 dybfs2-OST0002_UUID 216
 11 UP lwp dybfs2-MDT0000-lwp-OST0002 dybfs2-MDT0000-lwp-OST0002_UUID 4
 12 UP osd-ldiskfs dybfs2-OST0004-osd dybfs2-OST0004-osd_UUID 4
 13 UP obdfilter dybfs2-OST0004 dybfs2-OST0004_UUID 384
 14 UP lwp dybfs2-MDT0000-lwp-OST0004 dybfs2-MDT0000-lwp-OST0004_UUID 4
 15 UP osd-ldiskfs dybfs2-OST0005-osd dybfs2-OST0005-osd_UUID 4
 16 UP obdfilter dybfs2-OST0005 dybfs2-OST0005_UUID 232
 17 UP lwp dybfs2-MDT0000-lwp-OST0005 dybfs2-MDT0000-lwp-OST0005_UUID 4

All other OSTs are loaded and running normally, only OST0003 cannot complete full setup and mount.
Questions
The OSD OI index of OST0003 seems severely corrupted, the same FID maps to hundreds of inodes. Is there a safe way to rebuild or reset the OI index in Lustre 2.12.5 without reformatting the whole OST?
Is there an official tool or procedure to batch clean up massive duplicate FID conflicts?
If in-place repair is impossible, what is the safest step-by-step procedure to recreate and re-add this OST to the existing filesystem with minimal impact on current data?
Any guidance or experience sharing will be greatly appreciated.


Environment
Lustre version: 2.12.5
OS & kernel: 3.10.0-1127.8.2.el7_lustre.x86_64
Backend filesystem: ldiskfs


Best regards,
Qiuling YAO
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260511/087c8b49/attachment.html>