[lustre-discuss] Unable to mount the OST

[Ihsan Ur Rahman]

Hello Lustre community,

One of the OSTs failed. We have copied the data using rsync and reformatted
the OST with the following command:

mkfs.lustre --ost --index=29 --replace --reformat --fsname=lustre
--backfstype=ldiskfs --mgsnode=X.X.X.X at o2ib --mkfsoptions="-i 7000" /dev/sdd


The OST has been deactivated on both the client side and the MDS side. We
mounted it with the following command and copied the data back using rsync:

mount -t ldiskfs /dev/sdd /mnt/ost29


Once the copying was completed, I tried to mount it as a Lustre OST, but I
encountered the following errors:

# mount -t lustre /dev/sdd /mnt/29ost
mount.lustre: mount /dev/sdd at /mnt/29ost failed: File exists


When I tried a second time, I got this error:

# mount -t lustre /dev/sdd /mnt/29ost
mount.lustre: mount /dev/sdd at /mnt/29ost failed: Operation now in progress


I also tried:

# mount -t lustre -o abort_recov /dev/sdd /mnt/ost29
mount.lustre: mount /dev/sdd at /mnt/ost29 failed: File exists


On the OSS, when I run:

# lctl dl | grep OST001d
12 UP osd-ldiskfs lustre-OST001d-osd lustre-OST001d-osd_UUID 3


The OST still appears. I tried to remove it using:

lctl --device lustre-OST001d-osd cleanup
lctl --device lustre-OST001d-osd detach


However, when I run mount -t lustre /dev/sdd /mnt/ost29 again, OST001d
appears once more.

I have also found Lustre-related errors in the OSS kernel logs.

[74585.030434] LustreError: 15c-8: MGC10.1x.x.x.132 at o2ib: The configuration
> from log 'lustre-OST001d' failed (-115). This may be the result of
> communication errors between this node and the MGS, a bad configuration, or
> other errors. See the syslog for more information.
> [74585.034979] LustreError:
> 8737:0:(obd_mount_server.c:1397:server_start_targets()) failed to start
> server lustre-OST001d: -115
> [74585.036522] LustreError:
> 8737:0:(obd_mount_server.c:1992:server_fill_super()) Unable to start
> targets: -115
> [74585.037997] LustreError: 8737:0:(obd_config.c:610:class_cleanup())
> Device 13 not setup
> [74585.039532] Lustre: server umount lustre-OST001d complete
> [74585.039537] LustreError: 8737:0:(obd_mount.c:1608:lustre_fill_super())
> Unable to mount /dev/sdd (-115)
> [74744.284257] LDISKFS-fs (sdd): file extents enabled, maximum tree depth=5
> [74745.013194] LDISKFS-fs (sdd): mounted filesystem with ordered data
> mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
> [74745.477281] LustreError:
> 8763:0:(obd_mount_server.c:110:server_register_mount()) Already registered
> lustre-OST001d
> [74745.478835] LustreError:
> 8763:0:(obd_mount_server.c:1992:server_fill_super()) Unable to start
> targets: -17
> [74745.480369] LustreError:
> 8763:0:(obd_mount_server.c:1600:server_put_super()) no obd lustre-OST001d
> [74746.236387] Lustre: server umount lustre-OST001d complete
> [74746.236393] LustreError: 8763:0:(obd_mount.c:1608:lustre_fill_super())
> Unable to mount  (-17)
> [74751.705099] LDISKFS-fs (sdd): file extents enabled, maximum tree depth=5
> [74752.436188] LDISKFS-fs (sdd): mounted filesystem with ordered data
> mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
> [74752.902198] LustreError: 8776:0:(llog_osd.c:987:llog_osd_next_block())
> lustre-OST001d-osd: invalid llog tail at log id
> [0xb46f60d6:0xa1cf2803:0x0]:0 offset 20672 last_rec idx 0 tail idx
> 2250830848lrt len 2048 read_size 4288
> [74752.905177] LustreError: 8776:0:(llog.c:1419:llog_backup())
> MGC10.19.4.132 at o2ib: failed to backup log lustre-OST001d: rc = -22
> [74752.908120] LustreError:
> 8776:0:(mgc_request.c:1879:mgc_llog_local_copy()) MGC10.x.x.x.132 at o2ib:
> failed to copy remote log lustre-OST001d: rc = -22
> [74752.916309] LustreError:
> 8788:0:(nodemap_storage.c:206:nodemap_cache_find_create()) cannot destroy
> nodemap index: rc = -115
> [74753.304620] LustreError: 8788:0:(obd_config.c:559:class_setup()) setup
> lustre-OST001d failed (-115)
> [74753.306167] LustreError:
> 8788:0:(obd_config.c:1835:class_config_llog_handler()) MGC10.19.4.132 at o2ib:
> cfg command failed: rc = -115
> [74753.309113] Lustre:    cmd=cf003 0:lustre-OST001d  1:dev  2:0  3:f
>
> [74753.309227] LustreError: 15c-8: MGC10.19.4.132 at o2ib: The configuration
> from log 'lustre-OST001d' failed (-115). This may be the result of
> communication errors between this node and the MGS, a bad configuration, or
> other errors. See the syslog for more information.
> [74753.313693] LustreError:
> 8776:0:(obd_mount_server.c:1397:server_start_targets()) failed to start
> server lustre-OST001d: -115
> [74753.315270] LustreError: 8776:0:(obd_config.c:610:class_cleanup())
> Device 13 not setup
> [75312.334648] LDISKFS-fs (sdd): file extents enabled, maximum tree depth=5
> [75313.081093] LDISKFS-fs (sdd): mounted filesystem with ordered data
> mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
> [75313.081112] LustreError: 8822:0:(osd_handler.c:7815:osd_mount())
> lustre-OST001d-osd: failed to set lma on /dev/sdd root inode
> [75313.082671] LustreError: 8822:0:(obd_config.c:559:class_setup()) setup
> lustre-OST001d-osd failed (-30)
> [75313.084163] LustreError: 8822:0:(obd_mount.c:202:lustre_start_simple())
> lustre-OST001d-osd setup error -30
> [75313.085645] LustreError:
> 8822:0:(obd_mount_server.c:1958:server_fill_super()) Unable to start osd on
> /dev/sdd: -30
> [75313.087122] LustreError: 8822:0:(obd_mount.c:1608:lustre_fill_super())
> Unable to mount  (-30)
> [75313.088583] LustreError: 8822:0:(obd_mount.c:1608:lustre_fill_super())
> Skipped 1 previous similar message
> [75330.678571] LDISKFS-fs (sdd): file extents enabled, maximum tree depth=5
> [75331.407996] LDISKFS-fs (sdd): mounted filesystem with ordered data
> mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
> [75331.408015] LustreError: 8837:0:(osd_handler.c:7815:osd_mount())
> lustre-OST001d-osd: failed to set lma on /dev/sdd root inode
> [75331.409564] LustreError: 8837:0:(obd_config.c:559:class_setup()) setup
> lustre-OST001d-osd failed (-30)
> [75331.411040] LustreError: 8837:0:(obd_mount.c:202:lustre_start_simple())
> lustre-OST001d-osd setup error -30
> [75331.412528] LustreError:
> 8837:0:(obd_mount_server.c:1958:server_fill_super()) Unable to start osd on
> /dev/sdd: -30



You guys' support will be highly appreciated as we are stuck very badly.
lustre version is 2.12.6 .

Regards,

Ihsan Ur Rahman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20251021/309b6f93/attachment.htm>