<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
The OST objects do not *require* extended attributes, only the MDT inodes (to hold file layouts, FIDs, etc).<br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Oct 23, 2025, at 02:37, Ihsan Ur Rahman <ihsanurr67@gmail.com> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div dir="ltr">Hell,
<div><br>
</div>
<div>Unfortunately the extended attributes were not copied during the backup. so we have lost the data. </div>
<div><br>
</div>
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div>Ihsan </div>
</div>
<br>
<div class="gmail_quote gmail_quote_container">
<div dir="ltr" class="gmail_attr">On Wed, Oct 22, 2025 at 8:36 PM Ihsan Ur Rahman <<a href="mailto:ihsanurr67@gmail.com">ihsanurr67@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Dear Rick,
<div>Thank you for your reply. Yes i have unmounted before the mounting with "lustre" </div>
<div><br>
</div>
<div>regards,</div>
<div>Ihsan </div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Oct 22, 2025 at 5:19 PM Mohr, Rick <<a href="mailto:mohrrf@ornl.gov" target="_blank">mohrrf@ornl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Just curious, after you mounted /dev/sdd as ldiskfs, did you unmount it before trying to mount it as type "lustre"? I'm wondering if the "File exists" error is a result of trying to double mount it.<br>
<br>
--Rick<br>
<br>
<br>
On Tue, Oct 21, 2025 at 12:06 AM Ihsan Ur Rahman <<a href="mailto:ihsanurr67@gmail.com" target="_blank">ihsanurr67@gmail.com</a> <mailto:<a href="mailto:ihsanurr67@gmail.com" target="_blank">ihsanurr67@gmail.com</a>> <mailto:<a href="mailto:ihsanurr67@gmail.com" target="_blank">ihsanurr67@gmail.com</a>
<mailto:<a href="mailto:ihsanurr67@gmail.com" target="_blank">ihsanurr67@gmail.com</a>>>> wrote:<br>
<br>
<br>
Hello Lustre community,<br>
<br>
<br>
One of the OSTs failed. We have copied the data using rsync and reformatted the OST with the following command:<br>
<br>
<br>
mkfs.lustre --ost --index=29 --replace --reformat --fsname=lustre --backfstype=ldiskfs --mgsnode=X.X.X.X@o2ib --mkfsoptions="-i 7000" /dev/sdd<br>
<br>
<br>
<br>
<br>
The OST has been deactivated on both the client side and the MDS side. We mounted it with the following command and copied the data back using rsync:<br>
<br>
<br>
mount -t ldiskfs /dev/sdd /mnt/ost29<br>
<br>
<br>
<br>
<br>
Once the copying was completed, I tried to mount it as a Lustre OST, but I encountered the following errors:<br>
<br>
<br>
# mount -t lustre /dev/sdd /mnt/29ost<br>
mount.lustre: mount /dev/sdd at /mnt/29ost failed: File exists<br>
<br>
<br>
<br>
<br>
When I tried a second time, I got this error:<br>
<br>
<br>
# mount -t lustre /dev/sdd /mnt/29ost<br>
mount.lustre: mount /dev/sdd at /mnt/29ost failed: Operation now in progress<br>
<br>
<br>
<br>
<br>
I also tried:<br>
<br>
<br>
# mount -t lustre -o abort_recov /dev/sdd /mnt/ost29<br>
mount.lustre: mount /dev/sdd at /mnt/ost29 failed: File exists<br>
<br>
<br>
<br>
<br>
On the OSS, when I run:<br>
<br>
<br>
# lctl dl | grep OST001d<br>
12 UP osd-ldiskfs lustre-OST001d-osd lustre-OST001d-osd_UUID 3<br>
<br>
<br>
<br>
<br>
The OST still appears. I tried to remove it using:<br>
<br>
<br>
lctl --device lustre-OST001d-osd cleanup<br>
lctl --device lustre-OST001d-osd detach<br>
<br>
<br>
<br>
<br>
However, when I run mount -t lustre /dev/sdd /mnt/ost29 again, OST001d appears once more.<br>
<br>
<br>
I have also found Lustre-related errors in the OSS kernel logs.<br>
<br>
<br>
<br>
<br>
[74585.030434] LustreError: 15c-8: MGC10.1x.x.x.132@o2ib: The configuration from log 'lustre-OST001d' failed (-115). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more
information.<br>
[74585.034979] LustreError: 8737:0:(obd_mount_server.c:1397:server_start_targets()) failed to start server lustre-OST001d: -115<br>
[74585.036522] LustreError: 8737:0:(obd_mount_server.c:1992:server_fill_super()) Unable to start targets: -115<br>
[74585.037997] LustreError: 8737:0:(obd_config.c:610:class_cleanup()) Device 13 not setup<br>
[74585.039532] Lustre: server umount lustre-OST001d complete<br>
[74585.039537] LustreError: 8737:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount /dev/sdd (-115)<br>
[74744.284257] LDISKFS-fs (sdd): file extents enabled, maximum tree depth=5<br>
[74745.013194] LDISKFS-fs (sdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc<br>
[74745.477281] LustreError: 8763:0:(obd_mount_server.c:110:server_register_mount()) Already registered lustre-OST001d<br>
[74745.478835] LustreError: 8763:0:(obd_mount_server.c:1992:server_fill_super()) Unable to start targets: -17<br>
[74745.480369] LustreError: 8763:0:(obd_mount_server.c:1600:server_put_super()) no obd lustre-OST001d<br>
[74746.236387] Lustre: server umount lustre-OST001d complete<br>
[74746.236393] LustreError: 8763:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount (-17)<br>
[74751.705099] LDISKFS-fs (sdd): file extents enabled, maximum tree depth=5<br>
[74752.436188] LDISKFS-fs (sdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc<br>
[74752.902198] LustreError: 8776:0:(llog_osd.c:987:llog_osd_next_block()) lustre-OST001d-osd: invalid llog tail at log id [0xb46f60d6:0xa1cf2803:0x0]:0 offset 20672 last_rec idx 0 tail idx 2250830848lrt len 2048 read_size 4288<br>
[74752.905177] LustreError: 8776:0:(llog.c:1419:llog_backup()) MGC10.19.4.132@o2ib: failed to backup log lustre-OST001d: rc = -22<br>
[74752.908120] LustreError: 8776:0:(mgc_request.c:1879:mgc_llog_local_copy()) MGC10.x.x.x.132@o2ib: failed to copy remote log lustre-OST001d: rc = -22<br>
[74752.916309] LustreError: 8788:0:(nodemap_storage.c:206:nodemap_cache_find_create()) cannot destroy nodemap index: rc = -115<br>
[74753.304620] LustreError: 8788:0:(obd_config.c:559:class_setup()) setup lustre-OST001d failed (-115)<br>
[74753.306167] LustreError: 8788:0:(obd_config.c:1835:class_config_llog_handler()) MGC10.19.4.132@o2ib: cfg command failed: rc = -115<br>
[74753.309113] Lustre: cmd=cf003 0:lustre-OST001d 1:dev 2:0 3:f<br>
<br>
<br>
[74753.309227] LustreError: 15c-8: MGC10.19.4.132@o2ib: The configuration from log 'lustre-OST001d' failed (-115). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br>
[74753.313693] LustreError: 8776:0:(obd_mount_server.c:1397:server_start_targets()) failed to start server lustre-OST001d: -115<br>
[74753.315270] LustreError: 8776:0:(obd_config.c:610:class_cleanup()) Device 13 not setup<br>
[75312.334648] LDISKFS-fs (sdd): file extents enabled, maximum tree depth=5<br>
[75313.081093] LDISKFS-fs (sdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc<br>
[75313.081112] LustreError: 8822:0:(osd_handler.c:7815:osd_mount()) lustre-OST001d-osd: failed to set lma on /dev/sdd root inode<br>
[75313.082671] LustreError: 8822:0:(obd_config.c:559:class_setup()) setup lustre-OST001d-osd failed (-30)<br>
[75313.084163] LustreError: 8822:0:(obd_mount.c:202:lustre_start_simple()) lustre-OST001d-osd setup error -30<br>
[75313.085645] LustreError: 8822:0:(obd_mount_server.c:1958:server_fill_super()) Unable to start osd on /dev/sdd: -30<br>
[75313.087122] LustreError: 8822:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount (-30)<br>
[75313.088583] LustreError: 8822:0:(obd_mount.c:1608:lustre_fill_super()) Skipped 1 previous similar message<br>
[75330.678571] LDISKFS-fs (sdd): file extents enabled, maximum tree depth=5<br>
[75331.407996] LDISKFS-fs (sdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc<br>
[75331.408015] LustreError: 8837:0:(osd_handler.c:7815:osd_mount()) lustre-OST001d-osd: failed to set lma on /dev/sdd root inode<br>
[75331.409564] LustreError: 8837:0:(obd_config.c:559:class_setup()) setup lustre-OST001d-osd failed (-30)<br>
[75331.411040] LustreError: 8837:0:(obd_mount.c:202:lustre_start_simple()) lustre-OST001d-osd setup error -30<br>
[75331.412528] LustreError: 8837:0:(obd_mount_server.c:1958:server_fill_super()) Unable to start osd on /dev/sdd: -30
<br>
<br>
<br>
<br>
<br>
<br>
<br>
You guys' support will be highly appreciated as we are stuck very badly. lustre version is 2.12.6 .
<br>
<br>
<br>
<br>
<br>
Regards,<br>
<br>
<br>
<br>
<br>
Ihsan Ur Rahman <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>Cheers, Andreas</div>
<div>—</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud/DDN</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br>
</body>
</html>