<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="">
<div>
<div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">On
Mar 29, 2020, at 20:04, Gong-Do Hwang <</span><a href="mailto:grover.hwang@gmail.com" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;">grover.hwang@gmail.com</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">>
wrote:</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Thanks Andreas,
<div class=""><br class="">
</div>
<div class="">I ran "mkfs.lustre --ost --reformat --fsname lfs_home --index 6 --mgsnode 10.10.0.10@o2ib --servicenode 10.10.0.13@o2ib --failnode 10.10.0.14@o2ib /dev/mapper/mpathx", and at that time /dev/mapper/mpathx was mounted and served as an OST under
FS lfs. And the FS lfs ran well until I umount the /dev/mapper/mpathx in order to restart the mgt/mgs.<br class="">
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
The issue here is that the "--reformat" option will override the checks if a filesystem already exists on the device. That should not normally be used.</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class=""><br class="">
And after I re-mounted the ost I got the msg "mount.lustre FATAL: failed to write local files: Invalid argument</div>
mount.lustre: mount /dev/mapper/mpathx at /lfs/ost8 failed: Invalid argument" and the tunefs.luster --dryrun /dev/mapper/mapthx output is "tunefs.lustre --dryrun /dev/mapper/mpathx<br class="">
checking for existing Lustre data: found<br class="">
Reading CONFIGS/mountdata<br class="">
<br class="">
Read previous values:<br class="">
Target: lfs-OST0008<br class="">
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
This shows that the device was previously part of the "lfs" filesystem at index 8. While it is possible to change the filesystem name, the OST index should never change, so there is no tool for this.</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
Two things need to be done. You can rewrite the filesystem label with "e2label /dev/mapper/mpathx lfs-OST0008". Then you need to rebuild the "CONFIGS/mountdata" file.</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
The easiest way to generate a new mountdata file would be to run "mkfs.lustre" with the same options as the original OST on a temporary device (e.g. loopback device) but add in the "--replace" option so that the OST doesn't try to add itself to the filesystem
as a new OST. Then mount the temporary and original OSTs as type ldiskfs and copy the file CONFIGS/mountdata from temp to original OST to replace the broken one (probably a good idea to make a backup first).</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
Hopefully with these two changes you can mount your OST again.</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
Cheers, Andreas</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">Index: 6<br class="">
Lustre FS: lfs_home<br class="">
Mount type: ldiskfs<br class="">
Flags: 0x1042<br class="">
(OST update no_primnode )<br class="">
Persistent mount opts: ,errors=remount-ro<br class="">
Parameters: mgsnode=10.10.0.10@o2ib failover.node=10.10.0.13@o2ib:10.10.0.14@o2ib<br class="">
<br class="">
<br class="">
Permanent disk data:<br class="">
Target: lfs_home-OST0006<br class="">
Index: 6<br class="">
Lustre FS: lfs_home<br class="">
Mount type: ldiskfs<br class="">
Flags: 0x1042<br class="">
(OST update no_primnode )<br class="">
Persistent mount opts: ,errors=remount-ro<br class="">
Parameters: mgsnode=10.10.0.10@o2ib failover.node=10.10.0.13@o2ib:10.10.0.14@o2ib"<br class="">
<br class="">
I guess I actually have run the mkfs command twice so the Lustre FS in the previous value became lfs_home(originally is lfs).<br class="">
<br class="">
I tried to mount the partition use backup superblocks and all of them are empty. But from the dumpe2fs info,
<div class="">"Inode count: 41943040<br class="">
Block count: 10737418240<br class="">
Reserved block count: 536870912<br class="">
Free blocks: 1459812475<br class="">
Free inodes: 39708575" </div>
<div class="">seems there is still data on it.</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
The backup superblocks are for the underlying ext4/ldiskfs filesystem, so are not really related to this problem.</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">So my problem is, if the data on the partition is still intact, is there any way I can rebuild the file index? And is there anyway I can rewrite the CONFIGS/mountdata back to its original values?</div>
<div class="">Sorry for the lengthy messages and really appreciate your help!</div>
<div class=""><br class="">
</div>
<div class="">Best Regards,</div>
<div class=""><br class="">
</div>
<div class="">Grover</div>
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Mar 30, 2020 at 7:14 AM Andreas Dilger <<a href="mailto:adilger@whamcloud.com" target="_blank" class="">adilger@whamcloud.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div class="">It would be useful if you provided the actual error messages, so we can see where the problem is.
<div class=""><br class="">
</div>
<div class="">What command did you run on the OST?<br class="">
<div class=""><br class="">
</div>
<div class="">Does the OST still show that it has data in it (e.g. "df" or "dumpe2fs -h" shows lots of used blocks)?<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Mar 25, 2020, at 10:05, Gong-Do Hwang <<a href="mailto:grover.hwang@gmail.com" target="_blank" class="">grover.hwang@gmail.com</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">Dear Lustre,
<div class=""><br class="">
</div>
<div class="">Months ago when I tried to add a new disk to my new Lustre FS, I accidentally target the mkfs.lustre to a then mounted OST partition of another Lustre FS. Weird enough the command passed through, and without paying attention to it, I umount the
partition months later and couldn't mount it back, then I realized the mkfs.lustre command was legit.</div>
<div class=""><br class="">
</div>
<div class="">But my old lustre FS worked well through these months, so I guess the data in that OST is still there. But now the permanent CONFIG/mountdata is the new one, and I can still see my old config in the previous value.</div>
<div class=""><br class="">
</div>
<div class="">My question is is there any way I can write back the old CONFIG/mountdata and still keep all my files in that OST?</div>
<div class=""><br class="">
</div>
<div class="">I am using Luster 2.13.0 for my mgs/mdt/ost</div>
<div class=""><br class="">
</div>
<div class="">Thanks for your help and I really appreciate it!</div>
<div class=""><br class="">
</div>
<div class="">Grover</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>