<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On May 31, 2022, at 13:01, William D. Colburn <<a href="mailto:wcolburn@nrao.edu" class="">wcolburn@nrao.edu</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="">We had a filesystem corruption back in February, and we've been trying<br class="">
to salvage things since then. I've spent the past month slowly draining<br class="">
the corrupt OST, and over the weekend it finally finished. An lfs find<br class="">
on the filesystem says that thhow ere are no files stored on that OST. The<br class="">
OST is 100% full, and if I mount it as an ldiskfs I can see a little<br class="">
over five millions files in O/*/*.</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>How did you drain the OST? Was the OST totally deactivated, or "max_create_count=0"?</div>
<div>If it was deactivated, then this will prevent OST objects from being destroyed when the</div>
<div>MDT inode is deleted.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">Most of them have numbers as names, and some of them are named LAST_ID.</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>This is normal. The number is the object ID. You can check the OST objects with</div>
<div>ll_decode_filter_fid (when mounted as type ldiskfs) to report the parent MDT FID</div>
<div>that the object belongs/belonged to. Then "lfs fid2path" can be used to check if the</div>
<div>file still exists and/or if the OST object is still part of the layout (which it should not be).</div>
<div><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div class="">All of the numbered files seem to be user data, with owners, and real data in them</div>
</div>
</blockquote>
<blockquote type="cite" class="">
<div class="">
<div class="">(based on ls and the find command)<br class="">
<br class="">
I would like to clean out this OST and readd it to lustre, but I'm<br class="">
unsure of how to best approach this. I see several options:<br class="">
<br class="">
OPTION ONE: run lfsck against the entire filesystem with the full and<br class="">
previously corrupt OST mounted.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
This is unnecessary, since these objects are not used. With the "--orphan"</div>
<div>option it will link the OST objects into $mount/.lustre/lost+found where they</div>
<div>could be deleted, essentially the same as #4 below, but it is easier to just</div>
<div>delete the objects directly if you know they are not needed.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">OPTION TWO: run lfsck against only the corrupt OST in the hopes that<br class="">
cleans up all of the orphans on that OST.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
This won't help, since the OST LFSCK would not attach the object into the</div>
<div>visible namespace, so it won't really change the situation.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">OPTION THREE: mounted as ldiskfs remove O/*/[1234567890]*[1234567890]<br class="">
and then remount the file system.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
This would be one option. Note that with DNE the OST object names will be</div>
<div>in hex, so the above regexp would not catch all objects.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">OPTION FOUR: newfs the bad OST and readd it losing the old index.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
This would also work, if you use the "--replace" option when formatting.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">We tried option one once before, and it killed cluster jobs because it<br class="">
made files unreadable while they were in use. Option two might avoid<br class="">
that since it would not be affecting existing files. Option three<br class="">
sounds like it will work based on my limited knowledge of how lustre<br class="">
works, and would probably be the most expedient method. Option four is<br class="">
annoying because it leaves a hole in the lustre that is upsetting to our<br class="">
OCD tendencies.<br class="">
<br class="">
Any and all advice is appreciated here. Thank you.<br class="">
<br class="">
--Schlake<br class="">
Sysadmin IV, NRAO<br class="">
Work: 575-835-7281 (BACK IN THE OFFICE!)<br class="">
Cell: 575-517-5668 (out of work hours)<br class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>