<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Sep 21, 2023, at 16:06, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] <<a href="mailto:darby.vicker-1@nasa.gov" class="">darby.vicker-1@nasa.gov</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">I knew an lfsck would identify the orphaned objects. That’s great that it will move those objects to an area we can triage. With ownership still intact (and I assume time stamps too), I think this will be helpful for
at least some of the users to recover some of their data. Thanks Andreas. <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">I do have another question. Even with the MDT loss, the top level user directories on the file system are still showing current modification times. I was a little surprised to see this – my expectation was that the
most current time would be from the snapshot that we accidentally reverted to, 6/20/2023 in this case. Does this make sense? </span></div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
The timestamps of the directories are only stored on the MDT (unlike regular files which keep of the timestamp on both the MDT and OST). Is it possible that users (or possibly recovered clients with existing mountpoints) have started to access the filesystem
in the past few days since it was recovered, or an admin was doing something that would have caused the directories to be modified?</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div>Is it possible you have a newer copy of the MDT than you thought?</div>
<div><span style="font-family: "Courier New"; font-size: 11pt;" class=""> </span><br class="">
<blockquote type="cite" class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Courier New";" class="">[dvicker@dvicker ~]$ ls -lrt /ephemeral/ | tail<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Courier New";" class=""> 4 drwx------ 2 abjuarez abjuarez 4096 Sep 12 13:24 abjuarez/<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Courier New";" class=""> 4 drwxr-x--- 2 ksmith29 ksmith29 4096 Sep 13 15:37 ksmith29/<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Courier New";" class=""> 4 drwxr-xr-x 55 bjjohn10 bjjohn10 4096 Sep 13 16:36 bjjohn10/<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Courier New";" class=""> 4 drwxrwx--- 3 cbrownsc ccp_fast 4096 Sep 14 12:27 cbrownsc/<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Courier New";" class=""> 4 drwx------ 3 fgholiza fgholiza 4096 Sep 18 06:41 fgholiza/<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Courier New";" class=""> 4 drwx------ 5 mtfoste2 mtfoste2 4096 Sep 19 11:35 mtfoste2/<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Courier New";" class=""> 4 drwx------ 4 abenini abenini 4096 Sep 19 15:33 abenini/<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Courier New";" class=""> 4 drwx------ 9 pdetremp pdetremp 4096 Sep 19 16:49 pdetremp/</span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: "Courier New";" class="">[dvicker@dvicker ~]$<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="" class="">Andreas Dilger <<a href="mailto:adilger@whamcloud.com" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">adilger@whamcloud.com</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Thursday, September 21, 2023 at 2:33 PM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <<a href="mailto:darby.vicker-1@nasa.gov" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">darby.vicker-1@nasa.gov</a>><br class="">
<b class="">Cc:<span class="Apple-converted-space"> </span></b>"<a href="mailto:lustre-discuss@lists.lustre.org" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>[EXTERNAL] Re: [lustre-discuss] Data recovery with lost MDT data<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
</div>
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" align="left" style="border: 1.5pt solid black;">
<tbody class="">
<tr class="">
<td width="100%" style="width: 1167.59375px; border: none; background-color: rgb(255, 235, 156); padding: 3.75pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 10pt;" class="">CAUTION:</span></b><span style="" class=""><span class="Apple-converted-space"> </span></span><span style="font-size: 10pt;" class="">This email originated from outside of NASA. Please take care when clicking
links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.</span><span style="" class=""></span><o:p class=""></o:p></div>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: Calibri, sans-serif;">
<br class="">
<br class="">
<o:p class=""></o:p></p>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
In the absence of backups, you could try LFSCK to link all of the orphan OST objects into .lustre/lost+found (see lctl-lfsck_start.8 man page for details).<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
The data is still in the objects, and they should have UID/GID/PRJID assigned (if used) but they have no filenames. It would be up to you to make e.g. per-user lost+found directories in their home directories and move the files where they could access them
and see if they want to keep or delete the files. <span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
How easy/hard this is to do depends on whether the files have any content that can help identify them. <o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: Calibri, sans-serif;">
There was a Lustre hackathon project to save the Lustre JobID in a "user.job" xattr on every object, exactly to help identify the provenance of files after the fact (regardless of whether there is corruption), but it only just landed to master and will be in
2.16. That is cold comfort, but would help in the future. <o:p class=""></o:p></p>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
Cheers, Andreas<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<br class="">
<br class="">
<o:p class=""></o:p></div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: Calibri, sans-serif;">
On Sep 20, 2023, at 15:34, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>> wrote:<o:p class=""></o:p></p>
</blockquote>
</div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Hello,</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">We have recently accidentally deleted some of our MDT data. I think its gone for good but looking for advice to see if there is any way to recover. Thoughts appreciated. </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">We run two LFS’s on the same set of hardware. We didn’t set out to do this, but it kind of evolved. The original setup was only a single filesystem and was all ZFS – MDT and OST’s. Eventually, we had some small file
workflows that we wanted to get better performance on. To address this, we stood up another filesystem on the same hardware and used a an ldiskfs MDT. However, since were already using ZFS, under the hood the storage device we build the ldisk MDT on comes
from ZFS. That gets presented to the OS as /dev/zd0. We do a nightly backup of the MDT by cloning the ZFS dataset (this creates /dev/zd16, for whatever reason), snapshot the clone, mount that as ldiskfs, tar up the data and then destroy the snapshot and
clone. Well, occasionally this process gets interrupted, leaving the ZFS snapshot and clone hanging around. This is where things go south. Something happens that swaps the clone with the primary dataset. ZFS says you’re working with the primary but its
really the clone, and via versa. This happened about a year ago and we caught it, were able to “zfs promote” to swap them back and move on. More details on the ZFS and this mailing list here. </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><a href="https://zfsonlinux.topicbox.com/groups/zfs-discuss/Tcb8a3ef663db0031-M5a79e71768b20b2389efc4a4" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">https://zfsonlinux.topicbox.com/groups/zfs-discuss/Tcb8a3ef663db0031-M5a79e71768b20b2389efc4a4</a></span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2022-June/018154.html" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2022-June/018154.html</a></span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">It happened again earlier this week but we didn’t remember to check this and, in an effort to get the backups going again, destroyed what we thought were the snapshot and clone. In reality, we destroyed the primary dataset.
Even more unfortunately, the stale “snapshot” was about 3 months old. This stale snapshot was also preventing our MDT backups from running so we don’t have those to restore from either. (I know, we need better monitoring and alerting on this, we learned
that lesson the hard way. We had it in place after the June 2022 incident, it just wasn’t working properly.) So at the end of the day, the data lives on the OST’s we just can’t access it due to the lost metadata. Is there any chance at data recovery. I
don’t think so but want to explore any options. <span class="Apple-converted-space"> </span></span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Darby</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a><br class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span></div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>