<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Sep 26, 2023, at 09:33, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] <<a href="mailto:darby.vicker-1@nasa.gov" class="">darby.vicker-1@nasa.gov</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I tried statx from lustre-tests, but apparently don’t have it built correctly:<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
[root@hpfs-fsl-lmon0 ~]# statx<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Skip: system does not support statx syscall.<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
[root@hpfs-fsl-lmon0 ~]#<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
The stat isn’t too important right now – if I can get a smaller group of files separated into a subdirectory, everything is fast again. So I’m<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I also tried moving a group of files at once but that didn’t help either. Tested this by manually making a test script that does a “mv file1 file2 … fileN subdir1” on 1000 files. That took about 7.5 hours – the same 30 seconds per file. <o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
OK, looks like xargs is the answer. <span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 0.5in;" class="">
subdir=/scratch-lustre/work/dvicker/lost+found/dir3<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 0.5in;" class="">
mkdir $subdir<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 0.5in;" class="">
head -1000 lost+found.txt | xargs mv -t $subdir<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Moving 1000 files like this takes only about 80 seconds. I would think this would be equivalent to the manual “mv file1 file2 … fileN subdir1” method but its not. An ideas why this is so much faster? </div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
It could be the order in which the files are being processed. "find ... | head -1000" is generating a list of files in "readdir" order, which is the hashed order that they appear in large directory blocks, and probably only modifies a dozen blocks in the filesystem.
Using "mv file1 file2 ..." is probably in "sorted by filename" order, which looks nice but causes all kinds of random IO operations when updating a thousand different directory blocks to move them.</div>
<div><br class="">
</div>
<div>You wouldn't happen to be running this old filesystem with the MDT on HDD devices? That would make a *huge* difference in performance from "mostly linear" to "mostly random", otherwise I can't imagine there is a huge difference for an NVMe MDT. Possibly,
the extra layering of ldiskfs on top of ZFS is also causing it heartburn, especially if your zvol is not configured with 4KiB recordsize, since that will also cause a lot of write inflation.</div>
<div><br class="">
</div>
<div>Cheers, Andreas</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Andreas Dilger <<a href="mailto:adilger@whamcloud.com" style="color: blue; text-decoration: underline;" class="">adilger@whamcloud.com</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Monday, September 25, 2023 at 8:19 PM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <<a href="mailto:darby.vicker-1@nasa.gov" style="color: blue; text-decoration: underline;" class="">darby.vicker-1@nasa.gov</a>><br class="">
<b class="">Cc:<span class="Apple-converted-space"> </span></b>"<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [lustre-discuss] [BULK] Re: [EXTERNAL] Re: Data recovery with lost MDT data<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" align="left" style="border: 1.5pt solid black;">
<tbody class="">
<tr class="">
<td width="100%" style="width: 1167.3125px; border: none; background-color: rgb(255, 235, 156); padding: 3.75pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 10pt;" class="">CAUTION:</span></b><span style="" class=""><span class="Apple-converted-space"> </span></span><span style="font-size: 10pt;" class="">This email originated from outside of NASA. Please take care when clicking
links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.</span><span style="" class=""></span><o:p class=""></o:p></div>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<br class="">
<br class="">
<o:p class=""></o:p></p>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Probably using "stat" on each file is slow, since this is getting the file size from each OST object. You could try the "xstat" utility in the lustre-tests RPM (or build it directly) as it will only query the MDS for the requested attributes (owner at minimum).<span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Then you could split into per-date directories in a separate phase, if needed, run in parallel.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I can't suggest anything about the 13M entry directory, but it _should_ be much faster than 1 file per 30s even at that size. I suspect that the script is still doing something bad, since shell and GNU utilities are terrible for doing extra stat/cd/etc five
times on each file that is accessed, renamed, etc.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
You would be better off to use "find -print" to generate the pathnames and then operate on those, maybe with "xargs -P" and/or run multiple scripts in parallel on chunks of the file?<o:p class=""></o:p></p>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Cheers, Andreas<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<br class="">
<br class="">
<o:p class=""></o:p></div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
On Sep 25, 2023, at 17:34, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] <<a href="mailto:darby.vicker-1@nasa.gov" style="color: blue; text-decoration: underline;" class="">darby.vicker-1@nasa.gov</a>> wrote:<o:p class=""></o:p></p>
</blockquote>
</div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I’ve been attempting to move these lost+found files into subdirectories by user and date but I’m running into issues. <o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
My initial attempt was to loop through each file in .lustre/lost+found, stat the file and then move it to a subdirectory. However, both the stat and the move command are each taking about 30 seconds. With 13 million files, this isn’t going to work as that
would be about 25 years to organize the files. :) If it makes a difference, the bash script is simple:<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">source=/scratch-lustre/.lustre/lost+found/MDT0000</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">dest=/scratch-lustre/work</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""># Cd to the source</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">cd $source</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""># Loop through files</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">for file in * ; do</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> echo "file: $file"</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> echo " stat-ing file"</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> time read -r user date time <<< $( stat --format "%U %y" $file )</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> echo " user='$user' date='$date' time='$time'"</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> # Build the new destination in the user's directory</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> newdir=$dest/$user/lost+found/$date</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> echo " newdir=$newdir"</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> echo " checking/making direcotry"</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> if [ ! -d $newdir ] ; then</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> time mkdir -p $newdir</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> time chown ${user}: $newdir</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> fi</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> echo " moving file"</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> time mv $file $newdir</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">done</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I’m pretty sure the time to operate on these files is due to the very large number of files in a single directory. But confirmation of that would be good. I say that because I know too many files in a single directory can cause filesystem performance issues.
But also, the few files I have moved out of lost+found, I can stat and otherwise operate on very quickly. <o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
My next attempt was to move each file into pre-created subdirectories (this eliminates the stat). But again, this was serial and each move is (unsurprisingly) taking 30 seconds. “Only” 12.5 years to move the files. :)<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I’m currently attempting to speed this up by getting the entire file list (all 13e6 files) and moving groups of (10,000) files into a subdirectory at once (i.e. mv file1 file2 … fileN subdir1). I’m hoping a group move is faster and more efficient than moving
each file individually. Seems like it should be. That attempt is running now and I’m waiting for the first command to complete so I don’t know if this is faster yet or not. (More accurately, I’ve written this script in perl and I think the output is buffered
so I’m waiting to the the first output.) <span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Other suggestions welcome if you have ideas how to move these files into subdirectories more efficiently. <o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss-bounces@lists.lustre.org</a>>
on behalf of "Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss" <<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Reply-To:<span class="Apple-converted-space"> </span></b>"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <<a href="mailto:darby.vicker-1@nasa.gov" style="color: blue; text-decoration: underline;" class="">darby.vicker-1@nasa.gov</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Monday, September 25, 2023 at 8:56 AM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>Andreas Dilger <<a href="mailto:adilger@whamcloud.com" style="color: blue; text-decoration: underline;" class="">adilger@whamcloud.com</a>><br class="">
<b class="">Cc:<span class="Apple-converted-space"> </span></b>"<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [lustre-discuss] [BULK] Re: [EXTERNAL] Re: Data recovery with lost MDT data</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Our lfsck finished. It repair a lot and we have over 13 million files in lost+found to go through. I'll be writing a script to move these to somewhere accessible by the users and grouped by owner and probably date too (trying not to get too many files in
a single directory). Thanks again for the help with this. <span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-family: Calibri, sans-serif;" class="">For the benefit of others, this is how we started our lfsck:<br class=""><br class=""></span>[root@hpfs-fsl-mds1 hpfs3-eg3]# lctl set_param printk=+lfsck<o:p class=""></o:p></pre>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: "Courier New";" class="">[root@hpfs-fsl-mds1 hpfs3-eg3]# lctl lfsck_start -M scratch-MDT0000 -o</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: "Courier New";" class="">Started LFSCK on the device scratch-MDT0000: scrub layout namespace</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 10pt; font-family: "Courier New";" class="">[root@hpfs-fsl-mds1 hpfs3-eg3]#</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
It took most of the weekend to run. Here are the results. <span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">[root@hpfs-fsl-mds1 ~]# lctl lfsck_query -M scratch-MDT0000 </span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_init: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_scanning-phase1: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_scanning-phase2: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_completed: 1</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_failed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_stopped: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_paused: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_crashed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_partial: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_co-failed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_co-stopped: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_co-paused: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_mdts_unknown: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_init: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_scanning-phase1: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_scanning-phase2: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_completed: 22</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_failed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_stopped: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_paused: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_crashed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_partial: 2</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_co-failed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_co-stopped: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_co-paused: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_osts_unknown: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">layout_repaired: 38587653</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_init: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_scanning-phase1: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_scanning-phase2: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_completed: 1</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_failed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_stopped: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_paused: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_crashed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_partial: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_co-failed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_co-stopped: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_co-paused: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_mdts_unknown: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_init: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_scanning-phase1: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_scanning-phase2: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_completed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_failed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_stopped: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_paused: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_crashed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_partial: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_co-failed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_co-stopped: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_co-paused: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_osts_unknown: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">namespace_repaired: 1429495</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">[root@hpfs-fsl-mds1 ~]# lctl get_param -n mdd.scratch-MDT0000.lfsck_layout</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">name: lfsck_layout</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">magic: 0xb1732fed</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">version: 2</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">status: completed</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">flags:</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">param: all_targets,orphan</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">last_completed_time: 1695615657</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">time_since_last_completed: 35014 seconds</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">latest_start_time: 1695335260</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">time_since_latest_start: 315411 seconds</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">last_checkpoint_time: 1695615657</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">time_since_last_checkpoint: 35014 seconds</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">latest_start_position: 15</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">last_checkpoint_position: 1015668480</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">first_failure_position: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">success_count: 2</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">repaired_dangling: 22199282</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">repaired_unmatched_pair: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">repaired_multiple_referenced: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">repaired_orphan: 13489715</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">repaired_inconsistent_owner: 2898656</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">repaired_others: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">skipped: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">failed_phase1: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">failed_phase2: 1798679</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">checked_phase1: 369403698</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">checked_phase2: 15</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">run_time_phase1: 82842 seconds</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">run_time_phase2: 0 seconds</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">average_speed_phase1: 4459 items/sec</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">average_speed_phase2: 15 objs/sec</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">real_time_speed_phase1: N/A</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">real_time_speed_phase2: N/A</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">current_position: N/A</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">[root@hpfs-fsl-mds1 ~]# lctl get_param -n mdd.scratch-MDT0000.lfsck_namespace</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">name: lfsck_namespace</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">magic: 0xa06249ff</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">version: 2</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">status: completed</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">flags:</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">param: all_targets,orphan</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">last_completed_time: 1695419689</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">time_since_last_completed: 231012 seconds</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">latest_start_time: 1695335262</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">time_since_latest_start: 315439 seconds</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">last_checkpoint_time: 1695419689</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">time_since_last_checkpoint: 231012 seconds</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">latest_start_position: 15, N/A, N/A</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">last_checkpoint_position: 1015668480, N/A, N/A</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">first_failure_position: N/A, N/A, N/A</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">checked_phase1: 315743757</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">checked_phase2: 231782</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">updated_phase1: 1429495</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">updated_phase2: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">failed_phase1: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">failed_phase2: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">directories: 15911850</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">dirent_repaired: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">linkea_repaired: 1429495</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">nlinks_repaired: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">multiple_linked_checked: 1983873</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">multiple_linked_repaired: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">unknown_inconsistency: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">unmatched_pairs_repaired: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">dangling_repaired: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">multiple_referenced_repaired: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">bad_file_type_repaired: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">lost_dirent_repaired: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">local_lost_found_scanned: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">local_lost_found_moved: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">local_lost_found_skipped: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">local_lost_found_failed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">striped_dirs_scanned: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">striped_dirs_repaired: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">striped_dirs_failed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">striped_dirs_disabled: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">striped_dirs_skipped: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">striped_shards_scanned: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">striped_shards_repaired: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">striped_shards_failed: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">striped_shards_skipped: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">name_hash_repaired: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">linkea_overflow_cleared: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">agent_entries_repaired: 0</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">success_count: 2</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">run_time_phase1: 82885 seconds</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">run_time_phase2: 1542 seconds</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">average_speed_phase1: 3809 items/sec</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">average_speed_phase2: 150 objs/sec</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">average_speed_total: 3742 items/sec</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">real_time_speed_phase1: N/A</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">real_time_speed_phase2: N/A</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">current_position: N/A</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">[root@hpfs-fsl-mds1 ~]# </span><o:p class=""></o:p></pre>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
And on a client, the resulting lost+found directory:<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">[root@hpfs-fsl-lmon0 MDT0000]# pwd</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">/scratch-lustre/.lustre/lost+found/MDT0000</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">[root@hpfs-fsl-lmon0 MDT0000]# time \ls | wc -l</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">13063930</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""> <o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">real 15m20.604s</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">user 2m38.186s</span><o:p class=""></o:p></pre>
<pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span class="line">sys 0m4.116s</span><o:p class=""></o:p></pre>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[root@hpfs-fsl-lmon0 MDT0000]# \ls | head</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[0x20000bdbe:0x1eae1:0x0]-R-0</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[0x20000bdeb:0x12c3e:0x0]-R-0</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[0x20001f801:0x296f:0x0]-R-0</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[0x20001f801:0x57:0x0]-R-0</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[0x20001f801:0x58:0x0]-R-0</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[0x20001f805:0x1000:0x0]-R-0</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[0x20001f805:0x100:0x0]-R-0</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[0x20001f805:0x1001:0x0]-R-0</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[0x20001f805:0x1002:0x0]-R-0</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[0x20001f805:0x1003:0x0]-R-0</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[root@hpfs-fsl-lmon0 MDT0000]# ls -l [0x20000bdbe:0x1eae1:0x0]-R-0</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">-r-------- 1 damocles_runner damocles 3162 Sep 24 14:45 [0x20000bdbe:0x1eae1:0x0]-R-0 </span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[root@hpfs-fsl-lmon0 MDT0000]#</span><o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss-bounces@lists.lustre.org</a>>
on behalf of "Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss" <<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Reply-To:<span class="Apple-converted-space"> </span></b>"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <<a href="mailto:darby.vicker-1@nasa.gov" style="color: blue; text-decoration: underline;" class="">darby.vicker-1@nasa.gov</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Friday, September 22, 2023 at 2:49 PM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>Andreas Dilger <<a href="mailto:adilger@whamcloud.com" style="color: blue; text-decoration: underline;" class="">adilger@whamcloud.com</a>><br class="">
<b class="">Cc:<span class="Apple-converted-space"> </span></b>"<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>[BULK] Re: [lustre-discuss] [EXTERNAL] Re: Data recovery with lost MDT data</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I’m only showing you the last 10 directories below but there are about 30 or 40 directories with a pretty uniform distribution between 6/20 and now. If it was a situation where we had been rolled back to 6/20 but directories were starting to be updated again,
there should be a big gap with no updates. The rollback (when we deleted the “snapshot”) happened on Monday, 9/18. We could do another snapshot of the MDT, mount it read only and poke around in there if you think that would help. Actually, our backup process
(which is running normally again) is doing just that. It takes quite a long time to complete so there is opportunity for me to investigate. <o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Andreas Dilger <<a href="mailto:adilger@whamcloud.com" style="color: blue; text-decoration: underline;" class="">adilger@whamcloud.com</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Friday, September 22, 2023 at 1:36 AM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <<a href="mailto:darby.vicker-1@nasa.gov" style="color: blue; text-decoration: underline;" class="">darby.vicker-1@nasa.gov</a>><br class="">
<b class="">Cc:<span class="Apple-converted-space"> </span></b>"<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [EXTERNAL] Re: [lustre-discuss] Data recovery with lost MDT data</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" align="left" style="border: 1.5pt solid black;">
<tbody class="">
<tr class="">
<td width="100%" style="width: 1167.3125px; border: none; background-color: rgb(255, 235, 156); padding: 3.75pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 10pt;" class="">CAUTION:</span></b><span style="" class=""><span class="Apple-converted-space"> </span></span><span style="font-size: 10pt;" class="">This email originated from outside of NASA. Please take care when clicking
links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.</span><span style="" class=""></span><o:p class=""></o:p></div>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<o:p class=""></o:p></p>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On Sep 21, 2023, at 16:06, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] <<a href="mailto:darby.vicker-1@nasa.gov" style="color: blue; text-decoration: underline;" class="">darby.vicker-1@nasa.gov</a>> wrote:<o:p class=""></o:p></div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I knew an lfsck would identify the orphaned objects. That’s great that it will move those objects to an area we can triage. With ownership still intact (and I assume time stamps too), I think this will be helpful for at least some of the users to recover
some of their data. Thanks Andreas. <o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I do have another question. Even with the MDT loss, the top level user directories on the file system are still showing current modification times. I was a little surprised to see this – my expectation was that the most current time would be from the snapshot
that we accidentally reverted to, 6/20/2023 in this case. Does this make sense? <o:p class=""></o:p></div>
</div>
</div>
</blockquote>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
The timestamps of the directories are only stored on the MDT (unlike regular files which keep of the timestamp on both the MDT and OST). Is it possible that users (or possibly recovered clients with existing mountpoints) have started to access the filesystem
in the past few days since it was recovered, or an admin was doing something that would have caused the directories to be modified?<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Is it possible you have a newer copy of the MDT than you thought?<o:p class=""></o:p></div>
</div>
<div class="">
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-family: "Courier New";" class=""> </span><o:p class=""></o:p></p>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[dvicker@dvicker ~]$ ls -lrt /ephemeral/ | tail</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> 4 drwx------ 2 abjuarez abjuarez 4096 Sep 12 13:24 abjuarez/</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> 4 drwxr-x--- 2 ksmith29 ksmith29 4096 Sep 13 15:37 ksmith29/</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> 4 drwxr-xr-x 55 bjjohn10 bjjohn10 4096 Sep 13 16:36 bjjohn10/</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> 4 drwxrwx--- 3 cbrownsc ccp_fast 4096 Sep 14 12:27 cbrownsc/</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> 4 drwx------ 3 fgholiza fgholiza 4096 Sep 18 06:41 fgholiza/</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> 4 drwx------ 5 mtfoste2 mtfoste2 4096 Sep 19 11:35 mtfoste2/</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> 4 drwx------ 4 abenini abenini 4096 Sep 19 15:33 abenini/</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class=""> 4 drwx------ 9 pdetremp pdetremp 4096 Sep 19 16:49 pdetremp/</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-family: "Courier New";" class="">[dvicker@dvicker ~]$</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 12pt;" class="">From:<span class="apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Andreas Dilger <<a href="mailto:adilger@whamcloud.com" style="color: blue; text-decoration: underline;" class=""><span style="color: rgb(5, 99, 193);" class="">adilger@whamcloud.com</span></a>><br class="">
<b class="">Date:<span class="apple-converted-space"> </span></b>Thursday, September 21, 2023 at 2:33 PM<br class="">
<b class="">To:<span class="apple-converted-space"> </span></b>"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <<a href="mailto:darby.vicker-1@nasa.gov" style="color: blue; text-decoration: underline;" class=""><span style="color: rgb(5, 99, 193);" class="">darby.vicker-1@nasa.gov</span></a>><br class="">
<b class="">Cc:<span class="apple-converted-space"> </span></b>"<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class=""><span style="color: rgb(5, 99, 193);" class="">lustre-discuss@lists.lustre.org</span></a>"
<<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class=""><span style="color: rgb(5, 99, 193);" class="">lustre-discuss@lists.lustre.org</span></a>><br class="">
<b class="">Subject:<span class="apple-converted-space"> </span></b>[EXTERNAL] Re: [lustre-discuss] Data recovery with lost MDT data</span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" align="left" style="border: 1.5pt solid black;">
<tbody class="">
<tr class="">
<td width="100%" style="width: 1141px; border: none; background-color: rgb(255, 235, 156); padding: 3.75pt; background-position: initial initial; background-repeat: initial initial;" class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 10pt;" class="">CAUTION:</span></b><span class="apple-converted-space"><span style="font-size: 12pt;" class=""> </span></span><span style="font-size: 10pt;" class="">This email originated from outside of NASA. Please take
care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.</span><o:p class=""></o:p></div>
</div>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></p>
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class="">In the absence of backups, you could try LFSCK to link all of the orphan OST objects into .lustre/lost+found (see lctl-lfsck_start.8 man page for details).</span><o:p class=""></o:p></div>
</div>
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class="">The data is still in the objects, and they should have UID/GID/PRJID assigned (if used) but they have no filenames. It would be up to you to make e.g. per-user lost+found directories in their home directories and move
the files where they could access them and see if they want to keep or delete the files. <span class="apple-converted-space"> </span></span><o:p class=""></o:p></div>
</div>
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class="">How easy/hard this is to do depends on whether the files have any content that can help identify them. </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt;" class="">There was a Lustre hackathon project to save the Lustre JobID in a "user.job" xattr on every object, exactly to help identify the provenance of files after the fact (regardless of whether there is corruption), but it
only just landed to master and will be in 2.16. That is cold comfort, but would help in the future. </span><o:p class=""></o:p></p>
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 12pt;" class="">Cheers, Andreas</span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></p>
</div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span style="font-size: 12pt;" class="">On Sep 20, 2023, at 15:34, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class=""><span style="color: rgb(5, 99, 193);" class="">lustre-discuss@lists.lustre.org</span></a>>
wrote:</span><o:p class=""></o:p></p>
</blockquote>
</div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Hello,<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
We have recently accidentally deleted some of our MDT data. I think its gone for good but looking for advice to see if there is any way to recover. Thoughts appreciated. <o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
We run two LFS’s on the same set of hardware. We didn’t set out to do this, but it kind of evolved. The original setup was only a single filesystem and was all ZFS – MDT and OST’s. Eventually, we had some small file workflows that we wanted to get better
performance on. To address this, we stood up another filesystem on the same hardware and used a an ldiskfs MDT. However, since were already using ZFS, under the hood the storage device we build the ldisk MDT on comes from ZFS. That gets presented to the
OS as /dev/zd0. We do a nightly backup of the MDT by cloning the ZFS dataset (this creates /dev/zd16, for whatever reason), snapshot the clone, mount that as ldiskfs, tar up the data and then destroy the snapshot and clone. Well, occasionally this process
gets interrupted, leaving the ZFS snapshot and clone hanging around. This is where things go south. Something happens that swaps the clone with the primary dataset. ZFS says you’re working with the primary but its really the clone, and via versa. This
happened about a year ago and we caught it, were able to “zfs promote” to swap them back and move on. More details on the ZFS and this mailing list here. <o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<a href="https://zfsonlinux.topicbox.com/groups/zfs-discuss/Tcb8a3ef663db0031-M5a79e71768b20b2389efc4a4" style="color: blue; text-decoration: underline;" class=""><span style="color: rgb(5, 99, 193);" class="">https://zfsonlinux.topicbox.com/groups/zfs-discuss/Tcb8a3ef663db0031-M5a79e71768b20b2389efc4a4</span></a><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2022-June/018154.html" style="color: blue; text-decoration: underline;" class=""><span style="color: rgb(5, 99, 193);" class="">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2022-June/018154.html</span></a><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
It happened again earlier this week but we didn’t remember to check this and, in an effort to get the backups going again, destroyed what we thought were the snapshot and clone. In reality, we destroyed the primary dataset. Even more unfortunately, the stale
“snapshot” was about 3 months old. This stale snapshot was also preventing our MDT backups from running so we don’t have those to restore from either. (I know, we need better monitoring and alerting on this, we learned that lesson the hard way. We had it
in place after the June 2022 incident, it just wasn’t working properly.) So at the end of the day, the data lives on the OST’s we just can’t access it due to the lost metadata. Is there any chance at data recovery. I don’t think so but want to explore any
options. <span class="apple-converted-space"> </span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Darby<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline;" class=""><span style="color: rgb(5, 99, 193);" class="">lustre-discuss@lists.lustre.org</span></a><br class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="color: blue; text-decoration: underline;" class=""><span style="color: rgb(5, 99, 193);" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</span></a><o:p class=""></o:p></div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Cheers, Andreas</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">--</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Andreas Dilger</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Lustre Principal Architect</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Whamcloud</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>