<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
First thing, if you haven't already done so, would be to make a separate "dd" backup of the ldiskfs MDT(s) to some external storage before you do anything else. That will give you a fallback in case whatever changes you make don't work out well.
<div class=""><br class="">
</div>
<div class="">I would also suggest to contact the ZFS mailing list to ask if they can help restore the "new version" of the MDT at the ZFS level. You may also want to consider a separate ZFS-level backup because the core of the problem appears to be ZFS related.
Unfortunately, the opportunity to recover a newer version of the ldiskfs MDT at the ZFS level declines the more changes are made to the ZFS pool.
<div class=""><br class="">
</div>
<div class="">I don't think LFSCK will repair the missing files on the MDT, since the OSTs don't have enough information to regenerate the namespace. At most LFSCK will create stub files on the MDT under .lustre/lost+found that connect the objects for the
new files created after your MDT snapshot, but they won't have proper filenames. At most they will have UID/GID/timestamps to identify the owners/age, and the users would need to identify the files by content.</div>
<div class=""><br class="">
</div>
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 22, 2022, at 10:46, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">A quick follow up. I thought an lfsck would only clean up (i.e. remove orphaned MDT and OST objects) but it appears this might have a good shot at repairing the file system – specifically, recreating the MDT objects
with the --create-mdtobj option. We have started this command:<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">[root@hpfs-fsl-mds1 ~]# lctl lfsck_start -M scratch-MDT0000 --dryrun on --create-mdtobj on<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">And after running for about an hour we are already seeing this from the query:<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">layout_repaired: 4645105<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Can anyone confirm this will work for our situation – i.e. repair the metadata for the OST objects that were orphaned when our metadata got reverted? <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="" class="">"Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]" <<a href="mailto:darby.vicker-1@nasa.gov" class="">darby.vicker-1@nasa.gov</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Tuesday, June 21, 2022 at 5:27 PM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>"<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>Help with recovery of data<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
</div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Hi everyone,<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">We ran into a problem with our lustre filesystem this weekend and could use a sanity check and/or advice on recovery. <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">We are running on CentOS 7.9, ZFS 2.1.4 and Lustre 2.14. We are using ZFS OST’s but and an ldiskfs MDT (for better MDT performance). For various reasons, the ldiskfs is built on a zdev. Every night we (intend to) back
up the metadata by ZFS snapshot-ing the zdev, mount the MDT via ldiskfs and tar up the contents, umount and remove the ZFS snapshot. On Sunday (6/19 at about 4 pm), the metadata server crashed. It came back up fine but users started reporting many missing
files and directories today (6/21) – everything since about February 9th is gone. After quite a bit of investigation, it looks like the MDT got rolled back to a snapshot of the metadata from February.<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@hpfs-fsl-mds1 ~]# zfs list -t snap mds1-0/meta-scratch<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">NAME USED AVAIL REFER MOUNTPOINT<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">mds1-0/meta-scratch@snap 52.3G - 1.34T -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@hpfs-fsl-mds1 ~]# zfs get all mds1-0/meta-scratch@snap | grep creation<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">mds1-0/meta-scratch@snap creation Thu Feb 10 3:35 2022 -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@hpfs-fsl-mds1 ~]#<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">We discovered that our MDT backups have been stalled since February since the first step is to create mds1-0/meta-scratch@snap and that dataset already exists. The script was erroring out since the existing snapshot
still in place. We have rebooted this MDS several times (gracefully) since February with no issues but, apparently, whatever happened in the server crash on Sunday caused the MDT to revert to the February data. So, in theory, the data on the OST’s is still
there, we are just missing the metadata due to the ZFS glitch. <span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">So the first question - is anyone familiar with this failure mode of ZFS or if there is a way recover from it? I think its unlikely there are any direct ZFS recovery options but wanted to ask. <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Obviously, MDT backups would be our best recovery option but since this was all caused by the backup scripts stalling (and the subsequent rolling back to the last snapshot), our backups are the same age as the current
data on the filesystem. <span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@hpfs-fsl-mds1 ~]# ls -lrt /internal/ldiskfs_backups/<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">total 629789909<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 1657 Apr 30 2019 process.txt<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 445317560320 Jan 25 15:36 mds1-0_meta-scratch-2022_01_25.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 446230016000 Jan 26 15:31 mds1-0_meta-scratch-2022_01_26.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 448093808640 Jan 27 15:46 mds1-0_meta-scratch-2022_01_27.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 440368783360 Jan 28 16:56 mds1-0_meta-scratch-2022_01_28.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 442342113280 Jan 29 14:45 mds1-0_meta-scratch-2022_01_29.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 442922567680 Jan 30 15:03 mds1-0_meta-scratch-2022_01_30.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 443076515840 Jan 31 15:17 mds1-0_meta-scratch-2022_01_31.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 444589025280 Feb 1 15:11 mds1-0_meta-scratch-2022_02_01.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 443741409280 Feb 2 15:17 mds1-0_meta-scratch-2022_02_02.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 448209367040 Feb 3 15:24 mds1-0_meta-scratch-2022_02_03.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 453777090560 Feb 4 15:55 mds1-0_meta-scratch-2022_02_04.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 454211307520 Feb 5 14:37 mds1-0_meta-scratch-2022_02_05.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 454619084800 Feb 6 14:30 mds1-0_meta-scratch-2022_02_06.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 455459276800 Feb 7 15:26 mds1-0_meta-scratch-2022_02_07.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 457470945280 Feb 8 15:07 mds1-0_meta-scratch-2022_02_08.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 460592517120 Feb 9 15:21 mds1-0_meta-scratch-2022_02_09.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">-rw-r--r-- 1 root root 332377712640 Feb 10 12:04 mds1-0_meta-scratch-2022_02_10.tar<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@hpfs-fsl-mds1 ~]#<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Yes, I know, we will put in some monitoring for this in the future...<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Fortunately, we also have a robinhood system syncing with this file system. The sync is fairly up to date – the logs say a few days ago and I’ve used rbh-find to find some files that were created in the last few days.
So I think we have a shot at recovery. We have this command running now to see what it will do:<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">rbh-diff --apply=fs --dry-run --scan=/scratch-lustre<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">But it has already been running a long time with no output. Our file system is fairly large:<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@hpfs-fsl-lmon0 ~]# lfs df -h /scratch-lustre<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">UUID bytes Used Available Use% Mounted on<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-MDT0000_UUID 1011.8G 82.6G 826.7G 10% /scratch-lustre[MDT:0]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0000_UUID 49.6T 16.2T 33.4T 33% /scratch-lustre[OST:0]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0001_UUID 49.6T 17.4T 32.3T 35% /scratch-lustre[OST:1]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0002_UUID 49.6T 16.8T 32.8T 34% /scratch-lustre[OST:2]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0003_UUID 49.6T 17.2T 32.4T 35% /scratch-lustre[OST:3]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0004_UUID 49.6T 16.7T 32.9T 34% /scratch-lustre[OST:4]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0005_UUID 49.6T 16.9T 32.7T 35% /scratch-lustre[OST:5]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0006_UUID 49.6T 16.4T 33.2T 34% /scratch-lustre[OST:6]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0007_UUID 49.6T 15.6T 34.0T 32% /scratch-lustre[OST:7]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0008_UUID 49.6T 16.2T 33.4T 33% /scratch-lustre[OST:8]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0009_UUID 49.6T 16.4T 33.2T 34% /scratch-lustre[OST:9]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST000a_UUID 49.6T 15.8T 33.8T 32% /scratch-lustre[OST:10]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST000b_UUID 49.6T 17.4T 32.2T 36% /scratch-lustre[OST:11]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST000c_UUID 49.6T 17.1T 32.5T 35% /scratch-lustre[OST:12]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST000d_UUID 49.6T 15.8T 33.8T 32% /scratch-lustre[OST:13]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST000e_UUID 49.6T 15.7T 33.9T 32% /scratch-lustre[OST:14]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST000f_UUID 49.6T 16.4T 33.2T 33% /scratch-lustre[OST:15]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0010_UUID 49.6T 15.5T 34.1T 32% /scratch-lustre[OST:16]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0011_UUID 49.6T 16.6T 33.1T 34% /scratch-lustre[OST:17]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0012_UUID 49.6T 16.4T 33.2T 34% /scratch-lustre[OST:18]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0013_UUID 48.4T 16.3T 32.1T 34% /scratch-lustre[OST:19]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0014_UUID 49.6T 15.1T 34.5T 31% /scratch-lustre[OST:20]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0015_UUID 49.6T 16.0T 33.6T 33% /scratch-lustre[OST:21]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0016_UUID 49.6T 15.2T 34.4T 31% /scratch-lustre[OST:22]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">scratch-OST0017_UUID 49.6T 16.1T 33.5T 33% /scratch-lustre[OST:23]<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">filesystem_summary: 1.2P 391.1T 798.3T 33% /scratch-lustre<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@hpfs-fsl-lmon0 ~]#<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">We currently still have the robinhood process running (syncing the filesystem and the SQL DB) but we’ve umounted the LFS from all user facing machines so there should be no further changes to the
filesystem. <span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Does anyone have experience recovering from this kind of situation with robinhood? <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">FWIW, the SQL DB that robinhood lives on is also a ZFS filesystem that we also snapshot. But we don’t have much history and its unclear where the current RBH scans are (WRT the data loss). But
its likely SQL DB in the oldest snapshot below would not be affected by the 6/19 reboot event. <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@hpfs-fsl-lmon0 ~]# zfs list -t snap<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">NAME USED AVAIL REFER MOUNTPOINT<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-20-16:01:01 26.8G - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-20-17:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-20-18:01:01 557K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-20-19:01:01 558K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-20-20:01:01 558K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-20-21:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-20-22:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-20-23:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-00:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-01:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-02:01:02 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-03:01:01 561K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-04:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-05:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-06:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-07:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-08:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-09:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-10:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-11:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-12:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-13:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-14:01:01 560K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-15:01:01 518K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">lmon0-0/mysql@zincrsend_2022-06-21-16:01:01 512K - 105G -<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">[root@hpfs-fsl-lmon0 ~]#<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Is RBH our best recovery option?<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Would lfsck recover from this situation? I don’t think so...<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Any advice on recovery would be appreciated. <o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Thanks,<o:p class=""></o:p></span></div>
<div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; font-family: Courier;" class="">Darby<o:p class=""></o:p></span></div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">lustre-discuss
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span></div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</div>
</body>
</html>