[lustre-discuss] Analog of ll_recover_lost_found_objs for MDS

[Chris Hunter]

The "lfsck" feature added in lustre 2.7 will attempt recovery of MDT
orphan file objects from lost+found. Although likely doesn't help you
ATM.

You may wish to look at script attached to LU-5626
ll_fix_mdt_lost_found.sh. I was intended to scan e2fsck logs for
recovery of "FID in dirent" errors. Another tool ll_decode_linkea is
mentioned in LU-8289 to help recover MDT lost+found objects. I have not
tried either tool.

Since critical information is stored as extended attributes in MDT
files, even a partial backup (file-level) can be useful for recovery.
Lustre operations manual discusses using LVM snapshots for MDT backup.
https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#backupandrestore
(assuming you created your MDT as lvm in the beginning...)

good luck,
chris hunter
chuntera at gmail.com


> Hi James,
> 
> I'm not aware of a ready-to use tool, but if you have captured the
> output of e2fsck you can use that as a basis for a script that puts
> the
> files back to their original location.
> e2fsck usually prints out the full path and the inode numbers of the
> files/directories which it moves to lost+found and there, they are
> named
> "#$inode" (which makes scripting a bit ugly, but if you properly
> escape
> the '#'-sign and do some magic with awk, perl or alike, you can
> transform the log to a shell script that moves your files back to the
> orignal path). I have done this once after a file system corruption
> after an upgrade from 1.8 to 2.4 (which contained an ugly bug when
> enabling the "FID in dirent" feature).
> 
> The backup... well, it gives you the chance to go back to the state
> where you started *before* the e2fsck. That would be a chance to
> capture
> the output again, in case you did not store it (actually, you could do
> this offline, on a copy of the backup). Restoring the MDT out of the
> backup however is only useful as long as you did not go in production
> after the e2fsck. And as I said, you still have to "repair" the
> restored
> MDT (probably by doing the same steps as you already did on the live
> system), but a second chance is better than no chance to go back...
> The
> backup is also good to investigate what happened during the e2fsck (in
> case it did something weird) or to go in with e2fsdebug for manual
> investigations... (e.g. manually look up inode<->path relations).
> 
> Martin
> 
> 
> On 07/26/2016 04:08 PM, jbellinger wrote:
> > Is there, or could there be, something analogous to the OST recovery
> > tool that works on the lost+found on the MDT?  e2fsck went berserk.
> >
> > We're running 2.5.3.
> >
> >
> > Thanks,
> > James Bellinger
> >
> > Yes, we have an older (therefore somewhat inconsistent) backup of
> the
> > mdt, so we should be able to recover most things, _in theory_.  In
> > practice -- we'd love to hear other people's experience about
> recovery
> > using an inconsistent backup.
> > _______________________________________________