[lustre-discuss] undelete

[Dilger, Andreas]

On Jan 22, 2018, at 19:03, E.S. Rosenberg <esr+lustre at mail.hebrew.edu> wrote:
> 
> Dragging the old discussion back up....
> First of thanks for all the replies last time!
> 
> Last time in the end we didn't need to recover but now another user made a bigger mistake and we do need to recover data.

Sounds like it is time for backups and/or snapshots to avoid these issues in the future.  If you don't have space for a full filesystem backup, doing daily backups of the MDT simplifies such data recovery significantly.  Preferably the backup is done from a snapshot using "dd" or "e2image", but even without a snapshot it is better to do a backup from the raw device than not at all.

> I have shut down our Lustre filesystem and am going to do some simulations on a test system trying various undelete tools.
> 
> autopsy (sleuthkit) on the metadata shows that at least the structure is still there and hopefully we'll be able to recover more.

You will need to be able to recover the file layout from the deleted MDT inode (which existing ext4 recovery tools might help with), including the "lov" xattr, which is typically stored inside the inode itself unless the file was widely striped.

Secondly, you will also need to recover the matching OST inodes/objects that were deleted.  There may be deleted entries in the OST object directories (O/0/d*/) that tell you which inodes the objects were using.  Failing that, you may be able to tell from the "fid" xattr of deleted inodes which object they were.  Using the Lustre debugfs "stat <inode>" command may help on the OST.

You would need to undelete all of the objects in a multi-stripe file for that to be very useful.

> Has anyone ever done true recovery of Lustre or is it all just theoretical knowledge at the moment?
> 
> What are the consequences of say undeleting data on OSTs that is then not referenced on the MDS? Could I cause corruption of the whole filesystem by doing stuff like that?

As long as you are not corrupting the actual OST or MDT filesystems by undeleting an inode whose blocks were reallocated to another file, it won't harm things.  At worst it would mean OST objects that are not reachable through the MDT namespace.  Running an lfsck namespace scan (2.7+) would link such OST objects into the $MOUNT/.lustre/lost+found directory if they are not referenced from any MDT inode.

> (As far as the files themselves go they are most likely all single striped since that is our default and we are pre PFL so that should be easier I think).

That definitely simplifies things significantly.

Cheers, Andreas

> On Thu, May 4, 2017 at 2:21 AM, Dilger, Andreas <andreas.dilger at intel.com> wrote:
> On Apr 27, 2017, at 05:43, E.S. Rosenberg <esr+lustre at mail.hebrew.edu> wrote:
> >
> > A user just rm'd a big archive of theirs on lustre, any way to recover it before it gets destroyed by other writes?
> 
> Just noticed this email.
> 
> In some cases, an immediate power-off followed by some ext4 recovery tools (e.g. ext3grep) might get you some data back, but that is very uncertain.
> 
> With ZFS MDT/OST filesystems (or to a lesser extent LVM) it is possible to create periodic snapshots of the filesystems for recovery purposes.  ZFS handles this fairly well performance wise, LVM much less so.  With Lustre 2.10 there are new tools to manage the ZFS snapshots and allow mounting them as a separate Lustre filesystem.

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation