[lustre-discuss] Disaster recover files from ZFS OSTs

[Andreas Dilger]

On Feb 16, 2019, at 16:05, Hans Henrik Happe <happe at nbi.dk> wrote:
> 
> Moving a system away from Gluster to Lustre there is one feature we
> miss. With Gluster the name space and data can easily be found on the
> underlying filesystems. While we never needed it with Lustre, it would
> be nice to have it as a last resort. Lustre has been rock solid, while
> we have needed it plenty on Gluster.
> 
> Looking at the output of 'getstripe' I figured out how to find files
> using the objids mod 128 (that's was how many dX dirs i found). Easy
> with stripe-count=1, probably also with higher counts.
> 
> So given a backup of MDTs we should be able to do some poking around. We
> could also do a database of getstripe info. Perhaps robinhood could help.
> 
> Is there some formal description of Lustre object layout? It seems
> simple but I'm wondering if there are pitfalls. PFL seems to be pretty
> well described.
> 
> Perhaps there are already tools for this, that we have missed?

As you have already seen, the MDT is essentially a directory tree of all
the files in the filesystem, and the OST objects listed in the LOV EA
contain the file data.  If there are multiple stripes, the data is round-
robin across each object in stripe_size chunks.  For PFL, each component
is a regular file layout.

That said, I wouldn't suggest that you do filesystem recovery by rebuilding
files from the raw objects in this manner.  It would be a lot easier to make
a backup of the MDT in case of problems and use that to recover if the MDT
ever failed completely.

> Side note:
> 
> Looking at how Lustre put object files in large buckets, I was wondering
> if this ZFS issue could become an issue:
> 
> Guess these buckets are rarely listed?

All of the object lookups in these directories are done by name (object ID),
so they are not listed per se.  Even so, this issue as described doesn't
really apply to Lustre since it always uses FatZAPs and not MicroZAPs.  The
other issue described in the ticket is that performance declines when a
directory gets very full and is then emptied.  This isn't really a problem
as the object directories are continually used.

Cheers, Andreas
---
Andreas Dilger
Principal Lustre Architect
Whamcloud