[Lustre-discuss] MDT backup procedure

[Daniel Kulinski]

Thanks for this verbose reply.  It is exactly what I needed and what I suspected I would run into.  We are planning on multiple backup procedures.  Users will backup at checkpoints in their work flow, IT will backup the MDT nightly and we are also looking at the possibility of backup the complete file system.  

Thanks again for everyone's input, this gives me some good ammunition going forward for proposals.

Thanks,
  Dan Kulinski

-----Original Message-----
From: Andreas.Dilger at sun.com [mailto:Andreas.Dilger at sun.com] On Behalf Of Andreas Dilger
Sent: Wednesday, June 17, 2009 4:23 PM
To: Cliff White
Cc: Ramiro Alba Queipo; lustre-discuss at lists.lustre.org; Daniel Kulinski
Subject: Re: [Lustre-discuss] MDT backup procedure

On Jun 17, 2009  12:35 -0700, Cliff White wrote:
> Ramiro Alba Queipo wrote:
> > By reading Chapter 15 of Lustre Operations Manual, it follows that an
> > MDT backup is only useful if you are changing hardwary or the like.
> > I am afraid that you can not pretend to replace with a previous image an
> > failed MDT, as data in OSTs and MDT is not matching any more, right?
> 
> If you do a backup/immediate restore, it should be fine. If you restore 
> from an old image you will lose the changes made post-backup, but the 
> rest of the data should be fine.
> cliffw

Right - just like any backup, any changes made after the backup will of
course not be restored.  One additional issue is that some OST objects
will not be available if they were deleted after the backup, even though
the restored MDS will still reference them.  Accessing these files will
return -ENOENT.

At that point it would be possible (though not necessary) to run "lfsck"
to clean up the inconsistencies between the MDT and OST filesystems.
It is also possible to just re-delete the files that have "-ENOENT" and
restore (from some other filesystem-level backup) the rest of the files.

An MDS backup is a good idea, because it avoids having to restore 100TB+
(or whatever) of data from backup, leaving only a smaller number of changed
files that might need to be restored.  It should NOT be the only form of
backup for the filesystem, since it does not contain any of the FILE data.
You, or your users, should do backups of their critical files separately.

> > On Wed, 2009-06-17 at 09:41 -0600, Daniel Kulinski wrote:
> >> As we move forward with our lustre testing I am wondering about MDT
> >> backup.  
> >>
> >>  
> >>
> >> Is it feasible to unmount the MDT, create an image of it and remount
> >> it after the backup.  Of course this wouldn’t happen but nightly.
> >>
> >>  
> >>
> >> From what I can identify, in the case of an MDT failure we would have
> >> to do the following:
> >>
> >>  
> >>
> >> Restore from the last backup.
> >>
> >> Run an lfsck across the filesystem.
> >>
> >>  
> >>
> >> Am I missing anything else at this point?  We will also be doing file
> >> level backups of the filesystem as a whole but we are looking for
> >> quick ways to recover from an MDT failure.
> >>
> >>  
> >>
> >> Thanks,
> >>
> >>   Dan Kulinski
> >>
> >>
> >>
> >> -- 
> >> Aquest missatge ha estat analitzat per MailScanner 
> >> a la cerca de virus i d'altres continguts perillosos, 
> >> i es considera que está net. 
> >> MailScanner agraeix a transtec Computers pel seu suport. 
> >> _______________________________________________
> >> Lustre-discuss mailing list
> >> Lustre-discuss at lists.lustre.org
> >> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >>
> >> ------------------------------------------------------------------------
> >>
> >> _______________________________________________
> >> Lustre-discuss mailing list
> >> Lustre-discuss at lists.lustre.org
> >> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.