[lustre-discuss] Backup software for Lustre

Fri Mar 17 14:12:39 PDT 2017

Hi.  In what I thought was a valid test, I was unable to confirm that a
backup and restore retained the layouts.  Perhaps my expectation or process
was incorrect?  The process was:

1.  Create 4 files, each with different stripe sizes and stripe counts
(verified with getstripe).
2.  Back up the files using tar-1.26-31.el7.x86_64.
3.  Recreate a file system and restore the files.

Backup command:  tar --xattrs -zcvf /scratch.tgz /scratch
Restore command:  tar zxvf /scratch.tgz

After restoration, getstripe showed that each file had the default stripe
count (1) and stripe size (1MB).
FWIW:  After restoring, getfattr produced the same result for each file:
# getfattr -d -m - -R <file>
lustre.lov=0s0AvRCwEAAAAdAAAAAAAAAAAEAAACAAAAAAAQAAEAAAAFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
trusted.link=0s3/HqEQEAAAAuAAAAAAAAAAAAAAAAAAAAABYAAAACAAAEAAAAAAUAAAAAMS5kZA==
trusted.lma=0sAAAAAAAAAAAABAAAAgAAAB0AAAAAAAAA
trusted.lov=0s0AvRCwEAAAAdAAAAAAAAAAAEAAACAAAAAAAQAAEAAAAFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=

Brett
--
Protect Yourself Against Cybercrime
PDS Software Solutions LLC
https://www.TrustPDS.com <https://www.trustpds.com/>

On Wed, Mar 15, 2017 at 5:03 AM, Dilger, Andreas <andreas.dilger at intel.com>
wrote:

> I believe Zmanda is already using GNU tar (or RHEL tar) for the actual
> backup storage?  I that case it should already work, since we fixed tar
> long ago to backup and restore xattrs in a way that preserves Lustre
> layouts.
>
> Cheers, Andreas
>
> On Mar 14, 2017, at 15:47, Brett Lee <brettlee.lustre at gmail.com> wrote:
>
> Thanks for the details, Andreas!
>
> Maybe OpenSFS can fund Zmanda so that their backup software can include
> the Lustre metadata... :)
>
> Brett
> --
> Protect Yourself Against Cybercrime
> PDS Software Solutions LLC
> https://www.TrustPDS.com <https://www.trustpds.com/>
>
> On Tue, Mar 14, 2017 at 3:13 PM, Dilger, Andreas <andreas.dilger at intel.com
> > wrote:
>
>> To reply to this old thread, there are two different kinds of Lustre
>> backup solutions:
>> - file level backups that traverse the client POSIX filesystem, for which
>> any number of
>>   commercial solutions exist.  Making these solutions "capable of saving
>> Lustre metadata"
>>   boils down to two simple things - save the "lustre.lov" xattr during
>> backup (at a minimum,
>>   other xattrs also should be backed up), and then using mknod(2) +
>> setxattr() to restore
>>   the "lustre.lov" xattr before opening the file and restoring the data.
>>
>> - device level backups (e.g. "dd" for ldiskfs, and "zfs send/recv" for
>> ZFS).
>>
>> Using the file level backups allows backup/restore of subsets of the
>> filesystem, since many
>> HPC sites have Lustre filesystems that are too large to backup
>> completely.  I typically do
>> not recommend to use device-level backups for the OSTs, unless doing an
>> OST hardware migration,
>> and even then it is probably less disruptive to do Lustre-level file
>> migration off the OST
>> before swapping it out.
>>
>> Whether file level backups are used or not, I would recommend sites
>> always make periodic
>> device level backups of the MDT(s).  The amount of space needed for an
>> MDT backup is small
>> compared to the rest of the filesystem (e.g. a few TB at most), and can
>> avoid the need for
>> a full filesystem restore (e.g. multi-PB of data, if a full backup exists
>> at all) even
>> though all the data is still available on the OSTs.
>>
>> The MDT device-level backup can use relatively slow SATA drives, since
>> they will mostly be
>> used for linear writes (or occasionally linear reads for restore), so a
>> few multi-TB SATA III
>> drives is sufficient for storing a rotating set of MDT device backups.
>> At 150MB/s for even
>> a single SATA drive, this is about 2h/TB, which is reasonable to do once
>> a week (or more often
>> for smaller MDTs).
>>
>> While using an LVM snapshot of the ldiskfs MDT for the backup source is
>> desirable for consistency
>> reasons, having even an MDT backup from a mounted and in-use MDT is
>> better than nothing at
>> all when a problem is hit, since e2fsck can repair the in-use
>> inconsistencies fairly easily,
>> and Lustre can deal with inconsistencies between the MDT and OST
>> reasonably (at most returning
>> an -ENOENT error to the client for files that were deleted).
>>
>> Cheers, Andreas
>>
>> On Feb 7, 2017, at 12:32, Andrew Holway <andrew.holway at gmail.com> wrote:
>> >
>> > Would it be difficult to suspend IO and snapshot all the nodes
>> (assuming ZFS). Could you be sure that your MDS and OSS are synchronised?
>> >
>> > On 7 February 2017 at 19:52, Mike Selway <mselway at cray.com> wrote:
>> >> Hello Brett,
>> >>
>> >>                Actually, looking for someone who uses a commercialized
>> approach (that retains user metadata and Lustre extended metadata) and not
>> specifically the manual approaches of Chapter 17.
>> >>
>> >> Thanks!
>> >> Mike
>> >>
>> >> Mike Selway | Sr. Tiered Storage Architect | Cray Inc.
>> >> Work +1-301-332-4116 | mselway at cray.com
>> >> 146 Castlemaine Ct,   Castle Rock,  CO  80104 | www.cray.com
>> >>
>> >>
>> >>> From: Brett Lee [mailto:brettlee.lustre at gmail.com]
>> >>> Sent: Monday, February 06, 2017 11:45 AM
>> >>> To: Mike Selway <mselway at cray.com>
>> >>> Cc: lustre-discuss at lists.lustre.org
>> >>> Subject: Re: [lustre-discuss] Backup software for Lustre
>> >>>
>> >>> Hey Mike,
>> >>>
>> >>> "Chapter 17" and
>> >>> http://www.intel.com/content/www/us/en/lustre/backup-and-res
>> tore-training.html
>> >>>
>> >>> both contain methods to backup & restore the entire Lustre file
>> system.
>> >>>
>> >>> Are you looking for a solution that backs up only the (user) data
>> files and their associated metadata (e.g. xattrs)?
>> >>>
>> >>> Brett
>> >>> --
>> >>> Protect Yourself From Cybercrime
>> >>> PDS Software Solutions LLC
>> >>> https://www.TrustPDS.com
>> >>>
>> >>>> On Mon, Feb 6, 2017 at 11:12 AM, Mike Selway <mselway at cray.com>
>> wrote:
>> >>>>
>> >>>> Hello,
>> >>>>          Anyone aware of and/or using a Backup software package to
>> protect their LFS environment (not referring to the tools/scripts suggested
>> in Chapter 17).
>> >>>>
>> >>>> Regards,
>> >>>> Mike
>>
>> Cheers, Andreas
>> --
>> Andreas Dilger
>> Lustre Principal Architect
>> Intel Corporation
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170317/43fad4a1/attachment.htm>