<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Arial;
color:#000040;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">The version of tar included in RHEL 7 doesn\u2019t restore the lustre xattrs by default \u2013 you can use the following to extract files with the requisite
xattrs:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">tar --xattrs-include=lustre.* -xf <backup>.tar</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">This assumes the files were backed up with the --xattrs flag:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">tar --xattrs -cf <backup>.tar <file list></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">Note, that you don\u2019t appear to need to whitelist the Lustre xattrs when backing up, only when restoring.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt;font-family:Arial;color:black">Malcolm.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt;font-family:Arial;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="font-family:Calibri;color:black">From:
</span></b><span style="font-family:Calibri;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of "Dilger, Andreas" <andreas.dilger@intel.com><br>
<b>Date: </b>Monday, 20 March 2017 at 8:11 am<br>
<b>To: </b>Brett Lee <brettlee.lustre@gmail.com><br>
<b>Cc: </b>Lustre discussion <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] Backup software for Lustre<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">The use of openat() could be problematic since this precludes storing the xattr before the file is opened. That said, I don't see anywhere in your strace log that (f)setxattr() is called to restore the xattrs,
for either the regular files or directories, even after the file is opened or written? <o:p></o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div id="AppleMailSignature">
<p class="MsoNormal" style="margin-left:36.0pt">Does the RHEL tar have a whitelist of xattrs to be restored? The fact that there are Lustre xattrs after the restore appears to just be normal behavior for creating a file, not anything related to tar restoring
xattrs. <br>
<br>
Cheers, Andreas<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
On Mar 19, 2017, at 10:45, Brett Lee <<a href="mailto:brettlee.lustre@gmail.com">brettlee.lustre@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
Sure, happy to help. I did not see mknod+setxattr in the strace output. Included is a trimmed version of the strace output, along with a few more bits of information. Thanks!<br>
<br>
# cat /proc/fs/lustre/version <br>
lustre: 2.7.19.8<br>
# cat /etc/redhat-release <br>
CentOS Linux release 7.3.1611 (Core) <br>
# uname -r<br>
3.10.0-514.2.2.el7_lustre.x86_64<br>
# rpm -qa|grep tar<br>
tar-1.26-31.el7.x86_64<br>
# sha1sum `which tar` `which gtar`<br>
ea17ec98894212b2e2285eb2dd99aad76185ea7d /usr/bin/tar<br>
ea17ec98894212b2e2285eb2dd99aad76185ea7d /usr/bin/gtar<br>
<br>
Striping was set on the four directories before creating the files.<br>
mkdir -p /scratch/1; lfs setstripe -c 1 --stripe-size 128K /scratch/1; lfs getstripe /scratch/1<br>
mkdir -p /scratch/2; lfs setstripe -c 2 --stripe-size 256K /scratch/2; lfs getstripe /scratch/2<br>
mkdir -p /scratch/3; lfs setstripe -c 3 --stripe-size 768K /scratch/3; lfs getstripe /scratch/3<br>
mkdir -p /scratch/4; lfs setstripe -c 4 --stripe-size 1M /scratch/4; lfs getstripe /scratch/4<br>
After tar, all files and directories all had the default Lustre striping.<br>
<br>
# tar ztvf /scratch.tgz <br>
drwxr-xr-x root/root 0 2017-03-19 10:54 scratch/<br>
drwxr-xr-x root/root 0 2017-03-19 10:57 scratch/4/<br>
-rw-r--r-- root/root 4194304 2017-03-19 10:57 scratch/4/4.dd<br>
drwxr-xr-x root/root 0 2017-03-19 10:57 scratch/3/<br>
-rw-r--r-- root/root 4194304 2017-03-19 10:57 scratch/3/3.dd<br>
drwxr-xr-x root/root 0 2017-03-19 10:57 scratch/1/<br>
-rw-r--r-- root/root 4194304 2017-03-19 10:57 scratch/1/1.dd<br>
drwxr-xr-x root/root 0 2017-03-19 10:57 scratch/2/<br>
-rw-r--r-- root/root 4194304 2017-03-19 10:57 scratch/2/2.dd<br>
<br>
# strace tar zxvf /scratch.tgz > strace.out 2>&1<br>
execve("/usr/bin/tar", ["tar", "zxvf", "/scratch.tgz"], [/* 22 vars */]) = 0<br>
...<br>
(-cut - loading libraries)<br>
...<br>
fstat(1, {st_mode=S_IFREG|0644, st_size=10187, ...}) = 0<br>
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4a63d9f000<br>
write(1, "scratch/\n", 9scratch/<br>
) = 9<br>
mkdirat(AT_FDCWD, "scratch", 0700) = -1 EEXIST (File exists)<br>
newfstatat(AT_FDCWD, "scratch", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0<br>
write(1, "scratch/4/\n", 11scratch/4/<br>
) = 11<br>
mkdirat(AT_FDCWD, "scratch/4", 0700) = 0<br>
write(1, "scratch/4/4.dd\n", 15scratch/4/4.dd<br>
) = 15<br>
openat(AT_FDCWD, "scratch/4/4.dd", <br>
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_NONBLOCK|O_CLOEXEC, 0600) = 4<br>
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 5632) = 5632<br>
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 5632) = 5632<br>
...<br>
(-cut)<br>
...<br>
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512<br>
dup2(4, 4) = 4<br>
fstat(4, {st_mode=S_IFREG|0600, st_size=4194304, ...}) = 0<br>
utimensat(4, NULL, {{1489935825, 0}, {1489935444, 0}}, 0) = 0<br>
fchown(4, 0, 0) = 0<br>
fchmod(4, 0644) = 0<br>
close(4) = 0<br>
write(1, "scratch/3/\n", 11scratch/3/<br>
) = 11<br>
newfstatat(AT_FDCWD, "scratch/4", {st_mode=S_IFDIR|0700, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0<br>
utimensat(AT_FDCWD, "scratch/4", {{1489935825, 0}, {1489935444, 0}}, AT_SYMLINK_NOFOLLOW) = 0<br>
fchownat(AT_FDCWD, "scratch/4", 0, 0, AT_SYMLINK_NOFOLLOW) = 0<br>
fchmodat(AT_FDCWD, "scratch/4", 0755) = 0<br>
mkdirat(AT_FDCWD, "scratch/3", 0700) = 0<br>
write(1, "scratch/3/3.dd\n", 15scratch/3/3.dd<br>
) = 15<br>
openat(AT_FDCWD, "scratch/3/3.dd", O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_NONBLOCK|O_CLOEXEC, 0600) = 4<br>
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 6656) = 6656<br>
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
...<br>
(-cut - pick up with last file...)<br>
...<br>
d(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2476, si_status=0, si_utime=7, si_stime=0} ---<br>
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240<br>
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 7680) = 7680<br>
dup2(4, 4) = 4<br>
fstat(4, {st_mode=S_IFREG|0600, st_size=4194304, ...}) = 0<br>
utimensat(4, NULL, {{1489935825, 0}, {1489935432, 0}}, 0) = 0<br>
fchown(4, 0, 0) = 0<br>
fchmod(4, 0644) = 0<br>
close(4) = 0<br>
clock_gettime(CLOCK_REALTIME, {1489935825, 628399394}) = 0<br>
clock_gettime(CLOCK_REALTIME, {1489935825, 628414336}) = 0<br>
close(3) = 0<br>
wait4(2476, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2476<br>
newfstatat(AT_FDCWD, "scratch/2", {st_mode=S_IFDIR|0700, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0<br>
utimensat(AT_FDCWD, "scratch/2", {{1489935825, 0}, {1489935432, 0}}, AT_SYMLINK_NOFOLLOW) = 0<br>
fchownat(AT_FDCWD, "scratch/2", 0, 0, AT_SYMLINK_NOFOLLOW) = 0<br>
fchmodat(AT_FDCWD, "scratch/2", 0755) = 0<br>
newfstatat(AT_FDCWD, "scratch", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0<br>
utimensat(AT_FDCWD, "scratch", {{1489934977, 0}, {1489935261, 0}}, 0) = 0<br>
fchownat(AT_FDCWD, "scratch", 0, 0, 0) = 0<br>
close(1) = 0<br>
munmap(0x7f4a63d9f000, 4096) = 0<br>
close(2) = 0<br>
exit_group(0) = ?<br>
+++ exited with 0 +++<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><br clear="all">
<o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Brett<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Protect Yourself Against Cybercrime<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">PDS Software Solutions LLC<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><a href="https://www.trustpds.com/" target="_blank">https://www.TrustPDS.com</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">On Sun, Mar 19, 2017 at 7:39 AM, Dilger, Andreas <<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">I ran a test locally with RHEL 6.8 and the included tar 1.26 using strace, and tar is properly using mknod+setxattr to restore the "lov" xattr, and the stripe count and stripe size to be preserved. <o:p></o:p></p>
</div>
<div id="m_4334345014962728702AppleMailSignature">
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div id="m_4334345014962728702AppleMailSignature">
<p class="MsoNormal" style="margin-left:36.0pt">The OST index is not preserved with the xattr restore, since that may cause imbalance if the files were backed up in a different filesystem (e.g. one with fewer OSTs). The MDS will balance OST allocation as
needed for the current OST usage. <o:p></o:p></p>
</div>
<div id="m_4334345014962728702AppleMailSignature">
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div id="m_4334345014962728702AppleMailSignature">
<p class="MsoNormal" style="margin-left:36.0pt">Could you please run your tar on RHEL 7 with strace to see if it is doing this correctly?<o:p></o:p></p>
</div>
<div id="m_4334345014962728702AppleMailSignature">
<p class="MsoNormal" style="margin-left:36.0pt"><br>
Cheers, Andreas<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
<br>
On Mar 18, 2017, at 21:51, Brett Lee <<a href="mailto:brettlee.lustre@gmail.com" target="_blank">brettlee.lustre@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p style="margin-left:36.0pt">Hi Andreas, I expected that to be the case, but found out it was not. Instead, the restore restores everything - unless directed otherwise.<o:p></o:p></p>
<p style="margin-left:36.0pt">Backup == cmd + add xattrs.<br>
Restore == cmd + exclude xattrs.<o:p></o:p></p>
<p style="margin-left:36.0pt">Brett<br>
--<br>
Protect Yourself Against Cybercrime<br>
PDS Software Solutions LLC<br>
<a href="https://www.TrustPDS.com" target="_blank">https://www.TrustPDS.com</a><o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">On Mar 18, 2017 9:28 PM, "Dilger, Andreas" <<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Do you need to specify --xattrs (or similar) during the restore phase as well?<br>
<br>
Cheers, Andreas<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
<br>
On Mar 17, 2017, at 15:12, Brett Lee <<a href="mailto:brettlee.lustre@gmail.com" target="_blank">brettlee.lustre@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
Hi. In what I thought was a valid test, I was unable to confirm that a backup and restore retained the layouts. Perhaps my expectation or process was incorrect? The process was:<br>
<br>
1. Create 4 files, each with different stripe sizes and stripe counts (verified with getstripe).<br>
2. Back up the files using tar-1.26-31.el7.x86_64.<br>
3. Recreate a file system and restore the files.<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Backup command: tar --xattrs -zcvf /scratch.tgz /scratch<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Restore command: tar zxvf /scratch.tgz<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><br>
After restoration, getstripe showed that each file had the default stripe count (1) and stripe size (1MB).<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
FWIW: After restoring, getfattr produced the same result for each file:<br>
# getfattr -d -m - -R <file><br>
lustre.lov=0s0AvRCwEAAAAdAAAAAAAAAAAEAAACAAAAAAAQAAEAAAAFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=<br>
trusted.link=0s3/HqEQEAAAAuAAAAAAAAAAAAAAAAAAAAABYAAAACAAAEAAAAAAUAAAAAMS5kZA==<br>
trusted.lma=0sAAAAAAAAAAAABAAAAgAAAB0AAAAAAAAA<br>
trusted.lov=0s0AvRCwEAAAAdAAAAAAAAAAAEAAACAAAAAAAQAAEAAAAFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=<o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Brett<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Protect Yourself Against Cybercrime<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">PDS Software Solutions LLC<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><a href="https://www.trustpds.com/" target="_blank">https://www.TrustPDS.com</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">On Wed, Mar 15, 2017 at 5:03 AM, Dilger, Andreas <<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">I believe Zmanda is already using GNU tar (or RHEL tar) for the actual backup storage? I that case it should already work, since we fixed tar long ago to backup and restore xattrs in a way that preserves Lustre
layouts. <br>
<br>
Cheers, Andreas<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
<br>
On Mar 14, 2017, at 15:47, Brett Lee <<a href="mailto:brettlee.lustre@gmail.com" target="_blank">brettlee.lustre@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Thanks for the details, Andreas! <o:p>
</o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Maybe OpenSFS can fund Zmanda so that their backup software can include the Lustre metadata... :)<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><br clear="all">
<o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Brett<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Protect Yourself Against Cybercrime<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">PDS Software Solutions LLC<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><a href="https://www.trustpds.com/" target="_blank">https://www.TrustPDS.com</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">On Tue, Mar 14, 2017 at 3:13 PM, Dilger, Andreas <<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal" style="margin-left:36.0pt">To reply to this old thread, there are two different kinds of Lustre backup solutions:<br>
- file level backups that traverse the client POSIX filesystem, for which any number of<br>
commercial solutions exist. Making these solutions "capable of saving Lustre metadata"<br>
boils down to two simple things - save the "lustre.lov" xattr during backup (at a minimum,<br>
other xattrs also should be backed up), and then using mknod(2) + setxattr() to restore<br>
the "lustre.lov" xattr before opening the file and restoring the data.<br>
<br>
- device level backups (e.g. "dd" for ldiskfs, and "zfs send/recv" for ZFS).<br>
<br>
Using the file level backups allows backup/restore of subsets of the filesystem, since many<br>
HPC sites have Lustre filesystems that are too large to backup completely. I typically do<br>
not recommend to use device-level backups for the OSTs, unless doing an OST hardware migration,<br>
and even then it is probably less disruptive to do Lustre-level file migration off the OST<br>
before swapping it out.<br>
<br>
Whether file level backups are used or not, I would recommend sites always make periodic<br>
device level backups of the MDT(s). The amount of space needed for an MDT backup is small<br>
compared to the rest of the filesystem (e.g. a few TB at most), and can avoid the need for<br>
a full filesystem restore (e.g. multi-PB of data, if a full backup exists at all) even<br>
though all the data is still available on the OSTs.<br>
<br>
The MDT device-level backup can use relatively slow SATA drives, since they will mostly be<br>
used for linear writes (or occasionally linear reads for restore), so a few multi-TB SATA III<br>
drives is sufficient for storing a rotating set of MDT device backups. At 150MB/s for even<br>
a single SATA drive, this is about 2h/TB, which is reasonable to do once a week (or more often<br>
for smaller MDTs).<br>
<br>
While using an LVM snapshot of the ldiskfs MDT for the backup source is desirable for consistency<br>
reasons, having even an MDT backup from a mounted and in-use MDT is better than nothing at<br>
all when a problem is hit, since e2fsck can repair the in-use inconsistencies fairly easily,<br>
and Lustre can deal with inconsistencies between the MDT and OST reasonably (at most returning<br>
an -ENOENT error to the client for files that were deleted).<br>
<br>
Cheers, Andreas<br>
<br>
On Feb 7, 2017, at 12:32, Andrew Holway <<a href="mailto:andrew.holway@gmail.com" target="_blank">andrew.holway@gmail.com</a>> wrote:<br>
><br>
> Would it be difficult to suspend IO and snapshot all the nodes (assuming ZFS). Could you be sure that your MDS and OSS are synchronised?<br>
><br>
> On 7 February 2017 at 19:52, Mike Selway <<a href="mailto:mselway@cray.com" target="_blank">mselway@cray.com</a>> wrote:<br>
>> Hello Brett,<br>
>><br>
>> Actually, looking for someone who uses a commercialized approach (that retains user metadata and Lustre extended metadata) and not specifically the manual approaches of Chapter 17.<br>
>><br>
>> Thanks!<br>
>> Mike<br>
>><br>
>> Mike Selway | Sr. Tiered Storage Architect | Cray Inc.<br>
>> Work <a href="tel:%2B1-301-332-4116" target="_blank">+1-301-332-4116</a> | <a href="mailto:mselway@cray.com" target="_blank">
mselway@cray.com</a><br>
>> 146 Castlemaine Ct, Castle Rock, CO 80104 | <a href="http://www.cray.com" target="_blank">
www.cray.com</a><br>
>><br>
>><br>
>>> From: Brett Lee [mailto:<a href="mailto:brettlee.lustre@gmail.com" target="_blank">brettlee.lustre@gmail.com</a>]<br>
>>> Sent: Monday, February 06, 2017 11:45 AM<br>
>>> To: Mike Selway <<a href="mailto:mselway@cray.com" target="_blank">mselway@cray.com</a>><br>
>>> Cc: <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
>>> Subject: Re: [lustre-discuss] Backup software for Lustre<br>
>>><br>
>>> Hey Mike,<br>
>>><br>
>>> "Chapter 17" and<br>
>>> <a href="http://www.intel.com/content/www/us/en/lustre/backup-and-restore-training.html" target="_blank">
http://www.intel.com/content/www/us/en/lustre/backup-and-restore-training.html</a><br>
>>><br>
>>> both contain methods to backup & restore the entire Lustre file system.<br>
>>><br>
>>> Are you looking for a solution that backs up only the (user) data files and their associated metadata (e.g. xattrs)?<br>
>>><br>
>>> Brett<br>
>>> --<br>
>>> Protect Yourself From Cybercrime<br>
>>> PDS Software Solutions LLC<br>
>>> <a href="https://www.TrustPDS.com" target="_blank">https://www.TrustPDS.com</a><br>
>>><br>
>>>> On Mon, Feb 6, 2017 at 11:12 AM, Mike Selway <<a href="mailto:mselway@cray.com" target="_blank">mselway@cray.com</a>> wrote:<br>
>>>><br>
>>>> Hello,<br>
>>>> Anyone aware of and/or using a Backup software package to protect their LFS environment (not referring to the tools/scripts suggested in Chapter 17).<br>
>>>><br>
>>>> Regards,<br>
>>>> Mike<br>
<br>
Cheers, Andreas<br>
--<br>
Andreas Dilger<br>
Lustre Principal Architect<br>
Intel Corporation<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><br>
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>