<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="FR" link="blue" vlink="purple" style="word-wrap:break-word;-webkit-nbsp-mode: space;line-break:after-white-space">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Hi Andreas,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">I've been already looking at this ticket and I'm not thinking at the exact same feature (even if not incompatible).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">I think you're thinking at efficient way to backup/restore a Lustre filesystem. I'm more thinking at ways to plug a Lustre FS in front of an existing HSM backend. If the backend already
exists, you need to accept its current data structure (usually file/dir/object based) and not a tarball or disk image.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">But all of this is HSM_ARCHIVE oriented and I'm more thinking of HSM_RESTORE feature for directories. And restoring means detecting and blocking client accesses while the directory is
restored (or mounted).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">At the end, if the copytool is pushing that as a image and restoring a image it is up to copytool in my opinion.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">The question for me is more what should be the way for Lustre to manage Directories if they start to be releasable. What does that mean in term of LDLM locking, internal states, etc…
Do you have an idea for that? My research looking at current code did not give a clear path for that. Implementing something similar to what is done for file is not that easy either.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Aurélien<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b><span style="font-size:12.0pt;color:black">De :
</span></b><span style="font-size:12.0pt;color:black">lustre-devel <lustre-devel-bounces@lists.lustre.org> au nom de Andreas Dilger via lustre-devel <lustre-devel@lists.lustre.org><br>
<b>Répondre à : </b>Andreas Dilger <adilger@whamcloud.com><br>
<b>Date : </b>mercredi 8 décembre 2021 à 22:10<br>
<b>À : </b>"Degremont, Aurelien" <degremoa@amazon.fr><br>
<b>Cc : </b>"lustre-devel@lists.lustre.org" <lustre-devel@lists.lustre.org><br>
<b>Objet : </b>RE: [EXTERNAL] [lustre-devel] Recommendation for hsm_restore for directories<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="625" style="width:468.8pt;margin-left:35.4pt;border-collapse:collapse">
<tbody>
<tr style="height:15.25pt">
<td width="625" valign="top" style="width:468.8pt;border:solid #ED7D31 1.5pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.25pt">
<p><strong><span style="font-family:"Calibri",sans-serif;color:black;background:#FFFF99">CAUTION</span></strong><span style="color:black;background:#FFFF99">: This email originated from outside of the organization. Do not click links or open attachments unless
you can confirm the sender and know the content is safe.</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On Nov 24, 2021, at 10:33, Degremont, Aurelien via lustre-devel <<a href="mailto:lustre-devel@lists.lustre.org">lustre-devel@lists.lustre.org</a>> wrote:<o:p></o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hello all<br>
<br>
I'm doing some early work toward implementing a directory/namespace support for Lustre/HSM. The idea would be to support something similar to hsm_restore/hsm_release but for directories.<br>
<br>
My first thought was for the MDT to not grant LDLM lock for this directory and hold it, while the directory is getting restored by copytool, in a similar fashion than what is done for files. But the hard part here, is to have a way for copytool to access the
directory while its access is actually prevented by the above locking.<br>
Restoring a file is using this smart trick of restoring it in a different temporary file and using layout swap at the end to move data to the actual real file. It looks difficult to do the same thing here.<br>
<br>
Do you have any recommendation on the right way to frame this directory access for copytools?<o:p></o:p></p>
</div>
</div>
</blockquote>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I've thought about this in the past, and IMHO the best way to handle a directory or directory tree would be to save the directory and/or whole tree in an ldiskfs image rather than a tarball. The reasoning for
this is that in the future it would allows directly mounting the image for access by the MDT and/or client (Client Container Image), unlike a tarball that needs to be extracted each time before use.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">There is some discussion about this in
<a href="https://jira.whamcloud.com/browse/LU-13024">https://jira.whamcloud.com/browse/LU-13024</a><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black">Cheers, Andreas<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black">--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black">Andreas Dilger<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black">Lustre Principal Architect<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black">Whamcloud<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:35.4pt">
<o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
</div>
</body>
</html>