<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
<div><span></span></div>
<div>
<div>I've thought about partial HSM archive/restore a few times in conjunction with composite layouts. </div>
<div><br>
</div>
<div id="AppleMailSignature">It wouldn't be too hard to create PFL/composite layouts for chunks of the file arbitrarily (e.g. every 10GB for very large files) so that the file can be restored partially or incrementally or in parallel from multiple tapes. </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">This may also be useful for File Level Redundancy, so that the file can be resync'd in chunks if only part of the file is lost. </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">On the Lustre side, PFL has implemented virtually all of the support needed to do this in the 2.10 release. </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">For HSM this might involve putting the HSM archive information into the composite layout itself, so that different archive identifiers (UUIDs) can be stored for different extents of the file, or possibly different archived replica
versions of the file. That would need some work, and integration with the copy tools and coordinator to be able to understand components that represent partially-restored files. <br>
<br>
Cheers, Andreas</div>
<div><br>
On Jul 27, 2017, at 21:16, Patrick Farrell <<a href="mailto:paf@cray.com">paf@cray.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Hello all,</p>
<p><br>
</p>
<p>There has been some prior conversation around HSM partial archive/restore (Seagate at LUG'17:
<a previewremoved="true" id="LPlnk255171" href="http://cdn.opensfs.org/wp-content/uploads/2017/06/LUG-2017-Lustre-HSM-for-HPC-High-Speed-Distributed-Data-Mover-r1.11.pdf" class="OWAAutoLink">
</a><a previewremoved="true" id="LPlnk179289" href="http://cdn.opensfs.org/wp-content/uploads/2017/06/LUG-2017-Lustre-HSM-for-HPC-High-Speed-Distributed-Data-Mover-r1.11.pdf" class="OWAAutoLink">http://cdn.opensfs.org/wp-content/uploads/2017/06/LUG-2017-Lustre-HSM-for-HPC-High-Speed-Distributed-Data-Mover-r1.11.pdf</a></p>
<p><a previewremoved="true" id="LPlnk255171" href="http://cdn.opensfs.org/wp-content/uploads/2017/06/LUG-2017-Lustre-HSM-for-HPC-High-Speed-Distributed-Data-Mover-r1.11.pdf" class="OWAAutoLink"><br>
<br>
</a><span>With complex file layouts and fallocate support being called out as requirements for enabling this. But as far as I can tell, this has not progressed to any concrete design work.<br>
<br>
Is this correct? Is there work happening somewhere I'm not aware of?<br>
<br>
If not, is there interest in starting some? Cray has interest in this, and could help drive such a project, though we would need outside assistance (particularly with design) since we have no internal expertise relating to complex layouts.</span></p>
<p><br>
</p>
<p>Regards,<br>
<span></span></p>
<p><span>- Patrick<br>
</span></p>
<p><span><br>
</span></p>
<p><span><br>
</span></p>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>_______________________________________________</span><br>
<span>lustre-devel mailing list</span><br>
<span><a href="mailto:lustre-devel@lists.lustre.org">lustre-devel@lists.lustre.org</a></span><br>
<span><a href="http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a></span><br>
</div>
</blockquote>
</div>
</body>
</html>