[lustre-devel] Recommendation for hsm_restore for directories

[Andreas Dilger]

On Nov 24, 2021, at 10:33, Degremont, Aurelien via lustre-devel <lustre-devel at lists.lustre.org<mailto:lustre-devel at lists.lustre.org>> wrote:

Hello all

I'm doing some early work toward implementing a directory/namespace support for Lustre/HSM. The idea would be to support something similar to hsm_restore/hsm_release but for directories.

My first thought was for the MDT to not grant LDLM lock for this directory and hold it, while the directory is getting restored by copytool, in a similar fashion than what is done for files. But the hard part here, is to have a way for copytool to access the directory while its access is actually prevented by the above locking.
Restoring a file is using this smart trick of restoring it in a different temporary file and using layout swap at the end to move data to the actual real file. It looks difficult to do the same thing here.

Do you have any recommendation on the right way to frame this directory access for copytools?

I've thought about this in the past, and IMHO the best way to handle a directory or directory tree would be to save the directory and/or whole tree in an ldiskfs image rather than a tarball.  The reasoning for this is that in the future it would allows directly mounting the image for access by the MDT and/or client (Client Container Image), unlike a tarball that needs to be extracted each time before use.

There is some discussion about this in https://jira.whamcloud.com/browse/LU-13024

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20211208/54192be5/attachment.html>