[lustre-discuss] reflecting state of underlying store in Lustre with HSM

Kristian Kvilekval kris at viqi.org
Tue Jan 7 15:18:39 PST 2020

We have Lustre <- HSM -> S3

We have direct modifications to S3 that occur after the Lustre filesystem
is created
I was  wondering if there is any way to register a new/deleted file at the
Lustre level using HSM or other commands

Say a user uploads a file to S3, and I know  the mapped path in Lustre,
I would like to do
lfs hsm_register /path/to/file/in/S3/     # Create a metadata entry in
lfs hsm_restore /path/to/file/in/S3       # Fetch file from S3 into Lustre


On Tue, Jan 7, 2020 at 8:04 AM Colin Faber <cfaber at gmail.com> wrote:

> Can you provide an example of what you're attempting to accomplish?  Am I
> understanding correctly, that you've got a lustre file system, you're then
> writing data into this file system?
> On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval <kris at viqi.org> wrote:
>> We are using Lustre on AWS backed by S3 buckets.
>> When creating a new Lustre filesystem, S3 metadata can be automatically
>> imported  into Lustre.  When changes occur to the underlying S3 store,
>> these changes are not automatically reflected.
>> Is it possible to indicate the creation / deletion of the underlying S3
>> files after filesystem creation using HSM?
>> Is it possible to reimport the underlying metadata after creation?
>> Any pointers appreciated.
>> Thanks,
>> Kris
>> --
>> Kris Kvilekval, Ph.D.
>> ViQi Inc
>> (805)-699-6081
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Kris Kvilekval, Ph.D.
ViQi Inc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200107/0dc7bfdc/attachment.html>

More information about the lustre-discuss mailing list