[lustre-discuss] reflecting state of underlying store in Lustre with HSM

[Kristian Kvilekval]

We have Lustre <- HSM -> S3

We have direct modifications to S3 that occur after the Lustre filesystem
is created
I was  wondering if there is any way to register a new/deleted file at the
Lustre level using HSM or other commands

Say a user uploads a file to S3, and I know  the mapped path in Lustre,
I would like to do
lfs hsm_register /path/to/file/in/S3/     # Create a metadata entry in
Lustre
lfs hsm_restore /path/to/file/in/S3       # Fetch file from S3 into Lustre

Thx






On Tue, Jan 7, 2020 at 8:04 AM Colin Faber <cfaber at gmail.com> wrote:

> Can you provide an example of what you're attempting to accomplish?  Am I
> understanding correctly, that you've got a lustre file system, you're then
> writing data into this file system?
>
> On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval <kris at viqi.org> wrote:
>
>> We are using Lustre on AWS backed by S3 buckets.
>> When creating a new Lustre filesystem, S3 metadata can be automatically
>> imported  into Lustre.  When changes occur to the underlying S3 store,
>> these changes are not automatically reflected.
>>
>> Is it possible to indicate the creation / deletion of the underlying S3
>> files after filesystem creation using HSM?
>> Is it possible to reimport the underlying metadata after creation?
>>
>> Any pointers appreciated.
>>
>> Thanks,
>> Kris
>>
>> --
>> Kris Kvilekval, Ph.D.
>> ViQi Inc
>> (805)-699-6081
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>

-- 
Kris Kvilekval, Ph.D.
ViQi Inc
(805)-699-6081
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200107/0dc7bfdc/attachment.html>