[lustre-discuss] reflecting state of underlying store in Lustre with HSM

[quentin.bouget at cea.fr]

Hi everyone,

I don't know much about Amazon's FSx for Lustre, nor about any S3 
copytool. But I do know that lhsmtool_posix (the POSIX copytool shipped 
with the Lustre sources), supports an "import" operation comparable to 
the "lfs hsm_register" Kris describes.

Actually, Amazon's FSx for Lustre probably sets up Lustre filesystems 
with something like:

for each file in S3 bucket:
     S3-copytool import "$file" into /mnt/lustre

@Kris, I think you should ask Amazon's support if they provide a 
dedicated tool that does what you want.

Otherwise, **if you are not afraid of shooting yourself in the foot**, 
you can have a look at "llapi_hsm_import()" in 
lustre/utils/liblustreapi_hsm.c 
<https://git.whamcloud.com/?p=fs/lustre-release.git;a=blob;f=lustre/utils/liblustreapi_hsm.c;h=8d9938bfb8d69309aef4b32fa09e76e49c062723;hb=HEAD#l1372>. 
It might be just what you are looking for.

Quentin

On 08/01/2020 14:13, Matt Rásó-Barnett wrote:

> Hi Kris,
> I assume you are using Amazon's FSx for Lustre product 
> (https://aws.amazon.com/fsx/lustre/) for this, rather than rolling 
> Lustre HSM to S3 on AWS yourself?
>
> I'm afraid I don't know any more than you do on this product but it's 
> something I've been keen to play with as it sounds really interesting.
>
> However the process or policy-engine by which Amazon are mapping S3 
> objects to files in Lustre is not part of Lustre itself so you'd need 
> input from Amazon on this - maybe Aurelien (in BCC) can comment here?
>
> From their overview documenation my understanding is that the Lustre 
> FS is meant to be short lived in this model, so you create it 
> populated with the contents of the bucket *at that time*, run your 
> workload, then archive results back and delete the FS.
>
> So if your bucket has been updated post Lustre FS creation, you may 
> have to destroy the Lustre FS and recreate it to notice the changes. 
> HSM isn't designed to be a mechanism for synchronising two endpoints 
> that change independent of the other, so any change to the backend not 
> via the filesystem will not be tracked.
>
> This is just me guessing without having used it however, I'll be 
> interested to hear if you learn more about this from Amazon.
>
> Kind regards,
>
> Matt
>
> On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:
>> We have Lustre <- HSM -> S3
>>
>> We have direct modifications to S3 that occur after the Lustre 
>> filesystem
>> is created
>> I was  wondering if there is any way to register a new/deleted file 
>> at the
>> Lustre level using HSM or other commands
>>
>> Say a user uploads a file to S3, and I know  the mapped path in Lustre,
>> I would like to do
>> lfs hsm_register /path/to/file/in/S3/     # Create a metadata entry in
>> Lustre
>> lfs hsm_restore /path/to/file/in/S3       # Fetch file from S3 into 
>> Lustre
>>
>> Thx
>>
>>
>>
>>
>>
>>
>> On Tue, Jan 7, 2020 at 8:04 AM Colin Faber <cfaber at gmail.com> wrote:
>>
>>> Can you provide an example of what you're attempting to accomplish?  
>>> Am I
>>> understanding correctly, that you've got a lustre file system, 
>>> you're then
>>> writing data into this file system?
>>>
>>> On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval <kris at viqi.org> 
>>> wrote:
>>>
>>>> We are using Lustre on AWS backed by S3 buckets.
>>>> When creating a new Lustre filesystem, S3 metadata can be 
>>>> automatically
>>>> imported  into Lustre.  When changes occur to the underlying S3 store,
>>>> these changes are not automatically reflected.
>>>>
>>>> Is it possible to indicate the creation / deletion of the 
>>>> underlying S3
>>>> files after filesystem creation using HSM?
>>>> Is it possible to reimport the underlying metadata after creation?
>>>>
>>>> Any pointers appreciated.
>>>>
>>>> Thanks,
>>>> Kris
>>>>
>>>> -- 
>>>> Kris Kvilekval, Ph.D.
>>>> ViQi Inc
>>>> (805)-699-6081
>>>> _______________________________________________
>>>> lustre-discuss mailing list
>>>> lustre-discuss at lists.lustre.org
>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>>
>>>
>>
>> -- 
>> Kris Kvilekval, Ph.D.
>> ViQi Inc
>> (805)-699-6081
>
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200108/8825c810/attachment-0001.html>