<div dir="ltr">Thanks very much for the replies.. We are utilizing Lustre as a fast cache on-top of S3.<div> <div>AWS support has simply said they are not sure, but it is hard to break through their defensive first line of support. <div>You have given me several avenues to explore. I did try the --exist flag, but did not have a correct set of parameters. </div><div>I will also see if I can access the s3-copytool directly.</div><div>I will investigate and report back any findings.</div><div><br></div><div>Thank You,</div><div>Kris</div><div> </div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 8, 2020 at 7:03 AM Degremont, Aurelien <<a href="mailto:degremoa@amazon.com">degremoa@amazon.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Kris,<br>
<br>
As people said each HSM has its own copytool with its own constraints. To work properly with S3, additional metadata is stored in Lustre when files are imported from a S3 bucket.<br>
Questions specific to Amazon FSx For Lustre would be better asked to AWS support. <br>
<br>
Aurélien<br>
<br>
Le 08/01/2020 14:14, « Matt Rásó-Barnett » <<a href="mailto:matt@rasobarnett.com" target="_blank">matt@rasobarnett.com</a>> a écrit :<br>
<br>
Hi Kris,<br>
I assume you are using Amazon's FSx for Lustre product <br>
(<a href="https://aws.amazon.com/fsx/lustre/" rel="noreferrer" target="_blank">https://aws.amazon.com/fsx/lustre/</a>) for this, rather than rolling <br>
Lustre HSM to S3 on AWS yourself?<br>
<br>
I'm afraid I don't know any more than you do on this product but it's <br>
something I've been keen to play with as it sounds really interesting.<br>
<br>
However the process or policy-engine by which Amazon are mapping S3 <br>
objects to files in Lustre is not part of Lustre itself so you'd need <br>
input from Amazon on this - maybe Aurelien (in BCC) can comment here?<br>
<br>
From their overview documenation my understanding is that the Lustre FS <br>
is meant to be short lived in this model, so you create it populated <br>
with the contents of the bucket *at that time*, run your workload, then <br>
archive results back and delete the FS.<br>
<br>
So if your bucket has been updated post Lustre FS creation, you may have <br>
to destroy the Lustre FS and recreate it to notice the changes. HSM <br>
isn't designed to be a mechanism for synchronising two endpoints that <br>
change independent of the other, so any change to the backend not via <br>
the filesystem will not be tracked.<br>
<br>
This is just me guessing without having used it however, I'll be <br>
interested to hear if you learn more about this from Amazon.<br>
<br>
Kind regards,<br>
<br>
Matt<br>
<br>
On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:<br>
>We have Lustre <- HSM -> S3<br>
><br>
>We have direct modifications to S3 that occur after the Lustre filesystem<br>
>is created<br>
>I was wondering if there is any way to register a new/deleted file at the<br>
>Lustre level using HSM or other commands<br>
><br>
>Say a user uploads a file to S3, and I know the mapped path in Lustre,<br>
>I would like to do<br>
>lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in<br>
>Lustre<br>
>lfs hsm_restore /path/to/file/in/S3 # Fetch file from S3 into Lustre<br>
><br>
>Thx<br>
><br>
><br>
><br>
><br>
><br>
><br>
>On Tue, Jan 7, 2020 at 8:04 AM Colin Faber <<a href="mailto:cfaber@gmail.com" target="_blank">cfaber@gmail.com</a>> wrote:<br>
><br>
>> Can you provide an example of what you're attempting to accomplish? Am I<br>
>> understanding correctly, that you've got a lustre file system, you're then<br>
>> writing data into this file system?<br>
>><br>
>> On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval <<a href="mailto:kris@viqi.org" target="_blank">kris@viqi.org</a>> wrote:<br>
>><br>
>>> We are using Lustre on AWS backed by S3 buckets.<br>
>>> When creating a new Lustre filesystem, S3 metadata can be automatically<br>
>>> imported into Lustre. When changes occur to the underlying S3 store,<br>
>>> these changes are not automatically reflected.<br>
>>><br>
>>> Is it possible to indicate the creation / deletion of the underlying S3<br>
>>> files after filesystem creation using HSM?<br>
>>> Is it possible to reimport the underlying metadata after creation?<br>
>>><br>
>>> Any pointers appreciated.<br>
>>><br>
>>> Thanks,<br>
>>> Kris<br>
>>><br>
>>> --<br>
>>> Kris Kvilekval, Ph.D.<br>
>>> ViQi Inc<br>
>>> (805)-699-6081<br>
>>> _______________________________________________<br>
>>> lustre-discuss mailing list<br>
>>> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
>>> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
>>><br>
>><br>
><br>
>-- <br>
>Kris Kvilekval, Ph.D.<br>
>ViQi Inc<br>
>(805)-699-6081<br>
<br>
>_______________________________________________<br>
>lustre-discuss mailing list<br>
><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Kris Kvilekval, Ph.D.<br>ViQi Inc<br>(805)-699-6081</div>