[lustre-discuss] reflecting state of underlying store in Lustre with HSM

[Kristian Kvilekval]

Thanks very much for the replies.. We are utilizing Lustre as a fast cache
on-top of S3.

AWS support has simply said they are not sure, but it is hard to break
through their defensive first line of support.
You have given me several avenues to explore. I did try the --exist flag,
but did not have a correct set of parameters.
I will also see if I can access the s3-copytool directly.
I will investigate and report back any findings.

Thank You,
Kris


On Wed, Jan 8, 2020 at 7:03 AM Degremont, Aurelien <degremoa at amazon.com>
wrote:

> Hi Kris,
>
> As people said each HSM has its own copytool with its own constraints. To
> work properly with S3, additional metadata is stored in Lustre when files
> are imported from a S3 bucket.
> Questions specific to Amazon FSx For Lustre would be better asked to AWS
> support.
>
> Aurélien
>
> Le 08/01/2020 14:14, « Matt Rásó-Barnett » <matt at rasobarnett.com> a
> écrit :
>
>     Hi Kris,
>     I assume you are using Amazon's FSx for Lustre product
>     (https://aws.amazon.com/fsx/lustre/) for this, rather than rolling
>     Lustre HSM to S3 on AWS yourself?
>
>     I'm afraid I don't know any more than you do on this product but it's
>     something I've been keen to play with as it sounds really interesting.
>
>     However the process or policy-engine by which Amazon are mapping S3
>     objects to files in Lustre is not part of Lustre itself so you'd need
>     input from Amazon on this - maybe Aurelien (in BCC) can comment here?
>
>      From their overview documenation my understanding is that the Lustre
> FS
>     is meant to be short lived in this model, so you create it populated
>     with the contents of the bucket *at that time*, run your workload,
> then
>     archive results back and delete the FS.
>
>     So if your bucket has been updated post Lustre FS creation, you may
> have
>     to destroy the Lustre FS and recreate it to notice the changes. HSM
>     isn't designed to be a mechanism for synchronising two endpoints that
>     change independent of the other, so any change to the backend not via
>     the filesystem will not be tracked.
>
>     This is just me guessing without having used it however, I'll be
>     interested to hear if you learn more about this from Amazon.
>
>     Kind regards,
>
>     Matt
>
>     On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:
>     >We have Lustre <- HSM -> S3
>     >
>     >We have direct modifications to S3 that occur after the Lustre
> filesystem
>     >is created
>     >I was  wondering if there is any way to register a new/deleted file
> at the
>     >Lustre level using HSM or other commands
>     >
>     >Say a user uploads a file to S3, and I know  the mapped path in
> Lustre,
>     >I would like to do
>     >lfs hsm_register /path/to/file/in/S3/     # Create a metadata entry in
>     >Lustre
>     >lfs hsm_restore /path/to/file/in/S3       # Fetch file from S3 into
> Lustre
>     >
>     >Thx
>     >
>     >
>     >
>     >
>     >
>     >
>     >On Tue, Jan 7, 2020 at 8:04 AM Colin Faber <cfaber at gmail.com> wrote:
>     >
>     >> Can you provide an example of what you're attempting to
> accomplish?  Am I
>     >> understanding correctly, that you've got a lustre file system,
> you're then
>     >> writing data into this file system?
>     >>
>     >> On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval <kris at viqi.org>
> wrote:
>     >>
>     >>> We are using Lustre on AWS backed by S3 buckets.
>     >>> When creating a new Lustre filesystem, S3 metadata can be
> automatically
>     >>> imported  into Lustre.  When changes occur to the underlying S3
> store,
>     >>> these changes are not automatically reflected.
>     >>>
>     >>> Is it possible to indicate the creation / deletion of the
> underlying S3
>     >>> files after filesystem creation using HSM?
>     >>> Is it possible to reimport the underlying metadata after creation?
>     >>>
>     >>> Any pointers appreciated.
>     >>>
>     >>> Thanks,
>     >>> Kris
>     >>>
>     >>> --
>     >>> Kris Kvilekval, Ph.D.
>     >>> ViQi Inc
>     >>> (805)-699-6081
>     >>> _______________________________________________
>     >>> lustre-discuss mailing list
>     >>> lustre-discuss at lists.lustre.org
>     >>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>     >>>
>     >>
>     >
>     >--
>     >Kris Kvilekval, Ph.D.
>     >ViQi Inc
>     >(805)-699-6081
>
>     >_______________________________________________
>     >lustre-discuss mailing list
>     >lustre-discuss at lists.lustre.org
>     >http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>


-- 
Kris Kvilekval, Ph.D.
ViQi Inc
(805)-699-6081
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200108/e5be52e5/attachment.html>