<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Hi everyone,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">I don't know much about Amazon's FSx
for Lustre, nor about any S3 copytool. But I do know that
lhsmtool_posix (the POSIX copytool shipped with the Lustre
sources), supports an "import" operation comparable to the "lfs
hsm_register" Kris describes.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Actually, Amazon's FSx for Lustre
probably sets up Lustre filesystems with something like:</div>
<div class="moz-cite-prefix">
<pre>for each file in S3 bucket:
S3-copytool import "$file" into /mnt/lustre
</pre>
@Kris, I think you should ask Amazon's support if they provide a
dedicated tool that does what you want.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Otherwise, **if you are not afraid of
shooting yourself in the foot**, you can have a look at
"llapi_hsm_import()" in <a moz-do-not-send="true"
href="https://git.whamcloud.com/?p=fs/lustre-release.git;a=blob;f=lustre/utils/liblustreapi_hsm.c;h=8d9938bfb8d69309aef4b32fa09e76e49c062723;hb=HEAD#l1372">lustre/utils/liblustreapi_hsm.c</a>.
It might be just what you are looking for.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Quentin<br>
</div>
<p>On 08/01/2020 14:13, Matt Rásó-Barnett wrote:</p>
<blockquote type="cite"
cite="mid:20200108131330.hepaot2qrjrxzeod@yoshimo.localdomain">Hi
Kris,
<br>
I assume you are using Amazon's FSx for Lustre product
(<a class="moz-txt-link-freetext" href="https://aws.amazon.com/fsx/lustre/">https://aws.amazon.com/fsx/lustre/</a>) for this, rather than rolling
Lustre HSM to S3 on AWS yourself?
<br>
<br>
I'm afraid I don't know any more than you do on this product but
it's something I've been keen to play with as it sounds really
interesting.
<br>
<br>
However the process or policy-engine by which Amazon are mapping
S3 objects to files in Lustre is not part of Lustre itself so
you'd need input from Amazon on this - maybe Aurelien (in BCC) can
comment here?
<br>
<br>
From their overview documenation my understanding is that the
Lustre FS is meant to be short lived in this model, so you create
it populated with the contents of the bucket *at that time*, run
your workload, then archive results back and delete the FS.
<br>
<br>
So if your bucket has been updated post Lustre FS creation, you
may have to destroy the Lustre FS and recreate it to notice the
changes. HSM isn't designed to be a mechanism for synchronising
two endpoints that change independent of the other, so any change
to the backend not via the filesystem will not be tracked.
<br>
<br>
This is just me guessing without having used it however, I'll be
interested to hear if you learn more about this from Amazon.
<br>
<br>
Kind regards,
<br>
<br>
Matt
<br>
<br>
On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval
wrote:
<br>
<blockquote type="cite">We have Lustre <- HSM -> S3
<br>
<br>
We have direct modifications to S3 that occur after the Lustre
filesystem
<br>
is created
<br>
I was wondering if there is any way to register a new/deleted
file at the
<br>
Lustre level using HSM or other commands
<br>
<br>
Say a user uploads a file to S3, and I know the mapped path in
Lustre,
<br>
I would like to do
<br>
lfs hsm_register /path/to/file/in/S3/ # Create a metadata
entry in
<br>
Lustre
<br>
lfs hsm_restore /path/to/file/in/S3 # Fetch file from S3
into Lustre
<br>
<br>
Thx
<br>
<br>
<br>
<br>
<br>
<br>
<br>
On Tue, Jan 7, 2020 at 8:04 AM Colin Faber
<a class="moz-txt-link-rfc2396E" href="mailto:cfaber@gmail.com"><cfaber@gmail.com></a> wrote:
<br>
<br>
<blockquote type="cite">Can you provide an example of what
you're attempting to accomplish? Am I
<br>
understanding correctly, that you've got a lustre file system,
you're then
<br>
writing data into this file system?
<br>
<br>
On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval
<a class="moz-txt-link-rfc2396E" href="mailto:kris@viqi.org"><kris@viqi.org></a> wrote:
<br>
<br>
<blockquote type="cite">We are using Lustre on AWS backed by
S3 buckets.
<br>
When creating a new Lustre filesystem, S3 metadata can be
automatically
<br>
imported into Lustre. When changes occur to the underlying
S3 store,
<br>
these changes are not automatically reflected.
<br>
<br>
Is it possible to indicate the creation / deletion of the
underlying S3
<br>
files after filesystem creation using HSM?
<br>
Is it possible to reimport the underlying metadata after
creation?
<br>
<br>
Any pointers appreciated.
<br>
<br>
Thanks,
<br>
Kris
<br>
<br>
--
<br>
Kris Kvilekval, Ph.D.
<br>
ViQi Inc
<br>
(805)-699-6081
<br>
_______________________________________________
<br>
lustre-discuss mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
-- <br>
Kris Kvilekval, Ph.D.
<br>
ViQi Inc
<br>
(805)-699-6081
<br>
</blockquote>
<br>
<blockquote type="cite">_______________________________________________
<br>
lustre-discuss mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
<br>
</blockquote>
_______________________________________________
<br>
lustre-discuss mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<br>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
<br>
</blockquote>
<p><br>
</p>
</body>
</html>