[lustre-discuss] reflecting state of underlying store in Lustre with HSM
quentin.bouget at cea.fr
quentin.bouget at cea.fr
Wed Jan 8 06:40:51 PST 2020
I don't know much about Amazon's FSx for Lustre, nor about any S3
copytool. But I do know that lhsmtool_posix (the POSIX copytool shipped
with the Lustre sources), supports an "import" operation comparable to
the "lfs hsm_register" Kris describes.
Actually, Amazon's FSx for Lustre probably sets up Lustre filesystems
with something like:
for each file in S3 bucket:
S3-copytool import "$file" into /mnt/lustre
@Kris, I think you should ask Amazon's support if they provide a
dedicated tool that does what you want.
Otherwise, **if you are not afraid of shooting yourself in the foot**,
you can have a look at "llapi_hsm_import()" in
It might be just what you are looking for.
On 08/01/2020 14:13, Matt Rásó-Barnett wrote:
> Hi Kris,
> I assume you are using Amazon's FSx for Lustre product
> (https://aws.amazon.com/fsx/lustre/) for this, rather than rolling
> Lustre HSM to S3 on AWS yourself?
> I'm afraid I don't know any more than you do on this product but it's
> something I've been keen to play with as it sounds really interesting.
> However the process or policy-engine by which Amazon are mapping S3
> objects to files in Lustre is not part of Lustre itself so you'd need
> input from Amazon on this - maybe Aurelien (in BCC) can comment here?
> From their overview documenation my understanding is that the Lustre
> FS is meant to be short lived in this model, so you create it
> populated with the contents of the bucket *at that time*, run your
> workload, then archive results back and delete the FS.
> So if your bucket has been updated post Lustre FS creation, you may
> have to destroy the Lustre FS and recreate it to notice the changes.
> HSM isn't designed to be a mechanism for synchronising two endpoints
> that change independent of the other, so any change to the backend not
> via the filesystem will not be tracked.
> This is just me guessing without having used it however, I'll be
> interested to hear if you learn more about this from Amazon.
> Kind regards,
> On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:
>> We have Lustre <- HSM -> S3
>> We have direct modifications to S3 that occur after the Lustre
>> is created
>> I was wondering if there is any way to register a new/deleted file
>> at the
>> Lustre level using HSM or other commands
>> Say a user uploads a file to S3, and I know the mapped path in Lustre,
>> I would like to do
>> lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in
>> lfs hsm_restore /path/to/file/in/S3 # Fetch file from S3 into
>> On Tue, Jan 7, 2020 at 8:04 AM Colin Faber <cfaber at gmail.com> wrote:
>>> Can you provide an example of what you're attempting to accomplish?
>>> Am I
>>> understanding correctly, that you've got a lustre file system,
>>> you're then
>>> writing data into this file system?
>>> On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval <kris at viqi.org>
>>>> We are using Lustre on AWS backed by S3 buckets.
>>>> When creating a new Lustre filesystem, S3 metadata can be
>>>> imported into Lustre. When changes occur to the underlying S3 store,
>>>> these changes are not automatically reflected.
>>>> Is it possible to indicate the creation / deletion of the
>>>> underlying S3
>>>> files after filesystem creation using HSM?
>>>> Is it possible to reimport the underlying metadata after creation?
>>>> Any pointers appreciated.
>>>> Kris Kvilekval, Ph.D.
>>>> ViQi Inc
>>>> lustre-discuss mailing list
>>>> lustre-discuss at lists.lustre.org
>> Kris Kvilekval, Ph.D.
>> ViQi Inc
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the lustre-discuss