[lustre-discuss] status of HSM copytools?

[Matt Rásó-Barnett]

Hi Andrew,

We are using Lemur in production at Cambridge with our Lustre HSM setup, 
although in our case we are using the POSIX backend of it rather than 
the S3 backend.

We adopted it early at the time it was being developed under Intel, as 
the prospect of commercial support (we were Intel IEEL customers at the 
time), and some of the promising features that were being worked on with 
it (snapshot versions, checksums, performance), was very enticing.

But as you've seen, it was a casualty of Intel dropping Lustre, so it's 
not being developed any more. We unfortunately don't have the skills or 
resources internally to develop it ourselves, so I personally am looking 
at how we will transition away from it in the coming months as we 
refresh our HSM setup. I may switch back to using the robinhood 
developed lhsmtool_cmd with our POSIX backends for example.

Lemur is largely doing the job for us however, so it reached a decent 
stage of development before Intel dropped it, but I still wouldn't 
recommend starting with it without someone backing maintenance of it.

 From speaking with the Amazon representative at last year's LUG, I 
understood that they are using Lemur internally for their HSM 
integration as well. I had meant to follow up with them to see if they 
would consider open-sourcing their fork of the project as to my 
knowledge there aren't any other major users of Lemur.

I'd also be interested as you are in hearing what others are using, 
commericial or otherwise. I've never had the feeling that HSM is 
particularly widely used so it'd be good to share information for those 
of us who are!

Best wishes,

Matt

On Sat, Aug 22, 2020 at 10:54:23PM +1000, Andrew Elwell wrote:
>Hi folks,
>I'm looking round to see what's current / 'supported' / working in the
>state of copytools. Ideally one that can migrate to/from object stores
>(Ceph or S3). The github repo for Lemur
>(https://github.com/whamcloud/lemur/commits/master) doesn't seem to
>have had any substantial work since it left Intel - unlucky timing
>with the owner shift?
>I've seen another from Compute Canada
>(https://github.com/ComputeCanada/lustre-obj-copytool) but that too
>hasn't been touched for years.
>
>Anyone care to comment on some working ones? Horror stories? Ones to avoid?
>hey, I'm even (I'll probably regret this) open to _email_ from
>salesdroids if you have a working product and can point me to some
>users (but don't try and phone me or make me sit through a webinar).
>
>Many thanks
>
>
>Andrew
>_______________________________________________
>lustre-discuss mailing list
>lustre-discuss at lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org