[lustre-discuss] Data migration software

[Anna Fuchs]

Dear all,

if you have a large Lustre storage and a large tape archive and maybe 
even additionally some in-house cloud storage, which software do you use 
for more or less automatic data migration, that has good scaling?
Ideally it somehow supports Lustre project quota and more important a 
synchronized catalog to find the data.
E.g. if the data to be read is on tape, it somehow transparently moves 
it to the main faster storage (like Lustre) without the user explicitly 
knowing (at least not required to) where the data has been initially stored.
If another user wants to access the same shared file, the software would 
know it is already "buffered" on Lustre and wouldn't read it again from 
tape.
Or If the data has not been touched (really processed, not just touch) 
for a certain period of time, or the user runs out of Lustre quota, but 
has free archive space, it would be automatically archived on tape and 
so on.
Ideally, the software should not cost a billion for a year license or 
even be open source :)

Thank you
Anna Fuchs
--
Universität Hamburg
https://wr.informatik.uni-hamburg.de/people/anna_fuchs