[lustre-discuss] HSM -- requirements -- usage scenario and expectations

[Kevin Abbey]

Hi,

I was reading the HSM documentation and I don't understand the HSM 
requirements exactly.

Do we need to purchase an HSM solution to manage the migrated data? Can 
the POSIX CopyTool copy data to a zfs file system with compression and 
deduplication, as a slow tier for migrating data off of the primary lustre?

Consider:
  a single or dual OSS, 500TB total capacity, as it fills to be over 85% 
utilized, performance decreases, then migrate 30%, purge 20% and expect 
to observer performance increases.  Is this a viable scenario and 
expectation?

Can anyone share a link to a general use case, reference implementation 
without using proprietary 3rd party tools?

Below are references I have read partially thus far.  If there is a 
recorded presentation, video or slides please share the link or a title 
to search in google will be helpful.  I'm guessing that these concepts 
have already been debated during the development of HSM.  I apologize if 
the questions here are repetitive on the lustre-discuss list.

Thanks,
Kevin





An introduction to the newly HSM-enabled Lustre 2.5.x parallel file system
http://www.seagate.com/files/www-content/solutions-content/cloud-systems-and-solutions/high-performance-computing/_shared/docs/clusterstor-inside-lustre-hsm-ti.pdf


https://wiki.hpdd.intel.com/display/PUB/Lustre+2.5
http://insidehpc.com/2015/02/inside-lustre-hierarchical-storage-management-hsm/

Managing Data from High Performance Lustre to Deep Tape Archives
http://web.stanford.edu/group/dlss/pasig/PASIG_September2014/20140917_Presentations/20140917_08_Managing_Data_from_High_Performance_Lustre_to_Deep_Tape_Archives_Thomas_Schoenemeyer.pdf

http://opensfs.org/wp-content/uploads/2012/12/530-600_Aurelien_Degremont_lustre_hsm_lug11.pdf


-- 
Kevin Abbey
Systems Administrator
Center for Computational and Integrative Biology (CCIB)
http://ccib.camden.rutgers.edu/
  
Rutgers University - Science Building
315 Penn St.
Camden, NJ 08102
Telephone: (856) 225-6770
Fax:(856) 225-6312
Email: kevin.abbey at rutgers.edu