[lustre-discuss] HSM -- requirements -- usage scenario and expectations

[Cowe, Malcolm J]

The POSIX copytool can be used to transact with any archive that presents a POSIX interface. NFS is a common interface onto archives, for example. 

The POSIX CT is supplied in Lustre as a reference implementation of a CT -- many archives have their own interfaces, and these requires their own copytools, so the POSIX CT acts as a reference, a working example of the API. It is therefore, not necessarily particularly optimised.

A ZFS storage system may be a suitable archive; generally it is recommended that an archive can be presented/mounted on multiple HSM agent nodes simultaneously in order to provide multiple paths to the archive storage from Lustre. This allows for increased parallelism and availability. Presenting the ZFS storage to the HSM agents via NFS, for example.

Also be aware that the POSIX CT in Lustre will create a directory structure based on the FID for storing the Lustre files in the archive, with a separate directory tree for recording the name space. The name space tree uses soft links to refer to the actual files in the FID tree.

The following example may help to illustrate the structure of the archive when managed by the POSIX CT:

[root at c64-3a /]# find /archive
/archive
/archive/demo
/archive/demo/shadow
/archive/demo/shadow/f001
/archive/demo/0001
/archive/demo/0001/0000
/archive/demo/0001/0000/0400
/archive/demo/0001/0000/0400/0000
/archive/demo/0001/0000/0400/0000/0002
/archive/demo/0001/0000/0400/0000/0002/0000
/archive/demo/0001/0000/0400/0000/0002/0000/0x200000400:0x1:0x0
/archive/demo/0001/0000/0400/0000/0002/0000/0x200000400:0x1:0x0.lov

[root at c64-3a /]# ls -l /archive/demo/shadow/f001
lrwxrwxrwx 1 root root 52 Aug 1 19:40 /archive/demo/shadow/f001 -> ../0001/0000/0400/0000/0002/0000/0x200000400:0x1:0x0
[root at c64-3a /]# ls -lL /archive/demo/shadow/f001
-rw-r--r-- 1 root root 1048576 Jul 31 23:02 /archive/demo/shadow/f001

Different copytool implementations will have different structures, depending on the requirements of the archive. Not everything uses POSIX, after all.

Malcolm.

--
Malcolm Cowe
Intel High Performance Data Division

> -----Original Message-----
> From: lustre-discuss [mailto:lustre-discuss-bounces at lists.lustre.org] On
> Behalf Of Kevin Abbey
> Sent: Friday, May 08, 2015 11:50 PM
> To: lustre-discuss
> Subject: [lustre-discuss] HSM -- requirements -- usage scenario and
> expectations
> 
> Hi,
> 
> I was reading the HSM documentation and I don't understand the HSM
> requirements exactly.
> 
> Do we need to purchase an HSM solution to manage the migrated data?
> Can
> the POSIX CopyTool copy data to a zfs file system with compression and
> deduplication, as a slow tier for migrating data off of the primary lustre?
> 
> Consider:
>   a single or dual OSS, 500TB total capacity, as it fills to be over 85%
> utilized, performance decreases, then migrate 30%, purge 20% and
> expect
> to observer performance increases.  Is this a viable scenario and
> expectation?
> 
> Can anyone share a link to a general use case, reference implementation
> without using proprietary 3rd party tools?
> 
> Below are references I have read partially thus far.  If there is a
> recorded presentation, video or slides please share the link or a title
> to search in google will be helpful.  I'm guessing that these concepts
> have already been debated during the development of HSM.  I apologize
> if
> the questions here are repetitive on the lustre-discuss list.
> 
> Thanks,
> Kevin
> 
> 
> 
> 
> 
> An introduction to the newly HSM-enabled Lustre 2.5.x parallel file
> system
> http://www.seagate.com/files/www-content/solutions-content/cloud-
> systems-and-solutions/high-performance-
> computing/_shared/docs/clusterstor-inside-lustre-hsm-ti.pdf
> 
> 
> https://wiki.hpdd.intel.com/display/PUB/Lustre+2.5
> http://insidehpc.com/2015/02/inside-lustre-hierarchical-storage-
> management-hsm/
> 
> Managing Data from High Performance Lustre to Deep Tape Archives
> http://web.stanford.edu/group/dlss/pasig/PASIG_September2014/2014
> 0917_Presentations/20140917_08_Managing_Data_from_High_Perform
> ance_Lustre_to_Deep_Tape_Archives_Thomas_Schoenemeyer.pdf
> 
> http://opensfs.org/wp-content/uploads/2012/12/530-
> 600_Aurelien_Degremont_lustre_hsm_lug11.pdf
> 
> 
> --
> Kevin Abbey
> Systems Administrator
> Center for Computational and Integrative Biology (CCIB)
> http://ccib.camden.rutgers.edu/
> 
> Rutgers University - Science Building
> 315 Penn St.
> Camden, NJ 08102
> Telephone: (856) 225-6770
> Fax:(856) 225-6312
> Email: kevin.abbey at rutgers.edu
> 
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org