[Lustre-devel] SAM-QFS, ADM, and Lustre HSM

[Vicky White]

Comments on several messages; I am slowly catching up.


> I do see work to switch the HPSS APIs to ftp or pftp. If this is  
> already supported by HPSS, then, yes, no changes are required.
>   

HPSS supports ftp and pftp.   However, this seems to be a moot point as 
Thomas points out that CEA is using the HPSS client API library for 
their copy tool:


> At CEA, we are using our own copytool that directly uses HPSS API. 
> This already exists and is in production for years.
> I think there will be few modifications to adapt it to Lustre-HSM purpose
> (basically, add fid <-> HSM id mapping and backup of attributes, path, 
> stripe...)


>   There is also "htar" which is a tar-like interface to
> HPSS, but I don't think that was anyone's intention to use.


htar is a well proven and valuable tool for aggregation to HPSS.   It is 
widely used at HPSS sites as a stand-alone utility and has been 
incorporated into other interfaces. 


> Looks like HPSS will support EA in 7.1.2.0, June 2009
> I have asked Vicky here at ORNL to dig a bit into what the EA features will look like. 
>   

The last draft of this design I saw was from November.    Work on this 
is picking up right now and has been bumped to a high priority, due for 
release this June, as Galen says.   I am trying to find out if there is 
a later design and how much about it I can share.


> Do we have a set of requirements for EAs for HSM integration?

I never saw an answer to Galen's question above; did I miss it?  Now is 
the time to speak up if we need to influence the design of the HPSS EAs.


>   > We would need to decide whether the HPSS implementation can/should
>   > handle aggregating multiple small files into a single archive object.
>   > I think that is useful, and this is one reason I advocate being able
>   > to pass multiple files at once from the coordinator to the agent.
> Last I knew, they still don't build a container for small files. They 
> write
> a tape mark between each file. This means they are start/stopping the
> tape for small files. A lot of sites use SRB which builds a tar container.

As of HPSS 7.1, we build a container for small files before copying them 
to tape.  It's called Tape Aggregation and we call the container an 
aggregate.   Tape Aggregation is controlled via the HPSS migration 
policy, where the sysadm can configure whether or not to aggregate, the 
minimum and maximum files to place in each aggregate, and the maximum 
size of each aggregate.


Vicky