[Lustre-devel] Lustre HSM HLD draft

[Peter J Braam]

Just a few initial responses from me, I haven't read things 
systematically yet.

Canon, Richard Shane wrote:
> Aurelien and JC,
>
> Sorry that my feedback is late.  Here are my questions/remarks.
>
> General
> * Any thought on how quotas will be handled?
>
>   
This is very very important and will require a lot of detail.  Well 
spotted Shane!!!
> Coordinator
> * 3.4 - I was curious what the precise use case was that was driving
> this?  I don't disagree with it, but I was curious for more background
>   
In internal migrations many objects will be restriped to another set of 
objects to move the data.  The coordinator handles the completion and 
abortion of the agents accomplishing this.
> * 3.7.1 - The coordinator could become a scaling bottleneck.  We should
> think about how this will be scaled in the future
>   
In my writings I was always anticipating a family of load balancing 
coordinators.
> * 4.1 - Does the coordinator store the ext obj id or does the agent
>   
Coordinator, I suggest, in view of the fact that many agents may be 
required to move one file.
> * 4.3 item 2 - This looks like the coordinator could become a bottle
> neck for unlinks and slow down performance.  Could this be put in some
> type of async queue to be processed later (or some type of attic space)?
>
>   
I agree with this.

> Use Cases
> * 2.3 (Use cases) - I'm really keen on this feature.  I think it is very
> important in order to make small file performance work well.
> Unfortunately, it isn't clear how the file list gets communicated to the
> archive tool.  The coordinator and agent seem to only take one file at a
> time.  So how would this work exactly?
> * 2.4 - The copy tool should be allowed to preemptively restage files.
> I think this will work with the design, but we should make sure of this.
> This would be useful for restaging a whole tar file versus doing things
> piece-meal.
>
> Part IV
> 2 EAs - I'm worried that the EA list could get huge for holes.
>   
The EA merely points to an extent tree (similar to the allocation extent 
tree).
> 3.2 -item 3 - Who insures a file is archived before punches are made?
>   
The coordinator.
> 3.3 - Another use case...  The user checks to see if a file has been
> archived.
>
>
> Also, someone earlier made the point about the archive tool being able
> to reorder request.  This is really important since an archival system
> wants to know all the files being restaged in order to order tape mounts
> and reads.  
>
> Thanks for taking the lead on this.  It looks like there is a lot of
> interest in it.
>
> --Shane
>
> -----Original Message-----
> From: lustre-devel-bounces at lists.lustre.org
> [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of DEGREMONT
> Aurelien
> Sent: Thursday, February 07, 2008 5:53 AM
> To: lustre-devel at lists.lustre.org
> Subject: [Lustre-devel] Lustre HSM HLD draft
>
> Hello
>
> Here is a first draft for comments of the Lustre HSM HLD.
> It is intended to be a support for further analyzes and comments from
> CFS/Sun.
>
> The document covers the main parts of the HSM features but some elements
> are still lacking.
> The policy management and the space manager will be describe later.
>
> Let us know your comments and ideas about it.
>
> Regards,
>
> Aurelien Degremont
> CEA
>
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
>