[Lustre-devel] Lustre HSM - some talking points.

[Colin Ngam]

Vicky White wrote:

Hi Vicky,
> Colin Ngam wrote:
>> I do not see why we need to query these EAs in normal operation.  
>> These EAs will only be accessed when we need to perform Ultimate 
>> Disaster Recovery - when you have lost all data on disks and all you 
>> have are tapes.
>
>
> That would help.  I don't know what it would cost to store the EA in a 
> separate object to begin with, though, and that would be incurred on 
> every file.   Plus you have to consider the space it takes up.
>
>
>> I was thinking about XML - but it is "opaque" to Object SAMQFS so, it 
>> is up to the Lustre side.  Whatever it is, the Applications - 
>> Lustre-Restore for example, is the one that has to understand the 
>> format.  I am not a Tar Header expert - but I assume that these EAs 
>> can go with the file in the tar ball.
>
>
> I think what you put in the tar ball is up to you.   Putting the EAs 
> in there regardless of what the hsm was might simplify the design, so 
> you wouldn't have to extract the EA in a different way for each hsm.
>
> I was just trying to keep the hpss EA design in front of folks so that 
> if we were considering using that, we knew all the tradeoffs.
Good point.  I guess EA can be anywhere in the tar file, but, best if 
somehow it is put together for fast access.  But then, we do not need it 
unless it is for Ultimate Disaster Recovery .. that should never happen :-))

With respect to space - how about compression?  The problem is, I always 
thought space is cheap.  Does HPSS ever scrub/recycle?  Policy driven?

If EAs are going to be in a Database, I can see it can be a problem.

Path name does not need to be in the EA.  It is needed for the tar 
header only.  I guess the EA will consist of everything that Lustre 
needs to restore a file, completely.
>
>
>> I do not expect to keep any of it on line on disk cache, on the SAMFS 
>> side.  I see no reason.
>>
>> With respect to whether it should be consistent with HPSS - I would 
>> say if that is all we need and it is sufficient - why not.  
>> Otherwise, let's make it better than HPSS.  It must be in SUN's best 
>> interest to sell SAMQFS?
>
>
> Oh, I'm sure it is.
>
>
>> I do apologize Vicky, are you a SUN employee?
>
>
> No.   Were you going to feel sorry for me if I worked for Sun? ;)
No, it's kind of fun to design with .. and I want to say competitor, but 
I guess you do not really fall into that category.  Say hi to Kim K. or 
Dave W(Cray folks) for me if they cross your path.
>
>
> Vicky
>
PS-The programmer's reference is close to 400 pages!  Perhaps I should 
start with User's Guide.