[Lustre-discuss] File joining mechanism

[Tom.Wang]

Lex wrote:
>
>
> On Wed, Jan 13, 2010 at 5:40 PM, Tom.Wang <Tom.Wang at sun.com 
> <mailto:Tom.Wang at sun.com>> wrote:
>
>     Lex wrote:
>
>         Hi guys
>
>         I have some questions about file joining mechanism in lustre,
>         so i post here and hope you can help me to clarify it :)
>
>         As we know, lustre use the split writing and hierarchy
>         striping to distribute file object ( subfile)  to many osts in
>         the cluster. When a client request one file ( over mds
>         metadata ), these objects are retrieve from each ost to client.
>
>         So, the question is: when and where these objects are joined
>         together to become a real file ?
>
>     File joining is actually joining the file stripe information.
>      which is stored in MDS, so the joining actually happens in MDS,
>     instead of OSS.
>
>  
> Thank you for you fast reply, but i mean the physical joining file 
> process, not just information or metadata. As i know, file ( data ) 
> doesn't go over MDS, right ?  Sorry if i'm misunderstanding you
The stripe information is the layout information of file data, which 
indicates where those data is really stored on OST. And the joining 
process is actually joining these layout information, so the two 
file_data looks like joined together.  For example you have

file_A size 8k stripe_A    file_B size 16k stripe_B

then the stripe information of joinAB will be sth like file extent

[0, 8k] stripe_A   ---------------> original file_A data on OST
[8k, 24k] stripe_B --------------> original file_B data on OST

>  
>
>     Btw: Because of some reasons, joining file feature will be removed
>     temporarily in lustre2.0.
>
>
> Could you please tell me exactly what the reasons are ?
On 2.0, MDS is completely restructured (you can see that on lustre 
roadmap) because of cluster MDS, so join file also needs to be 
re-implemented. But it is a low priority project, and we do not have 
enough resource to do that now. We might do that in some later version.

Thanks
Wangdi
>  
>
>
>     Thanks
>     WangDi
>
>
>         Just guess, are there joined together in OSSs? If it's true,
>         how these OSSs connect together to join its objects together?
>         ( For example, assuming we have 2 OSSs, each OSS have 1 OST,
>         file are striped across to 2 these OSTs. When a client request
>         a file, i get to object in 2 different OST from 2 different
>         OSSs, so how these objects can combined together? )
>
>         Any help would be appreciated :)
>
>         Many thanks
>
>
>
>         ------------------------------------------------------------------------
>
>         _______________________________________________
>         Lustre-discuss mailing list
>         Lustre-discuss at lists.lustre.org
>         <mailto:Lustre-discuss at lists.lustre.org>
>         http://lists.lustre.org/mailman/listinfo/lustre-discuss
>          
>
>
>