[Lustre-discuss] File joining mechanism

[Lex]

Hi Wangdi

I understand the process that you said and give me a clearly example. So, In
the lustre system internal, there is no "physical" file joining process ? My
lustre client works as web server, serving download. In this case, the
"physical" data joining process are on our web client , isn't it ?

One more thing, just need a confirmation, in the joining file process, OSSs
doesn't have any role, right ? They just handle I/O action?

Sorry if i'm rising such a stupid question :)

Many thanks

On Thu, Jan 14, 2010 at 10:55 AM, Tom.Wang <Tom.Wang at sun.com> wrote:

> Lex wrote:
>
>
>>
>> On Wed, Jan 13, 2010 at 5:40 PM, Tom.Wang <Tom.Wang at sun.com <mailto:
>> Tom.Wang at sun.com>> wrote:
>>
>>    Lex wrote:
>>
>>        Hi guys
>>
>>        I have some questions about file joining mechanism in lustre,
>>        so i post here and hope you can help me to clarify it :)
>>
>>        As we know, lustre use the split writing and hierarchy
>>        striping to distribute file object ( subfile)  to many osts in
>>        the cluster. When a client request one file ( over mds
>>        metadata ), these objects are retrieve from each ost to client.
>>
>>        So, the question is: when and where these objects are joined
>>        together to become a real file ?
>>
>>    File joining is actually joining the file stripe information.
>>     which is stored in MDS, so the joining actually happens in MDS,
>>    instead of OSS.
>>
>>  Thank you for you fast reply, but i mean the physical joining file
>> process, not just information or metadata. As i know, file ( data ) doesn't
>> go over MDS, right ?  Sorry if i'm misunderstanding you
>>
> The stripe information is the layout information of file data, which
> indicates where those data is really stored on OST. And the joining process
> is actually joining these layout information, so the two file_data looks
> like joined together.  For example you have
>
> file_A size 8k stripe_A    file_B size 16k stripe_B
>
> then the stripe information of joinAB will be sth like file extent
>
> [0, 8k] stripe_A   ---------------> original file_A data on OST
> [8k, 24k] stripe_B --------------> original file_B data on OST
>
>
>
>>    Btw: Because of some reasons, joining file feature will be removed
>>    temporarily in lustre2.0.
>>
>>
>> Could you please tell me exactly what the reasons are ?
>>
> On 2.0, MDS is completely restructured (you can see that on lustre roadmap)
> because of cluster MDS, so join file also needs to be re-implemented. But it
> is a low priority project, and we do not have enough resource to do that
> now. We might do that in some later version.
>
> Thanks
> Wangdi
>
>>
>>
>>    Thanks
>>    WangDi
>>
>>
>>        Just guess, are there joined together in OSSs? If it's true,
>>        how these OSSs connect together to join its objects together?
>>        ( For example, assuming we have 2 OSSs, each OSS have 1 OST,
>>        file are striped across to 2 these OSTs. When a client request
>>        a file, i get to object in 2 different OST from 2 different
>>        OSSs, so how these objects can combined together? )
>>
>>        Any help would be appreciated :)
>>
>>        Many thanks
>>
>>
>>
>>
>>  ------------------------------------------------------------------------
>>
>>        _______________________________________________
>>        Lustre-discuss mailing list
>>        Lustre-discuss at lists.lustre.org
>>        <mailto:Lustre-discuss at lists.lustre.org>
>>
>>        http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100114/a58623b4/attachment.htm>