[Lustre-discuss] File joining mechanism

[Lex]

Thanks all you guys for helping me. Sorry for my late reply. You gave me
some very useful info. I will consider it carefully and maybe will ask your
help once again :)

Many thanks
On Sat, Jan 16, 2010 at 7:02 AM, Nicolas Williams
<Nicolas.Williams at sun.com>wrote:

> On Thu, Jan 14, 2010 at 09:49:13AM +0700, Lex wrote:
> > You mean lustre client? Our lustre client work as a web server for
> > downloading file, it's using ssd drive and the free space is quite small,
> > just few Gbs, but it's serving many 10GB size file quite well. If the
> > joining file process is on the client, how the big file can fit the poor
> > free space well like this ? Could you please explain it for me ?
>
> First, I thought you meant by "file join" something very different than
> what we (Lustre team) mean by "file join".  We mean this: an
> optimization of "cat file1 file2 ... > joined-file".  I think you meant:
> how the illusion of a single file is created out of multiple stripes.
>
> Do correct me if my interpretation of your question was incorrect.
>
> If I understood you correctly, then there is no "physically joined"
> version of a file, except in the case of files with a single stripe.
>
> The illusion of "joined" stripes is presented by the client.
>
> The client need not have enough memory to store a whole file.  It only
> needs enough for the application's needs.  E.g., if you have a 1TB file
> and 4GB of virtual memory, then you can't read the whole file into
> memory, but you can read a couple of GBs at a time, do some work, then
> read the next couple of GBs.
>
> Nico
> --
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100125/3e99696e/attachment.htm>