[Lustre-discuss] File joining mechanism

[Nicolas Williams]

On Thu, Jan 14, 2010 at 09:49:13AM +0700, Lex wrote:
> You mean lustre client? Our lustre client work as a web server for
> downloading file, it's using ssd drive and the free space is quite small,
> just few Gbs, but it's serving many 10GB size file quite well. If the
> joining file process is on the client, how the big file can fit the poor
> free space well like this ? Could you please explain it for me ?

First, I thought you meant by "file join" something very different than
what we (Lustre team) mean by "file join".  We mean this: an
optimization of "cat file1 file2 ... > joined-file".  I think you meant:
how the illusion of a single file is created out of multiple stripes.

Do correct me if my interpretation of your question was incorrect.

If I understood you correctly, then there is no "physically joined"
version of a file, except in the case of files with a single stripe.

The illusion of "joined" stripes is presented by the client.

The client need not have enough memory to store a whole file.  It only
needs enough for the application's needs.  E.g., if you have a 1TB file
and 4GB of virtual memory, then you can't read the whole file into
memory, but you can read a couple of GBs at a time, do some work, then
read the next couple of GBs.

Nico
--