[Lustre-discuss] Working with large amounts of files

[Andreas Dilger]

On Sep 24, 2009  22:56 +0200, Alvaro Aguilera wrote:
> I was wondering what ideas are in Lustre's roadmap to address the issues
> with serialization and management that appear when dealing with large
> amounts of files in parallel.
> Clusters are getting more and more cores, with some of them having hundreds
> of thousands already, file systems, however, continue with the paradigm of
> individual files, even on massively parallel contexts. Wouldn't it be better
> for parallel file system like Lustre to somehow bundle files together for a
> group of tasks? If I recall correctly, there is a proposal to extend POSIX
> to achieve some of this.

In many cases, applications are starting to move to large shared file
IO models, because that provides for an easier way to manage the data
once the IO is complete.  It can also (depending on the application)
make it easier to partition the workload onto a different number of
cores.

Some sites have used loopback mounts of Lustre-backed files with an
ext2 filesystem inside in order to provide very efficient access to
a large number of small files.  This is of course not coherent over
multiple nodes, but in some workloads that is acceptable.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.