[Lustre-discuss] Working with large amounts of files

[Alvaro Aguilera]

Hello,

I was wondering what ideas are in Lustre's roadmap to address the issues
with serialization and management that appear when dealing with large
amounts of files in parallel.
Clusters are getting more and more cores, with some of them having hundreds
of thousands already, file systems, however, continue with the paradigm of
individual files, even on massively parallel contexts. Wouldn't it be better
for parallel file system like Lustre to somehow bundle files together for a
group of tasks? If I recall correctly, there is a proposal to extend POSIX
to achieve some of this.

I would love to hear some insight about the topic.


Thanks,
Alvaro.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090924/65c128ab/attachment.htm>