[Lustre-discuss] lustre and small files overhead

Joe Barjo jobarjo78 at yahoo.fr
Fri Feb 29 06:37:27 PST 2008


Hi

We have a (small) 30 node sge based cluster with centos4 which will be
growing to maximum 50 core duos.
We use custom software that is based on gmake to launch parallel
compilation and computations with lot of small files and some large files.
We actualy use nfs and have a lot of problems with incoherencies between
nodes.

I'm currently evaluating lustre and have some questions about lustre
overhead with small files.
I succesfully installed the rpms on a test machine and launched the
local lmount.sh script.
The first thing I tried is to make a svn checkout into it. (lot of small
files...)
It takes 1m54 from our local svn server versus 15s into a local ext3
filesystem and 50s over nfs network.
During the checkout, the processor (amd64 3200) is busy with 90% system.

How come is there so much system process?
Is there something to tweak to lower this overhead?
Is there a specific tweak for small files?
Using multiple server nodes, will the performance be better?

Thanks



More information about the lustre-discuss mailing list