[Lustre-discuss] lustre and small files overhead

[Balagopal Pillai]

Hi,

        Please read the i/o tunables in the proc section in the lustre 
manual.
I tried that with the postmark benchmark to test improvement and there
was some improvement after trying the suggestions in the Lustre manual.


Regards
Balagopal

Joe Barjo wrote:
> Hi
>
> We have a (small) 30 node sge based cluster with centos4 which will be
> growing to maximum 50 core duos.
> We use custom software that is based on gmake to launch parallel
> compilation and computations with lot of small files and some large files.
> We actualy use nfs and have a lot of problems with incoherencies between
> nodes.
>
> I'm currently evaluating lustre and have some questions about lustre
> overhead with small files.
> I succesfully installed the rpms on a test machine and launched the
> local lmount.sh script.
> The first thing I tried is to make a svn checkout into it. (lot of small
> files...)
> It takes 1m54 from our local svn server versus 15s into a local ext3
> filesystem and 50s over nfs network.
> During the checkout, the processor (amd64 3200) is busy with 90% system.
>
> How come is there so much system process?
> Is there something to tweak to lower this overhead?
> Is there a specific tweak for small files?
> Using multiple server nodes, will the performance be better?
>
> Thanks
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>