[Lustre-discuss] Optimize parallel commpilation on lustre

[Maxence Dunnewind]

Hi,

To explain things, I'm working on Kerrighed (www.kerrighed.org)  and I am trying
to find a system "better than nfs" for 4 to 256 clients (atm, maybe more later).
The fs should :
- client cache coherent
- the most POSIX compatible
- the most efficient in "general use".

Lustre seems indeed pretty good for client cache coherency and (it seems) POSIX
compliance (ie: I can't fine other fs with same properties).

> I don't think it is realistic to expect that a cache-coherent distributed filesystem that can scale to 10000's of clients is also performing as fast as a single client on a local filesystem.  That Lustre is completing in 4:19 vs. 3:50 on the local filesystem (12% slower) is a pretty good result for Lustre, I think.
To be more exact, lustre on 4 nodes is completing on 4:19 vs 3:50 on one node :)
 
I know a filesystem can't be good for everything, my question could be
understood as "is there some basic (or a bit more complex) tunning to improve
compilation/ I/O on small files".
Lustre seems, as I said, good for cache coherency and POSIX compliance (maybe
note the best, but definitively not the worst).
> We're of course working on improving Lustre performance for this kind of situation, but it isn't really a priority for most of our customers.  I don't want to discourage you from using Lustre, and of course I'd also like Lustre to be faster than even the local filesystem but you should also look at the other benefits.
If you know some other filesystem with such properties, I can try them, but I
did not find any of them (it also must be free software).
> 
> A more fair test might be to do the local-node compile, and then copy the kernel and all the modules to each of the client nodes, since Lustre is also making the output files available on all of the clients.  It is also worthwhile (if you have the time) to determine whether your "make -j 16" is CPU bound, or IO bound on the local filesystem?  You might try pre-staging all of the input files on the client nodes, and have the compiler output go into a separate directory (not sure if this is possible with linux kernel compiles) so that the output files created during the run do not invalidate the directory caches.
I will try this.
 
 
> For comparison, on the same two systems (local fs vs. Lustre) try writing 32*10GB files from 32 clients (use rsh or NFS or whatever you want to transport data from clients to local filesystem) and see how performance compares. :-)
I would try to find some more "practical" tests, maybe something video related
on some "big" videos, or some parrallel work on big images, or ... 

Thanks for your answers, 

Regards,

Maxence
-- 
Maxence DUNNEWIND
Contact : maxence at dunnewind.net
Site : http://www.dunnewind.net
06 32 39 39 93
GPG : 18AE 61E4 D0B0 1C7C AAC9  E40D 4D39 68DB 0D2E B533
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100625/92f26652/attachment.pgp>