[Lustre-discuss] Optimize parallel commpilation on lustre

[Andreas Dilger]

On 2010-06-25, at 00:51, Maxence Dunnewind wrote:
>> I don't think it is realistic to expect that a cache-coherent distributed filesystem that can scale to 10000's of clients is also performing as fast as a single client on a local filesystem.  That Lustre is completing in 4:19 vs. 3:50 on the local filesystem (12% slower) is a pretty good result for Lustre, I think.
> To be more exact, lustre on 4 nodes is completing on 4:19 vs 3:50 on one node :)

I recognized this, but that it gets better performance as nodes are added is a _good_ sign for a filesystem that should perform well when there are many clients.  Of course, it would be ideal if the same performance can be had by a single client...

> I know a filesystem can't be good for everything, my question could be understood as "is there some basic (or a bit more complex) tunning to improve compilation/ I/O on small files".

If you are interested to do a tiny bit of hacking, it would be interesting to do an experiment to see what kind of performance can be gotten in your benchmark by a single client.  Currently, Lustre limits each client to a single filesystem-modifying metadata operation at one time, in order to prevent the clients from overwhelming the server, and to ensure that the clients can recover the filesystem correctly in case of a server crash.

However, for testing purposes it would be interesting to disable this limit to see how fast your clients run if this limit is removed.  I haven't tested the attached patch at all, so YMMV, but I'd be interested to see the results.

I'm not sure if it makes a difference in your case or not, but increasing the MDC RPCs in flight might also help performance.  Also, increasing the client cache size and the number of IO RPCs may also help.  On the clients run:

lctl set_param *.*.max_rpcs_in_flight=64
lctl set_param osc.*.max_dirty_mb=512

to see if these make any difference.

You may also test running the make directly on the MDS with a local Lustre mount to determine if the network latency is a significant factor in the performance. If you are using Ethernet instead of IB the latency could be hurting you, since kernel compiles are generally only doing a tiny amount of work per file and then you need to send a few RPCs to open and read the next file and the headers.  Some of this can be hidden by pre-reading all of the files into the client caches (new machines should have enough RAM, about 1GB or so), but the "open" operations still need to send an RPC to the MDS for each file open, so running on the MDS (or with a low-latency network like IB) may help compiles like this run more quickly.

Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mdc-multiop.diff
Type: application/octet-stream
Size: 803 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100625/9eead638/attachment.obj>