<div dir="ltr">David,<div><br></div><div>You are right, there is a lock. As Patrick mentioned, <a href="https://jira.hpdd.intel.com/browse/LU-1669">https://jira.hpdd.intel.com/browse/LU-1669</a> will solve your problems. Please check it out.</div><div><br></div><div>In my own experience, Lustre 2.7.0 client does solve such problem very well, and I got a very good performance so far.</div><div><br></div><div>Regards,</div><div>Cuong</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 20, 2015 at 4:46 AM, David A. Schneider <span dir="ltr"><<a href="mailto:davidsch@slac.stanford.edu" target="_blank">davidsch@slac.stanford.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">We do use checksums, but can't turn it off. It know we've measured some performance penalty with checksums. I'll check about configuring lustre clients to to use RDMA. We ran into something similar where our MPI programs were not taking advantage of the infini-band - we noticed much slower message passing then we expected - it sounds like there is a similar thing we can do with lustre, but I guess the locking is the main issue. All our compute nodes are currently running red hat 5 and it doesn't look like lustre 2.6 was tested with rhel5, but we have been talking about moving everything to at least rhel6, maybe rhel7, so there's hope, Thanks for the help!<br>
<br>
best,<br>
<br>
David<div class="HOEnZb"><div class="h5"><br>
<br>
On 05/19/15 11:10, Patrick Farrell wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Ah. I think I know what¹s going on here:<br>
<br>
In Lustre 2.x client versions prior to 2.6, only one process on a given<br>
client can write to a given file at a time, regardless of how the file is<br>
striped. So if you are writing to the same file, there will be little to<br>
no benefit of putting an extra process on the same node.<br>
<br>
A *single* process on a node could benefit, but not the split you¹ve<br>
described.<br>
<br>
The details, which are essentially just that a pair of per-file locks are<br>
used by any individual process writing to a file, are here:<br>
<a href="https://jira.hpdd.intel.com/browse/LU-1669" target="_blank">https://jira.hpdd.intel.com/browse/LU-1669</a><br>
<br>
<br>
On 5/19/15, 12:59 PM, "Mohr Jr, Richard Frank (Rick Mohr)" <<a href="mailto:rmohr@utk.edu" target="_blank">rmohr@utk.edu</a>><br>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On May 19, 2015, at 1:44 PM, Schneider, David A.<br>
<<a href="mailto:davidsch@slac.stanford.edu" target="_blank">davidsch@slac.stanford.edu</a>> wrote:<br>
<br>
Thanks for the suggestion! When I had each rank run on a separate<br>
compute node/host, I saw parallel performance (4 seconds for the 6GB of<br>
writing). When I ran the MPI job on one host (the hosts have 12 cores,<br>
by default we pack ranks onto as few hosts as possible), things happened<br>
serially, each rank finished about 2 seconds after a different rank.<br>
</blockquote>
Hmm. That does seem like there is some bottleneck on the client side that<br>
is limiting the throughput from a single client. Here are some things<br>
you could look into (although they might require more tinkering than you<br>
have permission to do):<br>
<br>
1) Based on your output from ³lctl list_nids², it looks like you are<br>
running IP-over-IB. Can you configure the clients to use RDMA? (They<br>
would have nids like x.x.x.x@o2ib.)<br>
<br>
2) Do you have the option of trying a newer client version? Earlier<br>
lustre versions used a single-thread ptlrpcd to manage network traffic,<br>
but newer versions have a multi-threaded implementation. You may need to<br>
compare compatibility with the Lustre version running on the servers<br>
though.<br>
<br>
3) Do you gave checksums disabled? Try running "lctl get_param<br>
osc.*.checksums². If the values are ³1², then checksums are enabled<br>
which can slow down performance. You could try setting the value to ³0²<br>
to see if that helps.<br>
<br>
--<br>
Rick Mohr<br>
Senior HPC System Administrator<br>
National Institute for Computational Sciences<br>
<a href="http://www.nics.tennessee.edu" target="_blank">http://www.nics.tennessee.edu</a><br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></blockquote>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Nguyen Viet Cuong<br></div>
</div>