<div dir="ltr"><div class="gmail_default" style="font-size:small">So apparently the issue is indeed with the combination of using a Lustre 2.10.1 router with 2.8 servers and clients. Downgrading the router to 2.9 seems to have solved the problem.</div><div class="gmail_default" style="font-size:small">(I can't run 2.8 on the router, because I'm running MOFED 4.1 for the Mellanox ConnectX-5, and I can't get 2.8 to build with that version...)</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Thanks, everyone, for your assistance!<br></div><div class="gmail_default" style="font-size:small">Kevin<br></div><div class="gmail_default" style="font-size:small"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 30, 2017 at 5:47 PM, Dilger, Andreas <span dir="ltr"><<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>The 2.10 release added support for multi-rail LNet, which may potentially be causing problems here. I would suggest to install an older LNet version on your routers to match your client/server. </div>
<div id="m_-7198464434895285311AppleMailSignature"><br>
</div>
<div id="m_-7198464434895285311AppleMailSignature">You may need to build your own RPMs for your new kernel, but can use --disable-server for configure to simplify things. <br>
<br>
Cheers, Andreas</div><div><div class="h5">
<div><br>
On Oct 31, 2017, at 04:45, Kevin M. Hildebrand <<a href="mailto:kevin@umd.edu" target="_blank">kevin@umd.edu</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">
<div class="gmail_default" style="font-size:small">Thanks, I completely missed that. Indeed the ko2iblnd parameters were different between the servers and the router. I've updated the parameters on the router to match those on the server, and things haven't
gotten any better. (The problem appears to be on the Ethernet side anyway, so you've probably helped me fix a problem I didn't know I had...)</div>
<div class="gmail_default" style="font-size:small">I don't see much discussion about configuring lnet parameters for Ethernet networks, I assume that's using ksocklnd. On that side, it appears that all of the ksocklnd parameters match between the router and
clients. Interesting that peer_timeout is 180, which is almost exactly when my client gets marked down on the router.<br>
</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">Server (and now router) ko2iblnd parameters:</div>
<div class="gmail_default" style="font-size:small">peer_credits 8<br>
peer_credits_hiw 4<br>
credits 256<br>
concurrent_sends 8<br>
ntx 512<br>
map_on_demand 0<br>
fmr_pool_size 512<br>
fmr_flush_trigger 384<br>
fmr_cache 1<br>
<br>
</div>
<div class="gmail_default" style="font-size:small">Client and router ksocklnd:</div>
<div class="gmail_default" style="font-size:small">peer_timeout 180<br>
peer_credits 8<br>
keepalive 30<br>
sock_timeout 50<br>
credits 256<br>
rx_buffer_size 0<br>
tx_buffer_size 0<br>
keepalive_idle 30<br>
round_robin 1<br>
sock_timeout 50<br>
<br>
</div>
<div class="gmail_default" style="font-size:small">Thanks,</div>
<div class="gmail_default" style="font-size:small">Kevin<br>
</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Oct 30, 2017 at 4:16 PM, Mohr Jr, Richard Frank (Rick Mohr)
<span dir="ltr"><<a href="mailto:rmohr@utk.edu" target="_blank">rmohr@utk.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span><br>
> On Oct 30, 2017, at 8:47 AM, Kevin M. Hildebrand <<a href="mailto:kevin@umd.edu" target="_blank">kevin@umd.edu</a>> wrote:<br>
><br>
> All of the hosts (client, server, router) have the following in ko2iblnd.conf:<br>
><br>
> alias ko2iblnd-opa ko2iblnd<br>
> options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4<br>
><br>
> install ko2iblnd /usr/sbin/ko2iblnd-probe<br>
<br>
</span>Those parameters will only get applied to omnipath interfaces (which you don’t have), so everything you have should just be running with default parameters. Since your lnet routers have a different version of lustre than your servers/clients, it might
be possible that the default values for the ko2iblnd parameters are different between the two versions. You can always check this by looking at the values in the files under /sys/module/ko2iblnd/parameter<wbr>s. It might be worthwhile to compare those values
on the lnet routers to the values on the servers to see if maybe there is a difference that could affect the behavior.<br>
<br>
--<br>
Rick Mohr<br>
Senior HPC System Administrator<br>
National Institute for Computational Sciences<br>
<a href="http://www.nics.tennessee.edu" rel="noreferrer" target="_blank">http://www.nics.tennessee.edu</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div></div><blockquote type="cite">
<div><span>______________________________<wbr>_________________</span><br>
<span>lustre-discuss mailing list</span><br>
<span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.<wbr>org</a></span><br>
<span><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a></span><br>
</div>
</blockquote>
</div>
</blockquote></div><br></div>