<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Rick,</div><div dir="ltr"><br><div>That ko2iblnd-opa was the cause, thanks.</div><div><br></div><div>In the default ko2iblnd.conf that was provided, there has
<span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">already<span> </span></span>the ko2iblnd-opa there appeared to be aliased to back to ko2iblnd, I didn't think twice about changing the alias and the -opa lines initially which is why it did not work by just modifying the map_on_demand parameter but not the rest in that file. </div><div><br></div><div>Thank you all.</div></div></div></div><div class="gmail_extra"><br></div><div class="gmail_extra">Regards,</div><div class="gmail_extra">- Pak</div><div class="gmail_extra"><br></div><div class="gmail_extra">My current ko2iblnd.conf that is working:</div></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div class="gmail_extra"><div class="gmail_extra">options ko2iblnd peer_credits=128 peer_credits_hiw=64 credits=1024 ntx=2048 map_on_demand=16 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4</div></div></div></div><div><div><div class="gmail_extra"><div class="gmail_extra">install ko2iblnd /usr/sbin/ko2iblnd-probe</div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div class="gmail_extra"><div><br></div></div><div class="gmail_extra">The default ko2iblnd.conf:<br></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div class="gmail_extra"><div class="gmail_extra"># Currently it isn't possible to auto-tune the o2iblnd parameters optimally</div></div></div><div><div class="gmail_extra"><div class="gmail_extra"># inside the kernel since the OFED API hides the details from us.</div></div></div><div><div class="gmail_extra"><div class="gmail_extra"># Unfortunately, there isn't a single set of parameters that provide optimal</div></div></div><div><div class="gmail_extra"><div class="gmail_extra"># performance on different HCA/HFI types, so this file provides optimized</div></div></div><div><div class="gmail_extra"><div class="gmail_extra"># tunables based on the installed IB interface cards.</div></div></div><div><div class="gmail_extra"><div class="gmail_extra">#</div></div></div><div><div class="gmail_extra"><div class="gmail_extra"># Card detection and tunable selection is handled via /usr/sbin/ko2iblnd-probe</div></div></div><div><div class="gmail_extra"><div class="gmail_extra"># at runtime when the ko2iblnd module is installed, either at boot or when</div></div></div><div><div class="gmail_extra"><div class="gmail_extra"># Lustre is first mounted.</div></div></div><div><div class="gmail_extra"><div class="gmail_extra"><br></div></div></div><div><div class="gmail_extra"><div class="gmail_extra">alias ko2iblnd-opa ko2iblnd</div></div></div><div><div class="gmail_extra"><div class="gmail_extra">options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 ntx=2048 map_on_demand=256 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4</div></div></div><div><div class="gmail_extra"><div class="gmail_extra"><br></div></div></div><div><div class="gmail_extra"><div class="gmail_extra">install ko2iblnd /usr/sbin/ko2iblnd-probe</div></div></div></blockquote><div dir="ltr"><div class="gmail_extra"><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 4, 2018 at 9:40 AM, Mohr Jr, Richard Frank (Rick Mohr) <span dir="ltr"><<a href="mailto:rmohr@utk.edu" target="_blank">rmohr@utk.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-"><br>
> On Sep 4, 2018, at 12:12 PM, Pak Lui <<a href="mailto:pak.lui@linaro.org">pak.lui@linaro.org</a>> wrote:<br>
> <br>
> I have tried "map_on_demand=16" to the "/etc/modprobe.d/ko2iblnd.<wbr>conf" that was suggested. Also tried "map_on_demand=0" as suggested here: <a href="http://wiki.lustre.org/Optimizing_o2iblnd_Performance" rel="noreferrer" target="_blank">http://wiki.lustre.org/<wbr>Optimizing_o2iblnd_Performance</a><br>
> <br>
> /etc/modprobe.d/ko2iblnd.conf<br>
> alias ko2iblnd-opa ko2iblnd<br>
> # tried, as suggested in <a href="http://wiki.lustre.org/Optimizing_o2iblnd_Performance" rel="noreferrer" target="_blank">http://wiki.lustre.org/<wbr>Optimizing_o2iblnd_Performance</a><br>
> #options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 ntx=2048 map_on_demand=0 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4<br>
> options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 ntx=2048 map_on_demand=16 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4<br>
> install ko2iblnd /usr/sbin/ko2iblnd-probe<br>
<br>
</span>The lines with ko2iblnd-opa are intended for OmniPath hardware. Since you are using IB, you will want to just set your options like this:<br>
<br>
options ko2iblnd peer_credits=…, etc.<br>
<br>
Have you verified that the firewall is not running? It’s possible a firewall might be allowing ping traffic but blocking the port needed by Lustre.<br>
<br>
--<br>
Rick Mohr<br>
Senior HPC System Administrator<br>
National Institute for Computational Sciences<br>
<a href="http://www.nics.tennessee.edu" rel="noreferrer" target="_blank">http://www.nics.tennessee.edu</a><br>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">Regards,<div>- Pak</div></div></div>
</div></div></div></div>