<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Richard, James,<div><br><div>I have tried "map_on_demand=16" to the "/etc/modprobe.d/ko2iblnd.conf" that was suggested. Also tried "map_on_demand=0" as suggested here: <a href="http://wiki.lustre.org/Optimizing_o2iblnd_Performance">http://wiki.lustre.org/Optimizing_o2iblnd_Performance</a><br></div><div><br></div><div><div>/etc/modprobe.d/ko2iblnd.conf<br></div></div></div></div></div></div></div></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div><div><div><div><div><div><div>alias ko2iblnd-opa ko2iblnd</div></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div># tried, as suggested in
<span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><a href="http://wiki.lustre.org/Optimizing_o2iblnd_Performance">http://wiki.lustre.org/Optimizing_o2iblnd_Performance</a></span></div></div></div></div></div></div></div><div><div><div><div><div><div><div>#options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 ntx=2048 map_on_demand=0 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4</div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 ntx=2048 map_on_demand=16 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">install ko2iblnd /usr/sbin/ko2iblnd-probe</div></div></div></div></div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>
<div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">As for the Lustre software versions that I am using:</span> <br></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(80,0,80);font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">> server: RHEL 7.5 (3.10.0-862.el7.x86_64), MLNX_OFED_LINUX-4.4-</span><br style="color:rgb(80,0,80);font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(80,0,80);font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">> 2.0.7.0, lustre 2.11.54</span><br style="color:rgb(80,0,80);font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(80,0,80);font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">> client: RHEL 7.5 (4.14.0-49.el7a.aarch64), MLNX_OFED_LINUX-4.4-</span><br style="color:rgb(80,0,80);font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(80,0,80);font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">> 2.0.7.0 , lustre 2.11.54</span><span> </span></div>
<br></div><div>As for the IB hardware, it is Mellanox ConnectX-5 Socket Direct. Only 1 IPoIB for mlx5_0 (for the ib0 interface) is configured.</div><div><br></div><div>Thanks,</div><div>- Pak</div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 4, 2018 at 9:00 AM, Richard Henwood <span dir="ltr"><<a href="mailto:Richard.Henwood@arm.com" target="_blank">Richard.Henwood@arm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Tue, 2018-09-04 at 08:06 -0700, Pak Lui wrote:<br>
> Hi all,<br>
><br>
> I am having issue with the Lustre client pinging the server using<br>
> o2ib.I want to find out if anyone has a suggestion on what could be<br>
> the problem. Thanks in advance.<br>
><br>
> lustre client pinging to server:<br>
> > [root@n0 ~]# lctl ping 192.168.13.8@o2ib<br>
> > failed to ping 192.168.13.8@o2ib: Input/output error <<<<<<<<br>
><br>
> lustre client pinging to server over IPoIB works:<br>
> > [root@n0~]# ping -c 1 192.168.13.8<br>
> > PING 192.168.13.8 (192.168.13.8) 56(84) bytes of data.<br>
> > 64 bytes from <a href="http://192.168.13.8" rel="noreferrer" target="_blank">192.168.13.8</a>: icmp_seq=1 ttl=64 time=0.376 ms<br>
><br>
> lustre client pinging to self or other client works:<br>
> > [root@n0 ~]# lctl ping 192.168.13.54@o2ib<br>
> > 12345-0@lo<br>
> > 12345-192.168.13.54@o2ib<br>
><br>
> lustre client pinging to self or otover IPoIB works:<br>
> > [root@n0~]# ping -c 1 192.168.13.54<br>
> > PING 192.168.13.54 (192.168.13.54) 56(84) bytes of data.<br>
> > 64 bytes from <a href="http://192.168.13.54" rel="noreferrer" target="_blank">192.168.13.54</a>: icmp_seq=1 ttl=64 time=0.017 ms<br>
><br>
> The lustre server and client have specified the modprobe for lnet:<br>
> > /etc/modprobe.conf<br>
> > options lnet networks=o2ib(ib0)<br>
><br>
> The client reports some error when trying to ping or mount from the<br>
> client to server:<br>
> modprobe lustre lnet<br>
> lctl ping 192.168.13.8@o2ib<br>
> mount -v -t lustre 192.168.13.8@o2ib:/zfs /mnt/zfs<br>
><br>
> > [root@n0 ~]# dmesg|tail<br>
> > [589805.093447] Lustre: Lustre: Build Version: 2.11.54<br>
> > [589805.272652] LNet: Using FastReg for registration<br>
> > [589805.275954] LNet: Added LNI 192.168.13.54@o2ib [8/256/0/180]<br>
> > [589813.278370] LNet:<br>
> > 22357:0:(o2iblnd_cb.c:3320:<wbr>kiblnd_check_conns()) Timed out tx for 1<br>
> > 92.168.13.186@o2ib: 589813 seconds<br>
> > [589835.518404] LustreError:<br>
> > 22463:0:(mgc_request.c:251:do_<wbr>config_log_add()) MGC192.168.13.8@o2i<br>
> > b: failed processing log, type 1: rc = -5<br>
> > [589843.118385] LustreError:<br>
> > 22488:0:(mgc_request.c:601:do_<wbr>requeue()) failed processing log: -5<br>
> > [589866.718389] LustreError: 15c-8: MGC192.168.13.8@o2ib: The<br>
> > configuration from log 'zfs-client' failed (-5). This may be the<br>
> > result of communication errors between this node and the MGS, a bad<br>
> > configuration, or other errors. See the syslog for more<br>
> > information.<br>
> > [589866.741623] Lustre: Unmounted zfs-client<br>
> > [589867.278516] LustreError:<br>
> > 22463:0:(obd_mount.c:1599:<wbr>lustre_fill_super()) Unable to mount (-<br>
> > 5)<br>
><br>
> server reports some error during mounting:<br>
> > [root@license ~]# Sep 4 07:26:56 license kernel: LNet:<br>
> > 25518:0:(o2iblnd_cb.c:2475:<wbr>kiblnd_passive_connect()) Can't accept<br>
> > conn from 192.168.13.54@o2ib (version 12): max_frags 16<br>
> > incompatible without FMR pool (256 wanted)<br>
><br>
> The lustre server setup:<br>
> > [root@license ~]# lfs df -h<br>
> > UUID bytes Used Available Use%<br>
> > Mounted on<br>
> > zfs-MDT0000_UUID 863.4M 7.5M 853.9M 1%<br>
> > /mnt/zfs[MDT:0]<br>
> > zfs-OST0000_UUID 1.7T 10.0G 1.7T 1%<br>
> > /mnt/zfs[OST:0]<br>
> ><br>
> > filesystem_summary: 1.7T 10.0G 1.7T 1%<br>
> > /mnt/zfs<br>
><br>
> server: RHEL 7.5 (3.10.0-862.el7.x86_64), MLNX_OFED_LINUX-4.4-<br>
> 2.0.7.0, lustre 2.11.54<br>
> client: RHEL 7.5 (4.14.0-49.el7a.aarch64), MLNX_OFED_LINUX-4.4-<br>
> 2.0.7.0 , lustre 2.11.54<br>
><br>
<br>
<br>
</div></div>It might be helpful to state the Lustre software versions that you have<br>
used.<br>
<br>
Also, given this is an Arm client with (with presumably 64K pg size),<br>
connecting to a x86 server (with presumably 4K pg size), have you added<br>
the map_on_demand=16 incantation to the server? I don't have direct<br>
experience of this, but heard it was needed in some Arm configurations<br>
(depending on server/client version):<br>
<br>
<a href="https://jira.whamcloud.com/browse/LU-10775" rel="noreferrer" target="_blank">https://jira.whamcloud.com/<wbr>browse/LU-10775</a><br>
<br>
May be James can advise?<br>
<br>
best regards,<br>
Richard<br>
<br>
--<br>
<a href="mailto:Richard.Henwood@arm.com">Richard.Henwood@arm.com</a><br>
Server Software Eco-System<br>
Tel: +1 512 410 9612<br>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Regards,<div>- Pak</div></div></div>
</div></div>