<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi all,</div><div><br></div><div>I am having issue with the Lustre client pinging the server using o2ib.I want to find out if anyone has a suggestion on what could be the problem. Thanks in advance.</div><div><br></div><div>
<div dir="ltr" style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">lustre client pinging to server:</div></div></div></div></div></div></div><blockquote style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div><div>[root@n0 ~]# lctl ping 192.168.13.8@o2ib</div></div></div></div></div></div><div><div><div><div><div><div>failed to ping 192.168.13.8@o2ib: Input/output error <<<<<<<</div></div></div></div></div></div></blockquote><div dir="ltr" style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">lustre client pinging to server over IPoIB works:<br></div></div></div></div></div></div><blockquote style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div><div><div>[root@n0~]# ping -c 1 192.168.13.8</div></div></div></div></div></div></div><div><div><div><div><div><div><div>PING 192.168.13.8 (192.168.13.8) 56(84) bytes of data.</div></div></div></div></div></div></div><div><div><div><div><div><div><div>64 bytes from <a href="http://192.168.13.8">192.168.13.8</a>: icmp_seq=1 ttl=64 time=0.376 ms</div></div></div></div></div></div></div></blockquote>
<br></div><div>lustre client pinging to self or other client works:</div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div><div>[root@n0 ~]# lctl ping 192.168.13.54@o2ib</div></div></div></div></div></div><div><div><div><div><div><div>12345-0@lo</div></div></div></div></div></div><div><div><div><div><div><div>12345-192.168.13.54@o2ib</div></div></div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">
<div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">lustre client pinging to self or otover IPoIB works:</div></div></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">[root@n0~]# ping -c 1 192.168.13.54</div></div></div></div></div></div></div><div><div><div><div><div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">PING 192.168.13.54 (192.168.13.54) 56(84) bytes of data.</div></div></div></div></div></div></div><div><div><div><div><div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">64 bytes from <a href="http://192.168.13.54">192.168.13.54</a>: icmp_seq=1 ttl=64 time=0.017 ms</div></div></div></div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div></div></div></div></div></div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><div>The lustre server and client have specified the modprobe for lnet:</div></div></div></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div><div><div><div>/etc/modprobe.conf</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div>options lnet networks=o2ib(ib0)</div></div></div></div></div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>The client reports some error when trying to ping or mount from the client to server:</div><div><div>modprobe lustre lnet</div><div>lctl ping 192.168.13.8@o2ib<br></div><div>mount -v -t lustre 192.168.13.8@o2ib:/zfs /mnt/zfs<br></div></div><div><br></div></div></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div><div><div><div>[root@n0 ~]# dmesg|tail</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div>[589805.093447] Lustre: Lustre: Build Version: 2.11.54</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div>[589805.272652] LNet: Using FastReg for registration</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div>[589805.275954] LNet: Added LNI 192.168.13.54@o2ib [8/256/0/180]</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div>[589813.278370] LNet: 22357:0:(o2iblnd_cb.c:3320:kiblnd_check_conns()) Timed out tx for 192.168.13.186@o2ib: 589813 seconds</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div>[589835.518404] LustreError: 22463:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.13.8@o2ib: failed processing log, type 1: rc = -5</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div>[589843.118385] LustreError: 22488:0:(mgc_request.c:601:do_requeue()) failed processing log: -5</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div>[589866.718389] LustreError: 15c-8: MGC192.168.13.8@o2ib: The configuration from log 'zfs-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div>[589866.741623] Lustre: Unmounted zfs-client</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div>[589867.278516] LustreError: 22463:0:(obd_mount.c:1599:lustre_fill_super()) Unable to mount (-5)</div></div></div></div></div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>
<div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">server reports some error during mounting:</div></div></div></div></div></div></div></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div>[root@license ~]# Sep 4 07:26:56 license kernel: LNet: 25518:0:(o2iblnd_cb.c:2475:kiblnd_passive_connect()) Can't accept conn from 192.168.13.54@o2ib (version 12): max_frags 16 incompatible without FMR pool (256 wanted)</div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>The lustre server setup:</div></div></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><div><div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">[root@license ~]# lfs df -h</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">UUID bytes Used Available Use% Mounted on</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">zfs-MDT0000_UUID 863.4M 7.5M 853.9M 1% /mnt/zfs[MDT:0]</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">zfs-OST0000_UUID 1.7T 10.0G 1.7T 1% /mnt/zfs[OST:0]</div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div></div></div></div></div></div></div></div><div><div><div><div><div><div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">filesystem_summary: 1.7T 10.0G 1.7T 1% /mnt/zfs</div></div></div></div></div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>
<br></div><div>
<div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">server: RHEL 7.5 (3.10.0-862.el7.x86_64), MLNX_OFED_LINUX-4.4-2.0.7.0, lustre 2.11.54<br></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">client: RHEL 7.5 (4.14.0-49.el7a.aarch64),<span> </span>MLNX_OFED_LINUX-4.4-2.0.7.0<span> </span><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">, lustre 2.11.54</span></div><div><br></div><div>Regards,<br></div></div><div>- Pak<br class="gmail-Apple-interchange-newline">
<br></div>
</div></div></div></div></div></div></div></div></div></div></div></div></div></div>