<div dir="ltr"><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>3 things....</div><div><br></div><div>Can you send your /etc/lnet.conf file</div><div>Can you also send /etc/modprobe.d/lnet.conf</div><div>and does a systemctl restart lnet produce an error? <br></div><div><br></div><div><br></div><div>Sid</div></div></div></div></div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 30, 2021 at 6:27 AM <<a href="mailto:lustre-discuss-request@lists.lustre.org">lustre-discuss-request@lists.lustre.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send lustre-discuss mailing list submissions to<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank">lustre-discuss-request@lists.lustre.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:lustre-discuss-owner@lists.lustre.org" target="_blank">lustre-discuss-owner@lists.lustre.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of lustre-discuss digest..."<br>
Today's Topics:<br>
<br>
1. Lustre client LNET problem from a novice (Yau Hing Tuen, Bill)<br>
<br><br><br>---------- Forwarded message ----------<br>From: "Yau Hing Tuen, Bill" <<a href="mailto:billyau_hpc@hku.hk" target="_blank">billyau_hpc@hku.hk</a>><br>To: <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>Cc: <br>Bcc: <br>Date: Thu, 29 Apr 2021 15:23:51 +0800<br>Subject: [lustre-discuss] Lustre client LNET problem from a novice<br>Dear All,<br>
<br>
Need some advice on the following situation: one of my servers <br>
(Lustre client only) could no longer connect to the Lustre server. <br>
Suspecting some problem on the LNET configuration, but I am too new to <br>
Lustre and does not have more clue on how to troubleshoot it.<br>
<br>
Kernel version: Linux 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 <br>
17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux<br>
Lustre version: 2.14.0 (pulled from git)<br>
Lustre debs built with GCC 9.3.0 on the server.<br>
<br>
Modprobe not cleanly complete as static lnet configuration does not work:<br>
# modprobe -v lustre<br>
insmod /lib/modules/5.4.0-65-generic/updates/kernel/net/libcfs.ko<br>
insmod /lib/modules/5.4.0-65-generic/updates/kernel/net/lnet.ko <br>
networks="o2ib0(ibp225s0f0)"<br>
insmod /lib/modules/5.4.0-65-generic/updates/kernel/fs/obdclass.ko<br>
insmod /lib/modules/5.4.0-65-generic/updates/kernel/fs/ptlrpc.ko<br>
modprobe: ERROR: could not insert 'lustre': Network is down<br>
<br>
So resort to try dynamic lnet configuration:<br>
<br>
# lctl net up<br>
LNET configure error 100: Network is down<br>
<br>
# lnetctl net show<br>
net:<br>
- net type: lo<br>
local NI(s):<br>
- nid: 0@lo<br>
status: up<br>
<br>
# lnetctl net add --net o2ib0 --if ibp225s0f0"<br>
add:<br>
- net:<br>
errno: -100<br>
descr: "cannot add network: Network is down"<br>
<br>
Having these error messages in dmesg after the above "lnetctl net <br>
add" command<br>
[265979.237735] LNet: 3893180:0:(config.c:1564:lnet_inet_enumerate()) <br>
lnet: Ignoring interface enxeeeb676d0232: it's down<br>
[265979.237738] LNet: 3893180:0:(config.c:1564:lnet_inet_enumerate()) <br>
Skipped 9 previous similar messages<br>
[265979.238395] LNetError: <br>
3893180:0:(o2iblnd.c:2655:kiblnd_hdev_get_attr()) Invalid mr size: 0x1000000<br>
[265979.267372] LNetError: <br>
3893180:0:(o2iblnd.c:2869:kiblnd_dev_failover()) Can't get device <br>
attributes: -22<br>
[265979.298129] LNetError: 3893180:0:(o2iblnd.c:3353:kiblnd_startup()) <br>
ko2iblnd: Can't initialize device: rc = -22<br>
[265980.353643] LNetError: 105-4: Error -100 starting up LNI o2ib<br>
<br>
Initial Diagnosis:<br>
# ip link show ibp225s0f0<br>
41: ibp225s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq <br>
state UP mode DEFAULT group default qlen 256<br>
link/infiniband <br>
00:00:11:08:fe:80:00:00:00:00:00:00:0c:42:a1:03:00:79:99:1c brd <br>
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff<br>
<br>
# ip address show ibp225s0f0<br>
41: ibp225s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq <br>
state UP group default qlen 256<br>
link/infiniband <br>
00:00:11:08:fe:80:00:00:00:00:00:00:0c:42:a1:03:00:79:99:1c brd <br>
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff<br>
inet <a href="http://10.10.10.3/16" rel="noreferrer" target="_blank">10.10.10.3/16</a> brd 10.10.255.255 scope global ibp225s0f0<br>
valid_lft forever preferred_lft forever<br>
inet6 fe80::e42:a103:79:991c/64 scope link<br>
valid_lft forever preferred_lft forever<br>
<br>
# ifconfig ibp225s0f0<br>
ibp225s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044<br>
inet 10.10.10.3 netmask 255.255.0.0 broadcast 10.10.255.255<br>
inet6 fe80::e42:a103:79:991c prefixlen 64 scopeid 0x20<link><br>
unspec 00-00-11-08-FE-80-00-00-00-00-00-00-00-00-00-00 <br>
txqueuelen 256 (UNSPEC)<br>
RX packets 14363998 bytes 1440476592 (1.4 GB)<br>
RX errors 0 dropped 0 overruns 0 frame 0<br>
TX packets 88 bytes 6648 (6.6 KB)<br>
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0<br>
<br>
# lsmod | grep ib<br>
ko2iblnd 233472 0<br>
lnet 552960 3 ko2iblnd,obdclass<br>
libcfs 487424 3 lnet,ko2iblnd,obdclass<br>
ib_umad 28672 0<br>
ib_ipoib 110592 0<br>
rdma_cm 61440 2 ko2iblnd,rdma_ucm<br>
ib_cm 57344 2 rdma_cm,ib_ipoib<br>
mlx5_ib 307200 0<br>
mlx_compat 65536 1 ko2iblnd<br>
ib_uverbs 126976 2 rdma_ucm,mlx5_ib<br>
ib_core 311296 9 <br>
rdma_cm,ib_ipoib,ko2iblnd,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm<br>
mlx5_core 933888 1 mlx5_ib<br>
libcrc32c 16384 4 nf_conntrack,nf_nat,btrfs,raid456<br>
<br>
Also tested ping, ibping and rping, all passed. I have no clue <br>
what's happening as the server was able to connect to Lustre.<br>
<br>
Regards,<br>
Bill Yau<br>
University of Hong Kong<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div>