<div dir="ltr"> Hi everybody,<br><br>Here at the university of Groningen we are now experiencing a strange Lustre error. If a client reboots, it fails to mount the Lustre storage. The client is not able to reach the MSG service. The storage and nodes are communicating over IB and unitil now without any problems. It looks like an issue inside LNET. Clients cannot LNET ping/connect the metadata and or storage. But the clients are able to LNET ping each other. Clients which not have been rebooted, are working fine and have their mounts on our Lustre filesystem.<br><br>Lustre client log:<br><br>Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32-573.el6.x86_64<br>LNet: Added LNI 172.23.54.51@o2ib [8/256/0/180]<br><br>LustreError: 15c-8: MGC172.23.55.211@o2ib: The configuration from log 'pgdata01-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br>LustreError: 3812:0:(llite_lib.c:1046:ll_fill_super()) Unable to process log: -5<br>Lustre: Unmounted pgdata01-client<br>LustreError: 3812:0:(obd_mount.c:1325:lustre_fill_super()) Unable to mount (-5)<br>LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) 172.23.55.212@o2ib rejected: consumer defined fatal error<br>LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) Skipped 1 previous similar message<br>Lustre: 3765:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1492789626/real 1492789626] req@ffff88105af2cc00 x1565303228072004/t0(0) o250->MGC172.23.55.211@o2ib@172.23.55.212@o2ib:26/25 lens 400/544 e 0 to 1 dl 1492789631 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br>Lustre: 3765:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 1 previous similar message<br>LustreError: 3826:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff882041ffc000 x1565303228071996/t0(0) o101->MGC172.23.55.211@o2ib@172.23.55.211@o2ib:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<br>LustreError: 3826:0:(client.c:1083:ptlrpc_import_delay_req()) Skipped 2 previous similar messages<br>LustreError: 15c-8: MGC172.23.55.211@o2ib: The configuration from log 'pghome01-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br>LustreError: 3826:0:(llite_lib.c:1046:ll_fill_super()) Unable to process log: -5<br><br>LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) 172.23.55.212@o2ib rejected: consumer defined fatal error<br>LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) Skipped 1 previous similar message<br>LNet: 3755:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from 172.23.55.211@o2ib failed: 5<br>LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) 172.23.55.211@o2ib rejected: consumer defined fatal error<br>LNetError: 2882:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) Skipped 1 previous similar message<br>LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed()) Deleting messages for 172.23.55.211@o2ib: connection failed<br>LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed()) Deleting messages for 172.23.55.212@o2ib: connection failed<br>LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from 172.23.55.212@o2ib failed: 5<br>LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Skipped 17 previous similar messages<br>LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed()) Deleting messages for 172.23.55.211@o2ib: connection failed<br>LNet: 3754:0:(o2iblnd_cb.c:475:kiblnd_rx_complete()) Rx from 172.23.55.212@o2ib failed: 5<br>LNet: 2882:0:(o2iblnd_cb.c:2072:kiblnd_peer_connect_failed()) Deleting messages for 172.23.55.212@o2ib: connection failed<br><br>LNET ping of a metadata-node:<br><br>[root@pg-gpu01 ~]# lctl ping 172.23.55.211@o2ib<br>failed to ping 172.23.55.211@o2ib: Input/output error<br><br>LNET ping of the number 2 metadata-node:<br><br>[root@pg-gpu01 ~]# lctl ping 172.23.55.212@o2ib<br>failed to ping 172.23.55.212@o2ib: Input/output error<br><br>LNET ping of a random compute-node:<br><br>[root@pg-gpu01 ~]# lctl ping 172.23.52.5@o2ib<br>12345-0@lo<br>12345-172.23.52.5@o2ib<br><br>LNET to OST01:<br><br>[root@pg-gpu01 ~]# lctl ping 172.23.55.201@o2ib<br>failed to ping 172.23.55.201@o2ib: Input/output error<br><br>LNET to OST02:<br><br>[root@pg-gpu01 ~]# lctl ping 172.23.55.202@o2ib<br>failed to ping 172.23.55.202@o2ib: Input/output error<br><br>'normal' pings (on ip level) works fine: <br><br>[root@pg-gpu01 ~]# ping 172.23.55.201<br>PING 172.23.55.201 (172.23.55.201) 56(84) bytes of data.<br>64 bytes from <a href="http://172.23.55.201">172.23.55.201</a>: icmp_seq=1 ttl=64 time=0.741 ms<br><br>[root@pg-gpu01 ~]# ping 172.23.55.202<br>PING 172.23.55.202 (172.23.55.202) 56(84) bytes of data.<br>64 bytes from <a href="http://172.23.55.202">172.23.55.202</a>: icmp_seq=1 ttl=64 time=0.704 ms<br><br>lctl on a rebooted node:<br><br>[root@pg-gpu01 ~]# lctl dl<br><br>lctl on a not rebooted node:<br><br>[root@pg-node005 ~]# lctl dl<br> 0 UP mgc MGC172.23.55.211@o2ib 94bd1c8a-512f-b920-9a4e-a6aced3d386d 5<br> 1 UP lov pgtemp01-clilov-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 4<br> 2 UP lmv pgtemp01-clilmv-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 4<br> 3 UP mdc pgtemp01-MDT0000-mdc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 4 UP osc pgtemp01-OST0001-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 5 UP osc pgtemp01-OST0003-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 6 UP osc pgtemp01-OST0005-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 7 UP osc pgtemp01-OST0007-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 8 UP osc pgtemp01-OST0009-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 9 UP osc pgtemp01-OST000b-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 10 UP osc pgtemp01-OST000d-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 11 UP osc pgtemp01-OST000f-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 12 UP osc pgtemp01-OST0011-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 13 UP osc pgtemp01-OST0002-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 14 UP osc pgtemp01-OST0004-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 15 UP osc pgtemp01-OST0006-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 16 UP osc pgtemp01-OST0008-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 17 UP osc pgtemp01-OST000a-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 18 UP osc pgtemp01-OST000c-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 19 UP osc pgtemp01-OST000e-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 20 UP osc pgtemp01-OST0010-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 21 UP osc pgtemp01-OST0012-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 22 UP osc pgtemp01-OST0013-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 23 UP osc pgtemp01-OST0015-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 24 UP osc pgtemp01-OST0017-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 25 UP osc pgtemp01-OST0014-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 26 UP osc pgtemp01-OST0016-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 27 UP osc pgtemp01-OST0018-osc-ffff88206906d400 281c441f-8aa3-ab56-8812-e459d308f47c 5<br> 28 UP lov pgdata01-clilov-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 4<br> 29 UP lmv pgdata01-clilmv-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 4<br> 30 UP mdc pgdata01-MDT0000-mdc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 31 UP osc pgdata01-OST0001-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 32 UP osc pgdata01-OST0003-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 33 UP osc pgdata01-OST0005-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 34 UP osc pgdata01-OST0007-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 35 UP osc pgdata01-OST0009-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 36 UP osc pgdata01-OST000b-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 37 UP osc pgdata01-OST000d-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 38 UP osc pgdata01-OST000f-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 39 UP osc pgdata01-OST0002-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 40 UP osc pgdata01-OST0004-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 41 UP osc pgdata01-OST0006-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 42 UP osc pgdata01-OST0008-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 43 UP osc pgdata01-OST000a-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 44 UP osc pgdata01-OST000c-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 45 UP osc pgdata01-OST000e-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 46 UP osc pgdata01-OST0010-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 47 UP osc pgdata01-OST0013-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 48 UP osc pgdata01-OST0015-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 49 UP osc pgdata01-OST0017-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 50 UP osc pgdata01-OST0014-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 51 UP osc pgdata01-OST0016-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 52 UP osc pgdata01-OST0018-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 53 UP osc pgdata01-OST0019-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 54 UP osc pgdata01-OST001a-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 55 UP osc pgdata01-OST001b-osc-ffff88204bab6400 996b1742-82eb-281c-c322-e244672d5225 5<br> 56 UP lov pghome01-clilov-ffff88204bb50000 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 4<br> 57 UP lmv pghome01-clilmv-ffff88204bb50000 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 4<br> 58 UP mdc pghome01-MDT0000-mdc-ffff88204bb50000 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5<br> 59 UP osc pghome01-OST0011-osc-ffff88204bb50000 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5<br> 60 UP osc pghome01-OST0012-osc-ffff88204bb50000 9ae8f2a9-1cdf-901f-160c-66f70e4c10d1 5<br><br>Please help, any clues/advice/hints/tips are appricated<br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><pre cols="72"><span style="font-family:arial,helvetica,sans-serif"><font size="2">Vriendelijke groet, <br><br>Ger Strikwerda
</font></span><span style="font-family:arial,helvetica,sans-serif"><font size="2">Chef Special
Rijksuniversiteit Groningen
Centrum voor Informatie Technologie
Unit Pragmatisch Systeembeheer
Smitsborg
Nettelbosje 1
9747 AJ Groningen
Tel. 050 363 9276
<br>"God is hard, God is fair<br> some men he gave brains, others he gave hair"<br></font></span></pre></div></div></div></div></div></div></div></div></div></div></div></div>
</div>