<span style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">Problem is similer to <a href="http://lists.lustre.org/pipermail/lustre-discuss/2008-May/007498.html">http://lists.lustre.org/pipermail/lustre-discuss/2008-May/007498.html</a></span><br style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">
<span style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">But by looking at the thread could not really get the solution for the problem.</span><br style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">
<br style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);"><span style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">I have two RHEL5 Linux servers installed with following packages -</span><br style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">
<br><span style="font-family: courier new,monospace;">kernel-lustre-smp-2.6.18-53.1.14.el5_lustre.1.6.5.1</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">kernel-ib-1.3-2.6.18_53.1.14.el5_lustre.1.6.5.1smp</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">lustre-ldiskfs-3.0.4-2.6.18_53.1.14.el5_lustre.1.6.5.1smp</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">lustre-1.6.5.1-2.6.18_53.1.14.el5_lustre.1.6.5.1smp</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">lustre-modules-1.6.5.1-2.6.18_53.1.14.el5_lustre.1.6.5.1smp</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">e2fsprogs-1.40.7.sun3-0redhat<br>
</span><br><br><span style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">machine 1: with ib0 IP address : 172.24.198.111</span><br style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);"><span style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">machine 2: with ib0 IP address : 172.24.198.112<br>
<br>/etc/modprobe.conf contains</span><br style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);"><span style="font-family: courier new,monospace;">options lnet networks=o2ib</span><br><br style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">
<span style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">TCP networking worked fine and now I am trying with Infiniband network finding it difficult in communicating with IB nodes mounting effort throghs me the following error</span><br>
<br><span style="font-family: courier new,monospace;">[root@p186 ~]# mount -t lustre -o loop /tmp/lustre-ost1 /mnt/ost1</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">mount.lustre: mount /dev/loop0 at /mnt/ost1 failed: Input/output error</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Is the MGS running?</span><br><br><span style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">/var/log/messages :</span><br><span style="font-family: courier new,monospace;">Jan 15 16:55:25 p186 kernel: kjournald starting. Commit interval 5 seconds</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 15 16:55:25 p186 kernel: LDISKFS FS on loop0, internal journal</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Jan 15 16:55:25 p186 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 15 16:55:25 p186 kernel: kjournald starting. Commit interval 5 seconds</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Jan 15 16:55:25 p186 kernel: LDISKFS FS on loop0, internal journal</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 15 16:55:25 p186 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Jan 15 16:55:25 p186 kernel: LDISKFS-fs: file extents enabled</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 15 16:55:25 p186 kernel: LDISKFS-fs: mballoc enabled</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Jan 15 16:55:30 p186 kernel: Lustre: Request x7 sent from MGC172.24.198.111@o2ib to NID 172.24.198.111@o2ib 5s ago has timed out (limit 5s).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 15 16:55:30 p186 kernel: LustreError: 7193:0:(obd_mount.c:1062:server_start_targets()) Required registration failed for lustre-OSTffff: -5</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 15 16:55:30 p186 kernel: LustreError: 15f-b: Communication error with the MGS. Is the MGS running?</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Jan 15 16:55:30 p186 kernel: LustreError: 7193:0:(obd_mount.c:1597:server_fill_super()) Unable to start targets: -5</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 15 16:55:30 p186 kernel: LustreError: 7193:0:(obd_mount.c:1382:server_put_super()) no obd lustre-OSTffff</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Jan 15 16:55:30 p186 kernel: LustreError: 7193:0:(obd_mount.c:119:server_deregister_mount()) lustre-OSTffff not registered</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 15 16:55:30 p186 kernel: LDISKFS-fs: mballoc: 0 blocks 0 reqs (0 success)</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Jan 15 16:55:30 p186 kernel: LDISKFS-fs: mballoc: 0 extents scanned, 0 goal hits, 0 2^N hits, 0 breaks, 0 lost</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 15 16:55:30 p186 kernel: LDISKFS-fs: mballoc: 0 generated and it took 0</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Jan 15 16:55:30 p186 kernel: LDISKFS-fs: mballoc: 0 preallocated, 0 discarded</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 15 16:55:30 p186 kernel: Lustre: server umount lustre-OSTffff complete</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Jan 15 16:55:30 p186 kernel: LustreError: 7193:0:(obd_mount.c:1951:lustre_fill_super()) Unable to mount (-5)</span><br>
<br><span style="color: rgb(51, 51, 153); font-family: verdana,sans-serif;">All pinging efforts also failed to the IB NIDS local/remote</span><br style="color: rgb(51, 51, 153); font-family: verdana,sans-serif;"><span style="color: rgb(51, 51, 153); font-family: verdana,sans-serif;">can ping the ip address :</span><br>
<span style="font-family: courier new,monospace;">[root@p186 ~]# ping 172.24.198.112</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">PING 172.24.198.112 (172.24.198.112) 56(84) bytes of data.</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">64 bytes from <a href="http://172.24.198.112">172.24.198.112</a>: icmp_seq=1 ttl=64 time=0.052 ms</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">64 bytes from <a href="http://172.24.198.112">172.24.198.112</a>: icmp_seq=2 ttl=64 time=0.024 ms</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">--- 172.24.198.112 ping statistics ---</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">2 packets transmitted, 2 received, 0% packet loss, time 1000ms</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">rtt min/avg/max/mdev = 0.024/0.038/0.052/0.014 ms</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[root@p186 ~]# ping 172.24.198.111</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">PING 172.24.198.111 (172.24.198.111) 56(84) bytes of data.</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">64 bytes from <a href="http://172.24.198.111">172.24.198.111</a>: icmp_seq=1 ttl=64 time=2.16 ms</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">64 bytes from <a href="http://172.24.198.111">172.24.198.111</a>: icmp_seq=2 ttl=64 time=0.296 ms</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">--- 172.24.198.111 ping statistics ---</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">2 packets transmitted, 2 received, 0% packet loss, time 1000ms</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">rtt min/avg/max/mdev = 0.296/1.231/2.166/0.935 ms</span><br><br><span style="color: rgb(51, 51, 153); font-family: verdana,sans-serif;">but cant ping the NIDS :</span><br>
<span style="font-family: courier new,monospace;">[root@p186 ~]# lctl ping 172.24.198.112@o2ib</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">failed to ping 172.24.198.112@o2ib: Input/output error</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">[root@p186 ~]# lctl ping 172.24.198.111@o2ib</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">failed to ping 172.24.198.111@o2ib: Input/output error</span><br>
<br><span style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">Any idea why lnet cant ping NIDS ?<br><br></span><span style="font-family: verdana,sans-serif; color: rgb(51, 51, 153);">some more configurations:</span><br>
<span style="font-family: courier new,monospace;">[root@p186 ~]# ibstat</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">CA 'mthca0'</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> CA type: MT23108</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> Number of ports: 2</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> Firmware version: 3.5.0</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> Hardware version: a1</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> Node GUID: 0x0002c9020021550c</span><br><br>Machines are connected via IB switch.<br><br>Looking forward for help.<br><br clear="all">~subbu