<div>Problem solved.</div>
<div>Earlier we had kept both my ethernet and infiniband network interfaces in the same subnet.</div>
<div> </div>
<div>After changing the ipoib network interfaces subnet to different subnet it worked. I think probably it makes sense to add a note about this subnet config in Lustre manual as well.</div>
<div> </div>
<div>Thanks again.<br></div>
<div>~subbu<br><br></div>
<div class="gmail_quote">On Tue, Jan 27, 2009 at 3:49 PM, subbu kl <span dir="ltr"><<a href="mailto:subbukl@gmail.com">subbukl@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<p>Liang,</p>
<p>please find the info you have asked below.<br></p>
<p>There are two nodes MDS and OSS1 connected throgh a silverstorme Infiniband switch and MDS running IB subnet manager running.</p>
<p></p>
<div><font face="courier new,monospace">[root@MDS ~]# cat /etc/modprobe.conf<br>alias eth0 bnx2<br>alias eth1 bnx2<br>alias scsi_hostadapter megaraid_sas<br>alias scsi_hostadapter1 ata_piix<br>alias scsi_hostadapter2 usb-storage<br>
alias ib0 ib_ipoib<br>alias ib1 ib_ipoib<br>alias net-pf-27 ib_sdp<br>options loop max_loop=64
<div class="Ih2E3d"><br>options lnet networks=o2ib(ib0)<br></div>options ib_madeye data=1<br><br>[root@MDS ~]# ifconfig<br>eth0 Link encap:Ethernet HWaddr 00:18:8B:40:63:C3<br> inet addr:172.24.198.128 Bcast:172.24.255.255 Mask:255.255.0.0<br>
inet6 addr: fe80::218:8bff:fe40:63c3/64 Scope:Link<br> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br> RX packets:4203 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:1069 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:1000<br> RX bytes:415345 (405.6 KiB) TX bytes:109548 (106.9 KiB)<br> Interrupt:169 Memory:f8000000-f8012100
<div class="Ih2E3d"><br><br>ib0 Link encap:InfiniBand HWaddr 80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00<br></div> inet addr:172.24.198.140 Bcast:172.24.255.255 Mask:255.255.0.0<br> inet6 addr: fe80::202:c902:22:cd49/64 Scope:Link<br>
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1<br> RX packets:8 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:11 errors:0 dropped:6 overruns:0 carrier:0<br> collisions:0 txqueuelen:256<br> RX bytes:4163 (4.0 KiB) TX bytes:4205 (4.1 KiB)<br>
<br>lo Link encap:Local Loopback<br> inet addr:127.0.0.1 Mask:255.0.0.0<br> inet6 addr: ::1/128 Scope:Host<br> UP LOOPBACK RUNNING MTU:16436 Metric:1<br> RX packets:1614 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:1614 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:0<br> RX bytes:5322452 (5.0 MiB) TX bytes:5322452 (5.0 MiB)<br><br>[root@MDS ~]# lctl list_nids
<div class="Ih2E3d"><br>172.24.198.140@o2ib<br></div>[root@MDS ~]# route -e<br>Kernel IP routing table<br>Destination Gateway Genmask Flags MSS Window irtt Iface<br>172.24.0.0 * 255.255.0.0 U 0 0 0 eth0<br>172.24.0.0 * 255.255.0.0 U 0 0 0 ib0<br>
169.254.0.0 * 255.255.0.0 U 0 0 0 ib0<br>default 172.24.198.250 0.0.0.0 UG 0 0 0 eth0<br><br>[root@MDS ~]# echo +neterror > /proc/sys/lnet/printk<br><br>[root@MDS ~]# echo +neterror > /proc/sys/lnet/printk<br>[root@MDS ~]# lctl list_nids
<div class="Ih2E3d"><br>172.24.198.140@o2ib<br></div>[root@MDS ~]# lctl ping 172.24.198.140@o2ib<br>12345-0@lo
<div class="Ih2E3d"><br>12345-172.24.198.140@o2ib<br></div>[root@MDS ~]# lctl ping 172.24.198.141@o2ib<br>failed to ping 172.24.198.141@o2ib: Input/output error<br><br><br>/var/log/messages :<br><br>Jan 27 15:41:41 MDS kernel: Lustre: 5649:0:(o2iblnd_cb.c:2704:kiblnd_cm_callback()) 172.24.198.141@o2ib: ROUTE ERROR -22<br>
Jan 27 15:41:41 MDS kernel: Lustre: 5649:0:(o2iblnd_cb.c:2118:kiblnd_peer_connect_failed()) Deleting messages for 172.24.198.141@o2ib: connection failed<br><br><br><br><br><br><br><br><br><br><br><br><br>[root@OSS1 ~]# cat /etc/modprobe.conf<br>
alias eth0 e1000<br>alias eth1 e1000<br>alias scsi_hostadapter megaraid_mbox<br>alias scsi_hostadapter1 qla2xxx<br>options loop max_loop=64
<div class="Ih2E3d"><br>alias ib0 ib_ipoib<br>options lnet networks=o2ib(ib0)<br></div>options ib_ipoib debug_level=1<br>options ib_ipoib mcast_debug_level=1<br>### BEGIN MPP Driver Comments ###<br>remove mppUpper if [ `ls -a /proc/mpp | wc -l` -gt 2 ]; then echo -e "Please Unload Physical HBA Driver prior to unloading mppUpper."; else /sbin/modprobe -r --ignore-remove mppUpper; fi<br>
# Additional config info can be found in /opt/mpp/modprobe.conf.mppappend.<br># The Above config info is needed if you want to make mkinitrd manually.<br># Please read the Readme file that came with MPP driver for building RamDisk manually.<br>
# Edit the '/etc/modprobe.conf' file and run 'mppUpdate' to create Ramdisk dynamically.<br>### END MPP Driver Comments ###<br>#alias ib1 ib_ipoib<br>alias net-pf-27 ib_sdp<br>options ib_madeye data=1<br><br>
[root@OSS1 ~]# ifconfig<br>eth0 Link encap:Ethernet HWaddr 00:13:72:5D:3B:65<br> inet addr:172.24.198.186 Bcast:172.24.255.255 Mask:255.255.0.0<br> inet6 addr: fe80::213:72ff:fe5d:3b65/64 Scope:Link<br> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br>
RX packets:7831 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:1007 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:100<br> RX bytes:809440 (790.4 KiB) TX bytes:99439 (97.1 KiB)<br> Base address:0xdcc0 Memory:df7e0000-df800000
<div class="Ih2E3d"><br><br>ib0 Link encap:InfiniBand HWaddr 80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00<br></div> inet addr:172.24.198.141 Bcast:172.24.255.255 Mask:255.255.0.0<br> inet6 addr: fe80::202:c902:21:550d/64 Scope:Link<br>
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1<br> RX packets:10 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:11 errors:0 dropped:7 overruns:0 carrier:0<br> collisions:0 txqueuelen:256<br> RX bytes:4097 (4.0 KiB) TX bytes:5202 (5.0 KiB)<br>
<br>lo Link encap:Local Loopback<br> inet addr:127.0.0.1 Mask:255.0.0.0<br> inet6 addr: ::1/128 Scope:Host<br> UP LOOPBACK RUNNING MTU:16436 Metric:1<br> RX packets:94 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:94 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:0<br> RX bytes:8962 (8.7 KiB) TX bytes:8962 (8.7 KiB)<br><br>virbr0 Link encap:Ethernet HWaddr 00:00:00:00:00:00<br> inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0<br> inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link<br>
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
<div class="Ih2E3d"><br> RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br></div> TX packets:49 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:0<br> RX bytes:0 (0.0 b) TX bytes:9166 (8.9 KiB)<br>
<br>[root@OSS1 ~]# lctl list_nids
<div class="Ih2E3d"><br>172.24.198.141@o2ib<br></div>[root@OSS1 ~]# route -e<br>Kernel IP routing table<br>Destination Gateway Genmask Flags MSS Window irtt Iface<br>192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0<br>172.24.0.0 * 255.255.0.0 U 0 0 0 eth0<br>
172.24.0.0 * 255.255.0.0 U 0 0 0 ib0<br>169.254.0.0 * 255.255.0.0 U 0 0 0 ib0<br>default 172.24.198.250 0.0.0.0 UG 0 0 0 eth0<br><br>[root@OSS1 ~]# echo +neterror > /proc/sys/lnet/printk<br>[root@OSS1 ~]# echo +neterror > /proc/sys/lnet/printk<br>
[root@OSS1 ~]# lctl list_nids
<div class="Ih2E3d"><br>172.24.198.141@o2ib<br></div>[root@OSS1 ~]# lctl ping 172.24.198.141@o2ib<br>12345-0@lo<br>12345-172.24.198.141@o2ib<br>[root@OSS1 ~]# lctl ping 172.24.198.140@o2ib<br>failed to ping 172.24.198.140@o2ib: Input/output error<br>
<br><br>/var/log/messages :<br><br>Jan 27 15:34:17 OSS1 kernel: Lustre: 2776:0:(o2iblnd_cb.c:2704:kiblnd_cm_callback()) 172.24.198.140@o2ib: ROUTE ERROR -22<br>Jan 27 15:34:17 OSS1 kernel: Lustre: 2776:0:(o2iblnd_cb.c:2118:kiblnd_peer_connect_failed()) Deleting messages for 172.24.198.140@o2ib: connection failed<br>
</font></div>
<div><br> </div>
<p>~subbu</p>
<div>
<div></div>
<div class="Wj3C7c"><br>
<div class="gmail_quote">On Sat, Jan 24, 2009 at 8:06 AM, Liang Zhen <span dir="ltr"><<a href="mailto:Zhen.Liang@sun.com" target="_blank">Zhen.Liang@sun.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">Subbu,<br>I think we can't see anything from tcpdump even run ping sucessfully, because we only need ipoib for connecting (not for transaction).<br>
I think we need these information for diagnosing:<br>1. modprobe.conf of two nodes with IB<br>2. ifconfig on these two nodes<br>3. routing table on these two nodes<br>4. try lctl ping itself on both nodes and see if any error (with +neterror)<br>
<br>Regards<br>Liang<br><br>subbu kl:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div>
<div>problem remained same, when I run lctl ping with tcpdump 4.0.0 I dont see any activity on ib0 !<br><br>another exhaustive Lustre debug log I took with lctl ping do you see any problem with it ?<br><br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(module.c:160:libcfs_psdev_open()) Process entered<br>
Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(module.c:164:libcfs_psdev_open()) kmalloced 'ldu': 8 at f5bc6620 (tot 7258558).<br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(module.c:171:libcfs_psdev_open()) Process leaving (rc=0 : 0 : 0)<br>
Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(module.c:228:libcfs_ioctl()) Process entered<br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(linux-module.c:49:libcfs_ioctl_getdata()) Process entered<br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(linux-module.c:90:libcfs_ioctl_getdata()) Process leaving (rc=0 : 0 : 0)<br>
Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(api-ni.c:1223:LNetNIInit()) refs 1<br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(api-ni.c:1614:lnet_ping()) kmalloced 'info': 144 at f0b95880 (tot 7258702).<br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-lnet.h:251:lnet_eq_alloc()) kmalloced 'eq': 48 at efda1a00 (tot 7258750).<br>
Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-eq.c:72:LNetEQAlloc()) kmalloced 'eq->eq_events': 240 at f0b95c80 (tot 7258990).<br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-lnet.h:279:lnet_md_alloc()) kmalloced 'md': 84 at ed16acc0 (tot 7259074).<br>
Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-lnet.h:327:lnet_msg_alloc()) kmalloced 'msg': 268 at f205a400 (tot 7259342).<br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-move.c:2395:LNetGet()) LNetGet -> 12345-172.24.198.140@o2ib<br>
Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(o2iblnd_cb.c:1531:kiblnd_send()) sending 0 bytes in 0 frags to 12345-172.24.198.140@o2ib<br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(o2iblnd.c:312:kiblnd_create_peer()) kmalloced 'peer': 56 at efda18c0 (tot 7259398).<br>
Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(o2iblnd_cb.c:1501:kiblnd_launch_tx()) peer[efda18c0] -> 172.24.198.140@o2ib (1)++<br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(o2iblnd_cb.c:1380:kiblnd_connect_peer()) peer[efda18c0] -> 172.24.198.140@o2ib (2)++<br>
Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(o2iblnd_cb.c:1507:kiblnd_launch_tx()) peer[efda18c0] -> 172.24.198.140@o2ib (3)--<br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-eq.c:209:LNetEQPoll()) Process entered<br>
Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-eq.c:146:lib_get_event()) Process entered<br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-eq.c:149:lib_get_event()) event: f0b95cf8, sequence: 1, eq->size: 2<br>Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-eq.c:152:lib_get_event()) Process leaving (rc=0 : 0 : 0)<br>
Jan 23 17:23:39 p186 kernel: Lustre: 2782:0:(o2iblnd_cb.c:2682:kiblnd_cm_callback()) 172.24.198.140@o2ib Addr resolved: 0<br>Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:146:lib_get_event()) Process entered<br>Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:149:lib_get_event()) event: f0b95cf8, sequence: 1, eq->size: 2<br>
Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:152:lib_get_event()) Process leaving (rc=0 : 0 : 0)<br>Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:239:LNetEQPoll()) Process leaving (rc=0 : 0 : 0)<br>Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(api-ni.c:1665:lnet_ping()) poll 0(-1 -1)<br>
Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-md.c:69:lnet_md_unlink()) Queueing unlink of md ed16acc0<br>Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:209:LNetEQPoll()) Process entered<br>Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:146:lib_get_event()) Process entered<br>
Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:149:lib_get_event()) event: f0b95cf8, sequence: 1, eq->size: 2<br>Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:152:lib_get_event()) Process leaving (rc=0 : 0 : 0)<br>
Jan 23 17:23:56 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=4294962944 : -4352 : ffffef00)<br>Jan 23 17:23:56 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=4294966784 : -512 : fffffe00)<br>
Jan 23 17:23:56 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=2817 : 2817 : b01)<br>Jan 23 17:23:56 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=2047 : 2047 : 7ff)<br>
Jan 23 17:23:56 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=4294740832 : -226464 : fffc8b60)<br>Jan 23 17:23:56 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=4286216485 : -8750811 : ff7a7925)<br>
Jan 23 17:23:56 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=5821091 : 5821091 : 58d2a3)<br>Jan 23 17:23:56 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=3356952 : 3356952 : 333918)<br>
Jan 23 17:23:56 p186 kernel: Lustre: 8276:0:(pinger.c:193:ptlrpc_pinger_main()) next ping in 25000 (8510847)<br>Jan 23 17:24:21 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=4294962944 : -4352 : ffffef00)<br>
Jan 23 17:24:21 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=4294966784 : -512 : fffffe00)<br>Jan 23 17:24:21 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=2817 : 2817 : b01)<br>
Jan 23 17:24:21 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=2047 : 2047 : 7ff)<br>Jan 23 17:24:21 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=4294740832 : -226464 : fffc8b60)<br>
Jan 23 17:24:21 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=4286216485 : -8750811 : ff7a7925)<br>Jan 23 17:24:21 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=5821091 : 5821091 : 58d2a3)<br>
Jan 23 17:24:21 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=3356952 : 3356952 : 333918)<br>Jan 23 17:24:21 p186 kernel: Lustre: 8276:0:(pinger.c:193:ptlrpc_pinger_main()) next ping in 25000 (8535847)<br>
Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(o2iblnd_cb.c:2704:kiblnd_cm_callback()) 172.24.198.140@o2ib: ROUTE ERROR -110<br>Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(o2iblnd.c:422:kiblnd_unlink_peer_locked()) peer[efda18c0] -> 172.24.198.140@o2ib (2)--<br>
Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(router.c:151:lnet_notify()) 172.24.198.141@o2ib notifying 172.24.198.140@o2ib: down<br>Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(router.c:82:lnet_notify_locked()) Old news<br>
Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(o2iblnd_cb.c:2118:kiblnd_peer_connect_failed()) Deleting messages for 172.24.198.140@o2ib: connection failed<br>Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(lib-md.c:73:lnet_md_unlink()) Unlinking md ed16acc0<br>
Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(lib-lnet.h:301:lnet_md_free()) kfreed 'md': 84 at ed16acc0 (tot 7259314).<br>Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(lib-lnet.h:344:lnet_msg_free()) kfreed 'msg': 268 at f205a400 (tot 7259046).<br>
Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(o2iblnd_cb.c:2706:kiblnd_cm_callback()) peer[efda18c0] -> 172.24.198.140@o2ib (1)--<br>Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(o2iblnd.c:357:kiblnd_destroy_peer()) kfreed 'peer': 56 at efda18c0 (tot 7258990).<br>
Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(lib-eq.c:146:lib_get_event()) Process entered<br>Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(lib-eq.c:149:lib_get_event()) event: f0b95cf8, sequence: 1, eq->size: 2<br>Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(lib-eq.c:170:lib_get_event()) Process leaving (rc=1 : 1 : 1)<br>
Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(lib-eq.c:232:LNetEQPoll()) Process leaving (rc=1 : 1 : 1)<br>Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(api-ni.c:1665:lnet_ping()) poll 1(4 -113) unlinked<br>Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(lib-lnet.h:259:lnet_eq_free()) kfreed 'eq': 48 at efda1a00 (tot 7258942).<br>
Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(lib-eq.c:135:LNetEQFree()) kfreed 'events': 240 at f0b95c80 (tot 7258702).<br>Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(api-ni.c:1772:lnet_ping()) kfreed 'info': 144 at f0b95880 (tot 7258558).<br>
Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(module.c:336:libcfs_ioctl()) Process leaving (rc=4294967291 : -5 : fffffffb)<br>Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(module.c:178:libcfs_psdev_release()) Process entered<br>
Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(module.c:183:libcfs_psdev_release()) kfreed 'ldu': 8 at f5bc6620 (tot 7258550).<br>Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(module.c:187:libcfs_psdev_release()) Process leaving (rc=0 : 0 : 0)<br>
<br>~subbu<br><br></div></div>
<div>On Fri, Jan 16, 2009 at 3:38 PM, subbu kl <<a href="mailto:subbukl@gmail.com" target="_blank">subbukl@gmail.com</a> <mailto:<a href="mailto:subbukl@gmail.com" target="_blank">subbukl@gmail.com</a>>> wrote:<br>
<br> Liang,<br><br> Right; you reproduced the exact problem. But as you can see in my<br> previous mail I think I have solved that problem by mannually<br> assiging IP to ib0 (check this line # ifconfig ib0 172.24.198.111<br>
and *"Added LNI" lines *)<br><br> we are back to sqare one now I guess ! LNET is up with mannually<br> assigned IPs. normal ping succeds between machines but not lctl ping.<br><br> so my current problem is this :<br>
<br> # lctl ping 172.24.198.112@o2ib<br> failed to ping 172.24.198.112@o2ib: Input/output error<br><br> /var/log/messages:<br><br><br> Jan 16 10:24:14 p128 kernel: Lustre: 2750:0:(o2iblnd_cb.c:2687:<br> kiblnd_cm_callback()) 172.24.198.112@o2ib: ROUTE ERROR -22<br>
Jan 16 10:24:14 p128 kernel: Lustre:<br> 2750:0:(o2iblnd_cb.c:2101:kiblnd_peer_connect_failed()) Deleting<br> messages for 172.24.198.112@o2ib: connection failed<br><br> how can I get rid of this connection problem?<br>
<br> ~subbu<br><br><br><br> On Fri, Jan 16, 2009 at 2:11 PM, Liang Zhen <<a href="mailto:Zhen.Liang@sun.com" target="_blank">Zhen.Liang@sun.com</a><br></div>
<div>
<div> <mailto:<a href="mailto:Zhen.Liang@sun.com" target="_blank">Zhen.Liang@sun.com</a>>> wrote:<br><br> Subbu,<br><br> We don't have any tip for setup IPoIB, looks like linux can't<br> find the ifaddr of ib0 on MDS(-99 is EADDRNOTAVAIL), so I<br>
think it's because you didn't assign any address to ib0 (or<br> failed to assign address to ib0) before loading o2iblnd in<br> the first try.<br> I can reproduce exactly same error by:<br>
1. modprobe ib_ipoib<br> 2. ifconfig ib0 up // without assign any address<br> 3. modprobe ko2iblnd<br> 4. lctl network up<br><br> Regards<br> Liang<br><br> subbu kl:<br><br> Liang,<br>
after executing following echo :<br> echo +neterror > /proc/sys/lnet/printk<br><br> now lctlt ping shows the following error<br><br> # lctl ping 172.24.198.112@o2ib<br> failed to ping 172.24.198.112@o2ib: Input/output error<br>
<br> Jan 16 10:24:14 p128 kernel: Lustre:<br> 2750:0:(o2iblnd_cb.c:2687:kiblnd_cm_callback())<br> 172.24.198.112@o2ib: ROUTE ERROR -22<br> Jan 16 10:24:14 p128 kernel: Lustre:<br> 2750:0:(o2iblnd_cb.c:2101:kiblnd_peer_connect_failed())<br>
Deleting messages for 172.24.198.112@o2ib: connection failed<br><br> Looks like some problem with "IB connection manager" !<br><br> 1. do we have any help docs to setup IPoIB and Lustre,<br>
lustre operation manual has very minimal info about this .<br> I think I am missing some IPoIB setup part here.<br> 2. or is it mannual assignment of IP addresses to "ib0"<br> is creating some problem<br>
<br><br> *Some more supporting info :<br> *subnet manager of following version is also running :<br> OpenSM 3.1.8<br><br> Initially I got this error for MDS mount<br><br> Jan 16 09:45:20 p128 kernel: LustreError:<br>
4991:0:(linux-tcpip.c:124:libcfs_ipif_query()) Can't get<br> IP address for interface ib0<br> Jan 16 09:45:20 p128 kernel: LustreError:<br> 4991:0:(o2iblnd.c:1563:kiblnd_startup()) Can't query IPoIB<br>
interface ib0: -99<br> Jan 16 09:45:21 p128 kernel: LustreError: 105-4: Error<br> -100 starting up LNI o2ib<br> Jan 16 09:45:21 p128 kernel: LustreError:<br> 4991:0:(events.c:707:ptlrpc_init_portals()) network<br>
initialisation failed<br> Jan 16 09:45:21 p128 modprobe: WARNING: Error inserting<br> ptlrpc<br> (/lib/modules/2.6.18-53.1.14.el5_lustre.1.6.5.1smp/kernel/fs/lustre/ptlrpc.ko):<br>
Input/output error<br> Jan 16 09:45:21 p128 modprobe: WARNING: Error inserting<br> osc<br> (/lib/modules/2.6.18-53.1.14.el5_lustre.1.6.5.1smp/kernel/fs/lustre/osc.ko):<br> Unknown symbol in module, or unknown parameter (see dmesg)<br>
Jan 16 09:45:21 p128 kernel: osc: Unknown symbol<br> ldlm_prep_enqueue_req<br> Jan 16 09:45:21 p128 kernel: osc: Unknown symbol<br> ldlm_resource_get<br> Jan 16 09:45:21 p128 kernel: osc: Unknown symbol<br>
ptlrpc_lprocfs_register_obd<br> .<br> .<br> .<br><br> then I mannually set the IP address for ib0 as folows :<br> # ifconfig ib0 172.24.198.111<br><br> [root@p186 ~]# ifconfig ib0<br>
ib0 Link encap:InfiniBand HWaddr<br> 80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00<br> inet addr:172.24.198.112 Bcast:172.24.255.255<br> Mask:255.255.0.0<br>
UP BROADCAST MULTICAST MTU:65520 Metric:1<br> RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:256<br>
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)<br><br> then it mounted sucessfully<br><br> *Jan 16 09:47:09 p128 kernel: Lustre: Added LNI<br> 172.24.198.111@o2ib [8/64]<br> Jan 16 09:47:09 p128 kernel: Lustre: MGS MGS started*<br>
Jan 16 09:47:09 p128 kernel: Lustre: Setting parameter<br> lustre-MDT0000.mdt.group_upcall in log lustre-MDT0000<br> Jan 16 09:47:09 p128 kernel: Lustre: Enabling user_xattr<br> Jan 16 09:47:09 p128 kernel: Lustre: lustre-MDT0000: new<br>
disk, initializing<br> Jan 16 09:47:09 p128 kernel: Lustre: MDT lustre-MDT0000<br> now serving dev<br> (lustre-MDT0000/64db1fc7-03ba-9803-4d20-ab0d2aa66116) with<br> recovery enabled<br>
Jan 16 09:47:09 p128 kernel: Lustre:<br> 5274:0:(lproc_mds.c:262:lprocfs_wr_group_upcall())<br> lustre-MDT0000: group upcall set to /usr/sbin/l_getgroups<br> Jan 16 09:47:09 p128 kernel: Lustre: lustre-MDT0000.mdt:<br>
set parameter group_upcall=/usr/sbin/l_getgroups<br> Jan 16 09:47:09 p128 kernel: Lustre: Server lustre-MDT0000<br> on device /dev/loop0 has started<br> .<br> .<br> .<br>
<br><br> ~subbu<br><br><br> On Thu, Jan 15, 2009 at 8:37 PM, Liang Zhen<br> <<a href="mailto:Zhen.Liang@sun.com" target="_blank">Zhen.Liang@sun.com</a> <mailto:<a href="mailto:Zhen.Liang@sun.com" target="_blank">Zhen.Liang@sun.com</a>><br>
</div></div> <mailto:<a href="mailto:Zhen.Liang@sun.com" target="_blank">Zhen.Liang@sun.com</a> <mailto:<a href="mailto:Zhen.Liang@sun.com" target="_blank">Zhen.Liang@sun.com</a>>>>
<div>
<div><br> wrote:<br><br> Subbu,<br><br> I'd suggest:<br> 1) make sure ko2iblnd has been brought up (please check<br> if there<br> is any error message when startup ko2iblnd)<br>
2) echo +neterror > /proc/sys/lnet/printk, then try<br> with lctl<br> ping, if it still can't work please post error messages<br><br> Regards<br> Liang<br>
<br> subbu kl:<br><br> Problem is similer to<br> <a href="http://lists.lustre.org/pipermail/lustre-discuss/2008-May/007498.html" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss/2008-May/007498.html</a><br>
But by looking at the thread could not really get<br> the solution<br> for the problem.<br><br> I have two RHEL5 Linux servers installed with<br> following packages -<br>
<br> kernel-lustre-smp-2.6.18-53.1.14.el5_lustre.1.6.5.1<br> kernel-ib-1.3-2.6.18_53.1.14.el5_lustre.1.6.5.1smp<br> lustre-ldiskfs-3.0.4-2.6.18_53.1.14.el5_lustre.1.6.5.1smp<br>
lustre-1.6.5.1-2.6.18_53.1.14.el5_lustre.1.6.5.1smp<br> lustre-modules-1.6.5.1-2.6.18_53.1.14.el5_lustre.1.6.5.1smp<br> e2fsprogs-1.40.7.sun3-0redhat<br><br><br>
machine 1: with ib0 IP address : 172.24.198.111<br> machine 2: with ib0 IP address : 172.24.198.112<br><br> /etc/modprobe.conf contains<br> options lnet networks=o2ib<br>
<br> TCP networking worked fine and now I am trying with<br> Infiniband<br> network finding it difficult in communicating with<br> IB nodes<br> mounting effort throghs me the following error<br>
<br> [root@p186 ~]# mount -t lustre -o loop<br> /tmp/lustre-ost1 /mnt/ost1<br> mount.lustre: mount /dev/loop0 at /mnt/ost1 failed:<br> Input/output error<br> Is the MGS running?<br>
<br> /var/log/messages :<br> Jan 15 16:55:25 p186 kernel: kjournald starting.<br> Commit<br> interval 5 seconds<br> Jan 15 16:55:25 p186 kernel: LDISKFS FS on loop0,<br>
internal journal<br> Jan 15 16:55:25 p186 kernel: LDISKFS-fs: mounted<br> filesystem<br> with ordered data mode.<br> Jan 15 16:55:25 p186 kernel: kjournald starting.<br>
Commit<br> interval 5 seconds<br> Jan 15 16:55:25 p186 kernel: LDISKFS FS on loop0,<br> internal journal<br> Jan 15 16:55:25 p186 kernel: LDISKFS-fs: mounted<br>
filesystem<br> with ordered data mode.<br> Jan 15 16:55:25 p186 kernel: LDISKFS-fs: file<br> extents enabled<br> Jan 15 16:55:25 p186 kernel: LDISKFS-fs: mballoc<br>
enabled<br> Jan 15 16:55:30 p186 kernel: Lustre: Request x7<br> sent from<br> MGC172.24.198.111@o2ib to NID 172.24.198.111@o2ib<br> 5s ago has<br> timed out (limit 5s).<br>
Jan 15 16:55:30 p186 kernel: LustreError:<br> 7193:0:(obd_mount.c:1062:server_start_targets())<br> Required<br> registration failed for lustre-OSTffff: -5<br> Jan 15 16:55:30 p186 kernel: LustreError: 15f-b:<br>
Communication<br> error with the MGS. Is the MGS running?<br> Jan 15 16:55:30 p186 kernel: LustreError:<br> 7193:0:(obd_mount.c:1597:server_fill_super())<br> Unable to start<br>
targets: -5<br> Jan 15 16:55:30 p186 kernel: LustreError:<br> 7193:0:(obd_mount.c:1382:server_put_super()) no obd<br> lustre-OSTffff<br> Jan 15 16:55:30 p186 kernel: LustreError:<br>
7193:0:(obd_mount.c:119:server_deregister_mount())<br> lustre-OSTffff not registered<br> Jan 15 16:55:30 p186 kernel: LDISKFS-fs: mballoc: 0<br> blocks 0<br> reqs (0 success)<br>
Jan 15 16:55:30 p186 kernel: LDISKFS-fs: mballoc: 0<br> extents<br> scanned, 0 goal hits, 0 2^N hits, 0 breaks, 0 lost<br> Jan 15 16:55:30 p186 kernel: LDISKFS-fs: mballoc: 0<br>
generated<br> and it took 0<br> Jan 15 16:55:30 p186 kernel: LDISKFS-fs: mballoc: 0<br> preallocated, 0 discarded<br> Jan 15 16:55:30 p186 kernel: Lustre: server umount<br>
lustre-OSTffff complete<br> Jan 15 16:55:30 p186 kernel: LustreError:<br> 7193:0:(obd_mount.c:1951:lustre_fill_super())<br> Unable to mount<br> (-5)<br>
<br> All pinging efforts also failed to the IB NIDS<br> local/remote<br> can ping the ip address :<br> [root@p186 ~]# ping 172.24.198.112<br> PING 172.24.198.112 (172.24.198.112) 56(84) bytes<br>
of data.<br> 64 bytes from 172.24.198.112 <<a href="http://172.24.198.112/" target="_blank">http://172.24.198.112</a>>:<br> icmp_seq=1 ttl=64 time=0.052 ms<br> 64 bytes from 172.24.198.112 <<a href="http://172.24.198.112/" target="_blank">http://172.24.198.112</a>>:<br>
icmp_seq=2 ttl=64 time=0.024 ms<br><br><br> --- 172.24.198.112 ping statistics ---<br> 2 packets transmitted, 2 received, 0% packet loss,<br> time 1000ms<br> rtt min/avg/max/mdev = 0.024/0.038/0.052/0.014 ms<br>
[root@p186 ~]# ping 172.24.198.111<br> PING 172.24.198.111 (172.24.198.111) 56(84) bytes<br> of data.<br> 64 bytes from 172.24.198.111 <<a href="http://172.24.198.111/" target="_blank">http://172.24.198.111</a>>:<br>
icmp_seq=1 ttl=64 time=2.16 ms<br> 64 bytes from 172.24.198.111 <<a href="http://172.24.198.111/" target="_blank">http://172.24.198.111</a>>:<br> icmp_seq=2 ttl=64 time=0.296 ms<br>
<br><br> --- 172.24.198.111 ping statistics ---<br> 2 packets transmitted, 2 received, 0% packet loss,<br> time 1000ms<br> rtt min/avg/max/mdev = 0.296/1.231/2.166/0.935 ms<br>
<br> but cant ping the NIDS :<br> [root@p186 ~]# lctl ping 172.24.198.112@o2ib<br> failed to ping 172.24.198.112@o2ib: Input/output error<br> [root@p186 ~]# lctl ping 172.24.198.111@o2ib<br>
failed to ping 172.24.198.111@o2ib: Input/output error<br><br> Any idea why lnet cant ping NIDS ?<br><br> some more configurations:<br> [root@p186 ~]# ibstat<br>
CA 'mthca0'<br> CA type: MT23108<br> Number of ports: 2<br> Firmware version: 3.5.0<br> Hardware version: a1<br>
Node GUID: 0x0002c9020021550c<br><br> Machines are connected via IB switch.<br><br> Looking forward for help.<br><br> ~subbu<br> ------------------------------------------------------------------------<br>
<br> _______________________________________________<br> Lustre-discuss mailing list<br> <a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<mailto:<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a>><br> <mailto:<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<mailto:<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a>>><br><br> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br><br><br><br> -- . . . s u b b u<br> "You've got to be original, because if you're like someone<br> else, what do they need you for?"<br> ------------------------------------------------------------------------<br>
<br> _______________________________________________<br> Lustre-discuss mailing list<br> <a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<mailto:<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a>><br> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br><br><br><br><br> -- . . . s u b b u<br> "You've got to be original, because if you're like someone else,<br> what do they need you for?"<br><br><br><br><br>-- <br>. . . s u b b u<br>
"You've got to be original, because if you're like someone else, what do they need you for?"<br>------------------------------------------------------------------------<br><br>_______________________________________________<br>
Lustre-discuss mailing list<br><a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br></div></div></blockquote><br></blockquote></div><br><br clear="all"><br>-- <br>. . . s u b b u<br>"You've got to be original, because if you're like someone else, what do they need you for?"<br></div>
</div></blockquote></div><br><br clear="all"><br>-- <br>. . . s u b b u<br>"You've got to be original, because if you're like someone else, what do they need you for?"<br>