problem remained same, when I run lctl ping with tcpdump 4.0.0 I dont see any activity on ib0 !<br><br>another exhaustive Lustre debug log I took with lctl ping do you see any problem with it ?<br>
<br>
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(module.c:160:libcfs_psdev_open()) Process entered</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(module.c:164:libcfs_psdev_open()) kmalloced 'ldu': 8 at f5bc6620 (tot 7258558).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(module.c:171:libcfs_psdev_open()) Process leaving (rc=0 : 0 : 0)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(module.c:228:libcfs_ioctl()) Process entered</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(linux-module.c:49:libcfs_ioctl_getdata()) Process entered</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(linux-module.c:90:libcfs_ioctl_getdata()) Process leaving (rc=0 : 0 : 0)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(api-ni.c:1223:LNetNIInit()) refs 1</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(api-ni.c:1614:lnet_ping()) kmalloced 'info': 144 at f0b95880 (tot 7258702).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-lnet.h:251:lnet_eq_alloc()) kmalloced 'eq': 48 at efda1a00 (tot 7258750).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186
kernel: Lustre: 14294:0:(lib-eq.c:72:LNetEQAlloc()) kmalloced
'eq->eq_events': 240 at f0b95c80 (tot 7258990).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-lnet.h:279:lnet_md_alloc()) kmalloced 'md': 84 at ed16acc0 (tot 7259074).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-lnet.h:327:lnet_msg_alloc()) kmalloced 'msg': 268 at f205a400 (tot 7259342).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-move.c:2395:LNetGet()) LNetGet -> 12345-172.24.198.140@o2ib</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186
kernel: Lustre: 14294:0:(o2iblnd_cb.c:1531:kiblnd_send()) sending 0
bytes in 0 frags to 12345-172.24.198.140@o2ib</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186
kernel: Lustre: 14294:0:(o2iblnd.c:312:kiblnd_create_peer()) kmalloced
'peer': 56 at efda18c0 (tot 7259398).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186
kernel: Lustre: 14294:0:(o2iblnd_cb.c:1501:kiblnd_launch_tx())
peer[efda18c0] -> 172.24.198.140@o2ib (1)++</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186
kernel: Lustre: 14294:0:(o2iblnd_cb.c:1380:kiblnd_connect_peer())
peer[efda18c0] -> 172.24.198.140@o2ib (2)++</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186
kernel: Lustre: 14294:0:(o2iblnd_cb.c:1507:kiblnd_launch_tx())
peer[efda18c0] -> 172.24.198.140@o2ib (3)--</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-eq.c:209:LNetEQPoll()) Process entered</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-eq.c:146:lib_get_event()) Process entered</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-eq.c:149:lib_get_event()) event: f0b95cf8, sequence: 1, eq->size: 2</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 14294:0:(lib-eq.c:152:lib_get_event()) Process leaving (rc=0 : 0 : 0)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:39 p186 kernel: Lustre: 2782:0:(o2iblnd_cb.c:2682:kiblnd_cm_callback()) 172.24.198.140@o2ib Addr resolved: 0</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:146:lib_get_event()) Process entered</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:149:lib_get_event()) event: f0b95cf8, sequence: 1, eq->size: 2</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:152:lib_get_event()) Process leaving (rc=0 : 0 : 0)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:239:LNetEQPoll()) Process leaving (rc=0 : 0 : 0)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(api-ni.c:1665:lnet_ping()) poll 0(-1 -1)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-md.c:69:lnet_md_unlink()) Queueing unlink of md ed16acc0</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:209:LNetEQPoll()) Process entered</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:146:lib_get_event()) Process entered</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:149:lib_get_event()) event: f0b95cf8, sequence: 1, eq->size: 2</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:40 p186 kernel: Lustre: 14294:0:(lib-eq.c:152:lib_get_event()) Process leaving (rc=0 : 0 : 0)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:56 p186
kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process
leaving (rc=4294962944 : -4352 : ffffef00)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:56 p186
kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process
leaving (rc=4294966784 : -512 : fffffe00)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:56 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=2817 : 2817 : b01)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:56 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=2047 : 2047 : 7ff)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:56 p186
kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process
leaving (rc=4294740832 : -226464 : fffc8b60)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:56 p186
kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process
leaving (rc=4286216485 : -8750811 : ff7a7925)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:56 p186
kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process
leaving (rc=5821091 : 5821091 : 58d2a3)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:56 p186
kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process
leaving (rc=3356952 : 3356952 : 333918)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:23:56 p186 kernel: Lustre: 8276:0:(pinger.c:193:ptlrpc_pinger_main()) next ping in 25000 (8510847)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:21 p186
kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process
leaving (rc=4294962944 : -4352 : ffffef00)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:21 p186
kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process
leaving (rc=4294966784 : -512 : fffffe00)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:21 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=2817 : 2817 : b01)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:21 p186 kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process leaving (rc=2047 : 2047 : 7ff)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:21 p186
kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process
leaving (rc=4294740832 : -226464 : fffc8b60)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:21 p186
kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process
leaving (rc=4286216485 : -8750811 : ff7a7925)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:21 p186
kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process
leaving (rc=5821091 : 5821091 : 58d2a3)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:21 p186
kernel: Lustre: 8276:0:(lvfs_lib.c:173:lprocfs_read_helper()) Process
leaving (rc=3356952 : 3356952 : 333918)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:21 p186 kernel: Lustre: 8276:0:(pinger.c:193:ptlrpc_pinger_main()) next ping in 25000 (8535847)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(o2iblnd_cb.c:2704:kiblnd_cm_callback()) 172.24.198.140@o2ib: ROUTE ERROR -110</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186
kernel: Lustre: 2794:0:(o2iblnd.c:422:kiblnd_unlink_peer_locked())
peer[efda18c0] -> 172.24.198.140@o2ib (2)--</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(router.c:151:lnet_notify()) 172.24.198.141@o2ib notifying 172.24.198.140@o2ib: down</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(router.c:82:lnet_notify_locked()) Old news</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186
kernel: Lustre: 2794:0:(o2iblnd_cb.c:2118:kiblnd_peer_connect_failed())
Deleting messages for 172.24.198.140@o2ib: connection failed</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(lib-md.c:73:lnet_md_unlink()) Unlinking md ed16acc0</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(lib-lnet.h:301:lnet_md_free()) kfreed 'md': 84 at ed16acc0 (tot 7259314).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(lib-lnet.h:344:lnet_msg_free()) kfreed 'msg': 268 at f205a400 (tot 7259046).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186
kernel: Lustre: 2794:0:(o2iblnd_cb.c:2706:kiblnd_cm_callback())
peer[efda18c0] -> 172.24.198.140@o2ib (1)--</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 2794:0:(o2iblnd.c:357:kiblnd_destroy_peer()) kfreed 'peer': 56 at efda18c0 (tot 7258990).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(lib-eq.c:146:lib_get_event()) Process entered</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(lib-eq.c:149:lib_get_event()) event: f0b95cf8, sequence: 1, eq->size: 2</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(lib-eq.c:170:lib_get_event()) Process leaving (rc=1 : 1 : 1)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(lib-eq.c:232:LNetEQPoll()) Process leaving (rc=1 : 1 : 1)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(api-ni.c:1665:lnet_ping()) poll 1(4 -113) unlinked</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(lib-lnet.h:259:lnet_eq_free()) kfreed 'eq': 48 at efda1a00 (tot 7258942).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(lib-eq.c:135:LNetEQFree()) kfreed 'events': 240 at f0b95c80 (tot 7258702).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(api-ni.c:1772:lnet_ping()) kfreed 'info': 144 at f0b95880 (tot 7258558).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(module.c:336:libcfs_ioctl()) Process leaving (rc=4294967291 : -5 : fffffffb)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(module.c:178:libcfs_psdev_release()) Process entered</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(module.c:183:libcfs_psdev_release()) kfreed 'ldu': 8 at f5bc6620 (tot 7258550).</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Jan 23 17:24:29 p186 kernel: Lustre: 14294:0:(module.c:187:libcfs_psdev_release()) Process leaving (rc=0 : 0 : 0)</span><br>
<br>~subbu<br><br><div class="gmail_quote">On Fri, Jan 16, 2009 at 3:38 PM, subbu kl <span dir="ltr"><<a href="mailto:subbukl@gmail.com">subbukl@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Liang,<br><br>Right; you reproduced the exact problem. But as you can see in my previous mail I think I have solved that problem by mannually assiging IP to ib0 (check this line # ifconfig ib0 172.24.198.111 and <b><span style="color: rgb(255, 0, 0);">"Added LNI" lines </span></b>)<br>
<br>we are back to sqare one now I guess ! LNET is up with mannually assigned IPs. normal ping succeds between machines but not lctl ping.<br><br>so my current problem is this :<div class="Ih2E3d"><br>
# lctl ping 172.24.198.112@o2ib<br>
failed to ping 172.24.198.112@o2ib: Input/output error<br>
<br></div>/var/log/messages:<div class="Ih2E3d"><br><br>
Jan 16 10:24:14 p128 kernel: Lustre: 2750:0:(o2iblnd_cb.c:2687:</div><div><div class="Ih2E3d">kiblnd_cm_callback()) 172.24.198.112@o2ib: <span style="color: rgb(255, 0, 0);">ROUTE ERROR -22</span><br>
Jan 16 10:24:14 p128 kernel: Lustre: 2750:0:(o2iblnd_cb.c:2101:kiblnd_peer_connect_failed()) Deleting messages for 172.24.198.112@o2ib: <span style="color: rgb(255, 0, 0);">connection failed</span><br><br></div>how can I get rid of this connection problem?<br>
</div><br>~subbu<div><div></div><div class="Wj3C7c"><br><br><br><div class="gmail_quote">On Fri, Jan 16, 2009 at 2:11 PM, Liang Zhen <span dir="ltr"><<a href="mailto:Zhen.Liang@sun.com" target="_blank">Zhen.Liang@sun.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Subbu,<br>
<br>
We don't have any tip for setup IPoIB, looks like linux can't find the ifaddr of ib0 on MDS(-99 is EADDRNOTAVAIL), so I think it's because you didn't assign any address to ib0 (or failed to assign address to ib0) before loading o2iblnd in the first try.<br>
I can reproduce exactly same error by:<br>
1. modprobe ib_ipoib<br>
2. ifconfig ib0 up // without assign any address<br>
3. modprobe ko2iblnd<br>
4. lctl network up<br>
<br>
Regards<br>
Liang<br>
<br>
subbu kl:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div></div><div>
Liang,<br>
after executing following echo :<br>
echo +neterror > /proc/sys/lnet/printk<br>
<br>
now lctlt ping shows the following error<br>
<br>
# lctl ping 172.24.198.112@o2ib<br>
failed to ping 172.24.198.112@o2ib: Input/output error<br>
<br>
Jan 16 10:24:14 p128 kernel: Lustre: 2750:0:(o2iblnd_cb.c:2687:kiblnd_cm_callback()) 172.24.198.112@o2ib: ROUTE ERROR -22<br>
Jan 16 10:24:14 p128 kernel: Lustre: 2750:0:(o2iblnd_cb.c:2101:kiblnd_peer_connect_failed()) Deleting messages for 172.24.198.112@o2ib: connection failed<br>
<br>
Looks like some problem with "IB connection manager" !<br>
<br>
1. do we have any help docs to setup IPoIB and Lustre, lustre operation manual has very minimal info about this . I think I am missing some IPoIB setup part here.<br>
2. or is it mannual assignment of IP addresses to "ib0" is creating some problem<br>
<br>
<br>
*Some more supporting info :<br>
*subnet manager of following version is also running : OpenSM 3.1.8<br>
<br>
Initially I got this error for MDS mount<br>
<br>
Jan 16 09:45:20 p128 kernel: LustreError: 4991:0:(linux-tcpip.c:124:libcfs_ipif_query()) Can't get IP address for interface ib0<br>
Jan 16 09:45:20 p128 kernel: LustreError: 4991:0:(o2iblnd.c:1563:kiblnd_startup()) Can't query IPoIB interface ib0: -99<br>
Jan 16 09:45:21 p128 kernel: LustreError: 105-4: Error -100 starting up LNI o2ib<br>
Jan 16 09:45:21 p128 kernel: LustreError: 4991:0:(events.c:707:ptlrpc_init_portals()) network initialisation failed<br>
Jan 16 09:45:21 p128 modprobe: WARNING: Error inserting ptlrpc (/lib/modules/2.6.18-53.1.14.el5_lustre.1.6.5.1smp/kernel/fs/lustre/ptlrpc.ko): Input/output error<br>
Jan 16 09:45:21 p128 modprobe: WARNING: Error inserting osc (/lib/modules/2.6.18-53.1.14.el5_lustre.1.6.5.1smp/kernel/fs/lustre/osc.ko): Unknown symbol in module, or unknown parameter (see dmesg)<br>
Jan 16 09:45:21 p128 kernel: osc: Unknown symbol ldlm_prep_enqueue_req<br>
Jan 16 09:45:21 p128 kernel: osc: Unknown symbol ldlm_resource_get<br>
Jan 16 09:45:21 p128 kernel: osc: Unknown symbol ptlrpc_lprocfs_register_obd<br>
.<br>
.<br>
.<br>
<br>
then I mannually set the IP address for ib0 as folows :<br># ifconfig ib0 172.24.198.111<br>
<br>
[root@p186 ~]# ifconfig ib0<br>
ib0 Link encap:InfiniBand HWaddr 80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00<br>
inet addr:172.24.198.112 Bcast:172.24.255.255 Mask:255.255.0.0<br>
UP BROADCAST MULTICAST MTU:65520 Metric:1<br>
RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br>
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:256<br>
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)<br>
<br>
then it mounted sucessfully<br>
<br><b><span style="color: rgb(255, 0, 0);">
Jan 16 09:47:09 p128 kernel: Lustre: Added LNI 172.24.198.111@o2ib [8/64]</span><br style="color: rgb(255, 0, 0);"><span style="color: rgb(255, 0, 0);">
Jan 16 09:47:09 p128 kernel: Lustre: MGS MGS started</span></b><br>
Jan 16 09:47:09 p128 kernel: Lustre: Setting parameter lustre-MDT0000.mdt.group_upcall in log lustre-MDT0000<br>
Jan 16 09:47:09 p128 kernel: Lustre: Enabling user_xattr<br>
Jan 16 09:47:09 p128 kernel: Lustre: lustre-MDT0000: new disk, initializing<br>
Jan 16 09:47:09 p128 kernel: Lustre: MDT lustre-MDT0000 now serving dev (lustre-MDT0000/64db1fc7-03ba-9803-4d20-ab0d2aa66116) with recovery enabled<br>
Jan 16 09:47:09 p128 kernel: Lustre: 5274:0:(lproc_mds.c:262:lprocfs_wr_group_upcall()) lustre-MDT0000: group upcall set to /usr/sbin/l_getgroups<br>
Jan 16 09:47:09 p128 kernel: Lustre: lustre-MDT0000.mdt: set parameter group_upcall=/usr/sbin/l_getgroups<br>
Jan 16 09:47:09 p128 kernel: Lustre: Server lustre-MDT0000 on device /dev/loop0 has started<br>
.<br>
.<br>
.<br>
<br>
<br>
~subbu<br>
<br>
<br></div></div><div><div></div><div>
On Thu, Jan 15, 2009 at 8:37 PM, Liang Zhen <<a href="mailto:Zhen.Liang@sun.com" target="_blank">Zhen.Liang@sun.com</a> <mailto:<a href="mailto:Zhen.Liang@sun.com" target="_blank">Zhen.Liang@sun.com</a>>> wrote:<br>
<br>
Subbu,<br>
<br>
I'd suggest:<br>
1) make sure ko2iblnd has been brought up (please check if there<br>
is any error message when startup ko2iblnd)<br>
2) echo +neterror > /proc/sys/lnet/printk, then try with lctl<br>
ping, if it still can't work please post error messages<br>
<br>
Regards<br>
Liang<br>
<br>
subbu kl:<br>
<br>
Problem is similer to<br>
<a href="http://lists.lustre.org/pipermail/lustre-discuss/2008-May/007498.html" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss/2008-May/007498.html</a><br>
But by looking at the thread could not really get the solution<br>
for the problem.<br>
<br>
I have two RHEL5 Linux servers installed with following packages -<br>
<br>
kernel-lustre-smp-2.6.18-53.1.14.el5_lustre.1.6.5.1<br>
kernel-ib-1.3-2.6.18_53.1.14.el5_lustre.1.6.5.1smp<br>
lustre-ldiskfs-3.0.4-2.6.18_53.1.14.el5_lustre.1.6.5.1smp<br>
lustre-1.6.5.1-2.6.18_53.1.14.el5_lustre.1.6.5.1smp<br>
lustre-modules-1.6.5.1-2.6.18_53.1.14.el5_lustre.1.6.5.1smp<br>
e2fsprogs-1.40.7.sun3-0redhat<br>
<br>
<br>
machine 1: with ib0 IP address : 172.24.198.111<br>
machine 2: with ib0 IP address : 172.24.198.112<br>
<br>
/etc/modprobe.conf contains<br>
options lnet networks=o2ib<br>
<br>
TCP networking worked fine and now I am trying with Infiniband<br>
network finding it difficult in communicating with IB nodes<br>
mounting effort throghs me the following error<br>
<br>
[root@p186 ~]# mount -t lustre -o loop /tmp/lustre-ost1 /mnt/ost1<br>
mount.lustre: mount /dev/loop0 at /mnt/ost1 failed:<br>
Input/output error<br>
Is the MGS running?<br>
<br>
/var/log/messages :<br>
Jan 15 16:55:25 p186 kernel: kjournald starting. Commit<br>
interval 5 seconds<br>
Jan 15 16:55:25 p186 kernel: LDISKFS FS on loop0, internal journal<br>
Jan 15 16:55:25 p186 kernel: LDISKFS-fs: mounted filesystem<br>
with ordered data mode.<br>
Jan 15 16:55:25 p186 kernel: kjournald starting. Commit<br>
interval 5 seconds<br>
Jan 15 16:55:25 p186 kernel: LDISKFS FS on loop0, internal journal<br>
Jan 15 16:55:25 p186 kernel: LDISKFS-fs: mounted filesystem<br>
with ordered data mode.<br>
Jan 15 16:55:25 p186 kernel: LDISKFS-fs: file extents enabled<br>
Jan 15 16:55:25 p186 kernel: LDISKFS-fs: mballoc enabled<br>
Jan 15 16:55:30 p186 kernel: Lustre: Request x7 sent from<br>
MGC172.24.198.111@o2ib to NID 172.24.198.111@o2ib 5s ago has<br>
timed out (limit 5s).<br>
Jan 15 16:55:30 p186 kernel: LustreError:<br>
7193:0:(obd_mount.c:1062:server_start_targets()) Required<br>
registration failed for lustre-OSTffff: -5<br>
Jan 15 16:55:30 p186 kernel: LustreError: 15f-b: Communication<br>
error with the MGS. Is the MGS running?<br>
Jan 15 16:55:30 p186 kernel: LustreError:<br>
7193:0:(obd_mount.c:1597:server_fill_super()) Unable to start<br>
targets: -5<br>
Jan 15 16:55:30 p186 kernel: LustreError:<br>
7193:0:(obd_mount.c:1382:server_put_super()) no obd lustre-OSTffff<br>
Jan 15 16:55:30 p186 kernel: LustreError:<br>
7193:0:(obd_mount.c:119:server_deregister_mount())<br>
lustre-OSTffff not registered<br>
Jan 15 16:55:30 p186 kernel: LDISKFS-fs: mballoc: 0 blocks 0<br>
reqs (0 success)<br>
Jan 15 16:55:30 p186 kernel: LDISKFS-fs: mballoc: 0 extents<br>
scanned, 0 goal hits, 0 2^N hits, 0 breaks, 0 lost<br>
Jan 15 16:55:30 p186 kernel: LDISKFS-fs: mballoc: 0 generated<br>
and it took 0<br>
Jan 15 16:55:30 p186 kernel: LDISKFS-fs: mballoc: 0<br>
preallocated, 0 discarded<br>
Jan 15 16:55:30 p186 kernel: Lustre: server umount<br>
lustre-OSTffff complete<br>
Jan 15 16:55:30 p186 kernel: LustreError:<br>
7193:0:(obd_mount.c:1951:lustre_fill_super()) Unable to mount<br>
(-5)<br>
<br>
All pinging efforts also failed to the IB NIDS local/remote<br>
can ping the ip address :<br>
[root@p186 ~]# ping 172.24.198.112<br>
PING 172.24.198.112 (172.24.198.112) 56(84) bytes of data.<br>
64 bytes from 172.24.198.112 <<a href="http://172.24.198.112" target="_blank">http://172.24.198.112</a>>:<br>
icmp_seq=1 ttl=64 time=0.052 ms<br>
64 bytes from 172.24.198.112 <<a href="http://172.24.198.112" target="_blank">http://172.24.198.112</a>>:<br>
icmp_seq=2 ttl=64 time=0.024 ms<br>
<br>
<br>
--- 172.24.198.112 ping statistics ---<br>
2 packets transmitted, 2 received, 0% packet loss, time 1000ms<br>
rtt min/avg/max/mdev = 0.024/0.038/0.052/0.014 ms<br>
[root@p186 ~]# ping 172.24.198.111<br>
PING 172.24.198.111 (172.24.198.111) 56(84) bytes of data.<br>
64 bytes from 172.24.198.111 <<a href="http://172.24.198.111" target="_blank">http://172.24.198.111</a>>:<br>
icmp_seq=1 ttl=64 time=2.16 ms<br>
64 bytes from 172.24.198.111 <<a href="http://172.24.198.111" target="_blank">http://172.24.198.111</a>>:<br>
icmp_seq=2 ttl=64 time=0.296 ms<br>
<br>
<br>
--- 172.24.198.111 ping statistics ---<br>
2 packets transmitted, 2 received, 0% packet loss, time 1000ms<br>
rtt min/avg/max/mdev = 0.296/1.231/2.166/0.935 ms<br>
<br>
but cant ping the NIDS :<br>
[root@p186 ~]# lctl ping 172.24.198.112@o2ib<br>
failed to ping 172.24.198.112@o2ib: Input/output error<br>
[root@p186 ~]# lctl ping 172.24.198.111@o2ib<br>
failed to ping 172.24.198.111@o2ib: Input/output error<br>
<br>
Any idea why lnet cant ping NIDS ?<br>
<br>
some more configurations:<br>
[root@p186 ~]# ibstat<br>
CA 'mthca0'<br>
CA type: MT23108<br>
Number of ports: 2<br>
Firmware version: 3.5.0<br>
Hardware version: a1<br>
Node GUID: 0x0002c9020021550c<br>
<br>
Machines are connected via IB switch.<br>
<br>
Looking forward for help.<br>
<br>
~subbu<br>
------------------------------------------------------------------------<br>
<br>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br></div></div>
<mailto:<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a>><div><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br>
<br>
<br>
<br>
<br>
-- <br>
. . . s u b b u<br>
"You've got to be original, because if you're like someone else, what do they need you for?"<br>
------------------------------------------------------------------------<br>
<br>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br>
</div></blockquote>
<br>
</blockquote></div><br><br clear="all"><br>-- <br>. . . s u b b u<br>"You've got to be original, because if you're like someone else, what do they need you for?"<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>. . . s u b b u<br>"You've got to be original, because if you're like someone else, what do they need you for?"<br>