<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>I don't know if the voltaire IB stack is the same as OFED but I'm guessing it has a subnet manager. Check that. I've had similar issues when my subnet manager has crashed.</div><br><div><div>On Jan 9, 2008, at 3:08 AM, Changer Van wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Network connection is down. I can not ping the other nodes.</div> <div>I ran the vstat command and found one of the port_state is 'port_initialize'.</div> <div>What does 'port_initialize' mean? Dose it mean my ib card is broken?</div> <div> </div> <div>1 HCA found:<br> hca_id=InfiniHost_III_Ex0<br> pci_location={BUS=0x20,DEV/FUNC=0x00}<br> vendor_id=0x02C9<br> vendor_part_id=0x6282<br> hw_ver=0xA0<br> fw_ver=5.1.400<br> PSID=MT_0140000001 <br> num_phys_ports=2<br> port=1<br> port_state=PORT_INITIALIZE<br> sm_lid=0x0000<br> port_lid=0x0000<br> port_lmc=0x00<br> max_mtu=2048 </div> <div> port=2<br> port_state=PORT_DOWN<br> sm_lid=0x0000<br> port_lid=0x0000<br> port_lmc=0x00<br> max_mtu=2048</div> <div>-- <br>Regards,<br>Changer <br><br></div> <div class="gmail_quote">On Jan 9, 2008 3:27 AM, Klaus Steden <<a href="mailto:klaus.steden@thomson.net">klaus.steden@thomson.net</a>> wrote:<br> <blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"> <div><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px"><br>If you're using IPoIB, you can use standard TCP/IP diagnostic tools the same way you would on an Ethernet link (ifconfig, ping, traceroute, telnet, etc.) <br><br>If you're using a copper-to-optical converter in your data path as well, the Emcore MIAs have link lights on them which will tell you if a physical link is present (check the documentation). I know with STP InfiniBand connectors, there is some ambiguity about terminology with some vendors and manufacturers, and the fibre arrangement doesn't provide a lot of wiggle room. <br><br>Klaus<br><br>On 1/7/08 7:56 PM, "Changer Van" <<a href="mailto:changerv@gmail.com" target="_blank">changerv@gmail.com</a>>did etch on stone tablets:<br><br></span></font> <blockquote> <div class="Ih2E3d"><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px"><br><br>On Jan 8, 2008 1:35 AM, Isaac Huang <<a href="mailto:He.Huang@sun.com" target="_blank">He.Huang@sun.com</a>> wrote:<br> </span></font> <blockquote><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px">On Mon, Jan 07, 2008 at 06:20:52PM +0800, Changer Van wrote:<br>> ......<br>> # dmesg<br>><br>> LustreError: 4273:0:(viblnd.c :1890:kibnal_startup())<br>><br>> Can't find an active port on InfiniHost_III_Ex0<br><br>It meant that viblnd couldn't find a port whose link state was active <br>on the hca InfiniHost_III_Ex0, i.e . no link on the device was usable.<br><br>Was there any other error messages from viblnd before this one? <br></span></font></blockquote><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px">There was no error messages but a related message <br>like 'ADDRCONF(NETDEV_UP):ipoib0: link is not ready'.<br></span></font> <blockquote><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px">Did you see this problem on just one node?<br></span></font></blockquote></div><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px"> <div class="Ih2E3d">There are four nodes which can not mount the lustre system.<br>The other nodes can mount the lustre but got the following error messages:<br> <br># dmesg<br>divert: not allocating divert_blk for non-ethernet device ipoib0 <br>ERROR : IPOIB_UD : ipoib_ud_find_dev_by_dst:(ipoib_ud_arp.c):<br></div> ip_route_output_key(<a href="http://127.0.0.1/" target="_blank">127.0.0.1</a> <a href="http://127.0.0.1/" target="_blank"><http://127.0.0.1> </a> ) failed <div class="Ih2E3d"><br>new: ipoib_allow_arp_joins: 1<br>ERROR : IPOIB_UD : ipoib_ud_find_dev_by_dst:(ipoib_ud_arp.c):<br></div> ip_route_output_key(<a href="http://11.0.0.4/" target="_blank">11.0.0.4</a> <a href="http://11.0.0.4/" target="_blank"> <http://11.0.0.4></a> ) failed <div class="Ih2E3d"><br>ERROR : IPOIB_UD : ipoib_ud_find_dev_by_dst:(ipoib_ud_arp.c):<br></div> ip_route_output_key(<a href="http://11.0.0.4/" target="_blank">11.0.0.4</a> <a href="http://11.0.0.4/" target="_blank"> <http://11.0.0.4></a> ) failed <div class="Ih2E3d"><br>ERROR : IPOIB_UD : ipoib_ud_find_dev_by_dst:(ipoib_ud_arp.c):<br></div> ip_route_output_key(<a href="http://11.0.0.4/" target="_blank">11.0.0.4</a> <a href="http://11.0.0.4/" target="_blank"> <http://11.0.0.4></a> ) failed <div class="Ih2E3d"><br> <br>How can I check the link on the device? Thanks in advance.<br></div></span></font></blockquote><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px"><br></span></font></div></blockquote> </div><br><br clear="all"><br> _______________________________________________<br>Lustre-discuss mailing list<br><a href="mailto:Lustre-discuss@clusterfs.com">Lustre-discuss@clusterfs.com</a><br>https://mail.clusterfs.com/mailman/listinfo/lustre-discuss<br></blockquote></div><br><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div>Aaron Knister</div><div>Associate Systems Analyst</div><div><div><div>Center for Ocean-Land-Atmosphere Studies</div></div><div></div></div><div><br class="khtml-block-placeholder"></div><div>(301) 595-7000</div><div><a href="mailto:aaron@iges.org">aaron@iges.org</a></div><div><br class="khtml-block-placeholder"></div><br class="Apple-interchange-newline"></span></div></span></div></span><br class="Apple-interchange-newline"> </div><br></body></html>