<div>Yes, the subnet manager has crashed. </div>
<div>I rebooted the infiniband switch, everything is fine now.</div>
<div> </div>
<div>Regards,</div>
<div>Changer<br><br> </div>
<div><span class="gmail_quote">On 1/9/08, <b class="gmail_sendername">Aaron Knister</b> <<a href="mailto:aaron@iges.org">aaron@iges.org</a>> wrote:</span>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div style="WORD-WRAP: break-word">
<div>I don't know if the voltaire IB stack is the same as OFED but I'm guessing it has a subnet manager. Check that. I've had similar issues when my subnet manager has crashed.</div><br>
<div>
<div><span class="e" id="q_1175f0d5a81a86ab_1">
<div>On Jan 9, 2008, at 3:08 AM, Changer Van wrote:</div><br></span></div>
<blockquote type="cite">
<div><span class="e" id="q_1175f0d5a81a86ab_3">
<div>Network connection is down. I can not ping the other nodes.</div>
<div>I ran the vstat command and found one of the port_state is 'port_initialize'.</div>
<div>What does 'port_initialize' mean? Dose it mean my ib card is broken?</div>
<div> </div>
<div>1 HCA found:<br> hca_id=InfiniHost_III_Ex0<br> pci_location={BUS=0x20,DEV/FUNC=0x00}<br> vendor_id=0x02C9<br> vendor_part_id=0x6282<br> hw_ver=0xA0<br> fw_ver=5.1.400<br> PSID=MT_0140000001
<br> num_phys_ports=2<br> port=1<br> port_state=PORT_INITIALIZE<br> sm_lid=0x0000<br> port_lid=0x0000<br> port_lmc=0x00<br> max_mtu=2048
</div>
<div> port=2<br> port_state=PORT_DOWN<br> sm_lid=0x0000<br> port_lid=0x0000<br> port_lmc=0x00<br> max_mtu=2048</div>
<div>-- <br>Regards,<br>Changer <br><br> </div>
<div class="gmail_quote">On Jan 9, 2008 3:27 AM, Klaus Steden <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:klaus.steden@thomson.net" target="_blank">klaus.steden@thomson.net</a>> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px"><br>If you're using IPoIB, you can use standard TCP/IP diagnostic tools the same way you would on an Ethernet link (ifconfig, ping, traceroute, telnet, etc.)
<br><br>If you're using a copper-to-optical converter in your data path as well, the Emcore MIAs have link lights on them which will tell you if a physical link is present (check the documentation). I know with STP InfiniBand connectors, there is some ambiguity about terminology with some vendors and manufacturers, and the fibre arrangement doesn't provide a lot of wiggle room.
<br><br>Klaus<br><br>On 1/7/08 7:56 PM, "Changer Van" <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:changerv@gmail.com" target="_blank">changerv@gmail.com</a>>did etch on stone tablets:
<br><br></span></font>
<blockquote>
<div><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px"><br><br>On Jan 8, 2008 1:35 AM, Isaac Huang <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:He.Huang@sun.com" target="_blank">
He.Huang@sun.com</a>> wrote:<br></span></font>
<blockquote><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px">On Mon, Jan 07, 2008 at 06:20:52PM +0800, Changer Van wrote:<br>> ......<br>> # dmesg<br>><br>> LustreError: 4273:0:(viblnd.c
:1890:kibnal_startup())<br>><br>> Can't find an active port on InfiniHost_III_Ex0<br><br>It meant that viblnd couldn't find a port whose link state was active <br>on the hca InfiniHost_III_Ex0,
i.e . no link on the device was usable.<br><br>Was there any other error messages from viblnd before this one? <br></span></font></blockquote><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px">There was no error messages but a related message
<br>like 'ADDRCONF(NETDEV_UP):ipoib0: link is not ready'.<br></span></font>
<blockquote><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px">Did you see this problem on just one node?<br></span></font></blockquote></div><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px">
<div>There are four nodes which can not mount the lustre system.<br>The other nodes can mount the lustre but got the following error messages:<br> <br># dmesg<br>divert: not allocating divert_blk for non-ethernet device ipoib0
<br>ERROR : IPOIB_UD : ipoib_ud_find_dev_by_dst:(ipoib_ud_arp.c):<br> </div> ip_route_output_key(<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://127.0.0.1/" target="_blank">127.0.0.1</a> <a onclick="return top.js.OpenExtLink(window,event,this)" href="http://127.0.0.1/" target="_blank">
<http://127.0.0.1> </a>) failed
<div><br>new: ipoib_allow_arp_joins: 1<br>ERROR : IPOIB_UD : ipoib_ud_find_dev_by_dst:(ipoib_ud_arp.c):<br> </div> ip_route_output_key(<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://11.0.0.4/" target="_blank">
11.0.0.4</a> <a onclick="return top.js.OpenExtLink(window,event,this)" href="http://11.0.0.4/" target="_blank"><http://11.0.0.4></a> ) failed
<div><br>ERROR : IPOIB_UD : ipoib_ud_find_dev_by_dst:(ipoib_ud_arp.c):<br> </div> ip_route_output_key(<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://11.0.0.4/" target="_blank">11.0.0.4</a> <a onclick="return top.js.OpenExtLink(window,event,this)" href="http://11.0.0.4/" target="_blank">
<http://11.0.0.4></a> ) failed
<div><br>ERROR : IPOIB_UD : ipoib_ud_find_dev_by_dst:(ipoib_ud_arp.c):<br> </div> ip_route_output_key(<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://11.0.0.4/" target="_blank">11.0.0.4</a> <a onclick="return top.js.OpenExtLink(window,event,this)" href="http://11.0.0.4/" target="_blank">
<http://11.0.0.4></a> ) failed
<div><br> <br>How can I check the link on the device? Thanks in advance.<br> </div></span></font></blockquote><font face="Verdana, Helvetica, Arial"><span style="FONT-SIZE: 12px"><br></span></font></div></blockquote></div>
<br><br clear="all"><br></span></div>_______________________________________________<br>Lustre-discuss mailing list<br><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:Lustre-discuss@clusterfs.com" target="_blank">
Lustre-discuss@clusterfs.com</a><br><a onclick="return top.js.OpenExtLink(window,event,this)" href="https://mail.clusterfs.com/mailman/listinfo/lustre-discuss" target="_blank">https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
</a><br></blockquote></div><br>
<div><span style="WORD-SPACING: 0px; FONT: 12px Helvetica; TEXT-TRANSFORM: none; COLOR: rgb(0,0,0); TEXT-INDENT: 0px; WHITE-SPACE: normal; LETTER-SPACING: normal; BORDER-COLLAPSE: separate">
<div style="WORD-WRAP: break-word"><span style="WORD-SPACING: 0px; FONT: 12px Helvetica; TEXT-TRANSFORM: none; COLOR: rgb(0,0,0); TEXT-INDENT: 0px; WHITE-SPACE: normal; LETTER-SPACING: normal; BORDER-COLLAPSE: separate">
<div style="WORD-WRAP: break-word"><span style="WORD-SPACING: 0px; FONT: 12px Helvetica; TEXT-TRANSFORM: none; COLOR: rgb(0,0,0); TEXT-INDENT: 0px; WHITE-SPACE: normal; LETTER-SPACING: normal; BORDER-COLLAPSE: separate">
<div>Aaron Knister</div>
<div>Associate Systems Analyst</div>
<div>
<div>
<div>Center for Ocean-Land-Atmosphere Studies</div></div>
<div></div></div>
<div><br> </div>
<div>(301) 595-7000</div>
<div><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:aaron@iges.org" target="_blank">aaron@iges.org</a></div>
<div><br> </div><br></span></div></span></div></span><br> </div><br> </div></blockquote></div><br><br clear="all"><br>-- <br>Regards,<br>Changer