<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Whoops, I meant to include the mount-time error message....<div><br></div><div><br></div><div><span class="Apple-style-span" style="font-family: Times; font-size: 16px; "><pre id="comment_text_2"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">/etc/init.d/lustre-client start
IB HCA detected - will try to sleep until link state becomes ACTIVE
State becomes ACTIVE
Loading Lustre lnet module with option networks=o2ib: [ OK ]
Loading Lustre kernel module: [ OK ]
mount -t lustre 10.13.24.40@o2ib:/ufhpc /ufhpc/scratch:
mount.lustre: mount 10.13.24.40@o2ib:/ufhpc at /ufhpc/scratch failed: Cannot
send after transport endpoint shutdown
[FAILED]
Error: Failed to mount 10.13.24.40@o2ib:/ufhpc
mount -t lustre 10.13.24.90@o2ib:/crn /crn/scratch: mount.lustre: mount
10.13.24.90@o2ib:/crn at /crn/scratch failed: Cannot send after transport
endpoint shutdown
[FAILED]
Error: Failed to mount 10.13.24.90@o2ib:/crn
mount -t lustre 10.13.24.85@o2ib:/hpcdata /ufhpc/hpcdata: mount.lustre: mount
10.13.24.85@o2ib:/hpcdata at /ufhpc/hpcdata failed: Cannot send after transport
endpoint shutdown
[FAILED]
Error: Failed to mount 10.13.24.85@o2ib:/hpcdata</span></font></pre><pre id="comment_text_2"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">Charlie Taylor</span></font></pre><pre id="comment_text_2"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">UF HPC Center</span></font></pre></span></div><div><br></div><div><br><div><div>On May 27, 2008, at 9:46 AM, Charles Taylor wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><br><br>We have a few nodes that locked up due to memory oversubscription. <br>After rebooting, they can no longer communicate with our any of our <br>three MDSs over IB and, consequently, we cannot mount our Lustre <br>1.6.4.2 file systems on these nodes any longer. All other <br>communication via the IB port (ipoib for pings, ssh, etc) seems <br>fine. If we re-cable the node to use the second IB port, <br>communication is re-established and we can mount the file system. In <br>other words, by switching to the second IB port, we can once again <br>communicate with the MDSs and everything works as expected. Note <br>that this is only for a few nodes (out of 400) that seem to have <br>gotten in a bad state with regard to lustre.<br><br>Relevant info:<br><br>Lustre 1.6.4.2<br><br>CentOS 4.5 w/ updated kernel.<br><br>Linux r5b-s30.ufhpc 2.6.18-8.1.14.el5.L-1642 #1 SMP Wed Feb 20 <br>10:59:48 EST 2008 x86_64 x86_64 x86_64 GNU/Linux<br><br>OFED 1.2<br><br>HCA #0: MT25208 Tavor Compat, Lion Cub, revision A0<br> Primary image is valid, unknown source<br> Secondary image is valid, unknown source<br><br> Vital Product Data<br> Product Name: Lion cub<br> P/N: MHEA28-1TC<br> E/C: A2<br> S/N: MT0637X00650<br> Freq/Power: PCIe x8<br> Checksum: Ok<br> Date Code: N/A<br><br>We don't know because we have not tried rebooting the MDS's yet (kind <br>of painful) but I'm guessing that if we rebooted them, the issue would <br>go away. I suppose it could be a problem at the IB layer (LID re- <br>assignment or some such) but since Lustre is the only app that seems <br>to be manifesting the issue that seems unlikely. I'm just wondering <br>if anyone else has encountered this and might know of a way to clear <br>it out (some obscure lnet command) without rebooting the MDS.<br><br><br>Charlie Taylor<br>UF HPC Center<br>_______________________________________________<br>Lustre-discuss mailing list<br><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>http://lists.lustre.org/mailman/listinfo/lustre-discuss<br></blockquote></div><br></div></body></html>