<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>I made this change and clients are still being evicted. This is very frustrating. It happens over tcp and infiniband. My timeout is 1000. Anybody know why don't the clients reconnect?</div><br><div><div>On Mar 4, 2008, at 3:55 PM, Aaron S. Knister wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="font-family: 'Times New Roman'; font-size: 12pt; color: rgb(0, 0, 0); ">I think I tried that before and it didn't help, but I will try it again. Thanks for the suggestion.<br><br>-Aaron<br><br>----- Original Message -----<br>From: "Charles Taylor" <<a href="mailto:taylor@hpc.ufl.edu">taylor@hpc.ufl.edu</a>><br>To: "Aaron S. Knister" <<a href="mailto:aaron@iges.org">aaron@iges.org</a>><br>Cc: "lustre-discuss" <<a href="mailto:lustre-discuss@clusterfs.com">lustre-discuss@clusterfs.com</a>>, "Thomas Wakefield" <<a href="mailto:twake@cola.iges.org">twake@cola.iges.org</a>><br>Sent: Tuesday, March 4, 2008 3:41:04 PM GMT -05:00 US/Canada Eastern<br>Subject: Re: [Lustre-discuss] Cannot send after transport endpoint shutdown (-108)<br><br>We've seen this before as well. Our experience is that the <br>obd_timeout is far too small for large clusters (ours is 400+ <br>nodes) and the only way we avoid these errors is by setting it to <br>1000 which seems high to us but appears to work and puts an end to <br>the transport endpoint shutdowns.<br><br>On the MDS....<br><br>lctl conf_param srn.sys.timeout=1000<br><br>You may have to do this on the OSS's as well unless you restart the <br>OSS's but I could be wrong on that. You should check it everywhere <br>with...<br><br>cat /proc/sys/lustre/timeout<br><br><br>On Mar 4, 2008, at 3:31 PM, Aaron S. Knister wrote:<br><br>> This morning I've had both my infiniband and tcp lustre clients <br>> hiccup. They are evicted from the server presumably as a result of <br>> their high load and consequent timeouts. My question is- why don't <br>> the clients re-connect. The infiniband and tcp clients both give <br>> the following message when I type "df" - Cannot send after <br>> transport endpoint shutdown (-108). I've been battling with this on <br>> and off now for a few months. I've upgraded my infiniband switch <br>> firmware, all the clients and servers are running the latest <br>> version of lustre and the lustre patched kernel. Any ideas?<br>><br>> -Aaron<br>> _______________________________________________<br>> Lustre-discuss mailing list<br>><span class="Apple-converted-space"> </span><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>><span class="Apple-converted-space"> </span><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><span class="Apple-converted-space"> </span><br><br></div></span></blockquote></div><br><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div>Aaron Knister</div><div>Associate Systems Analyst</div><div><div><div>Center for Ocean-Land-Atmosphere Studies</div></div><div></div></div><div><br class="khtml-block-placeholder"></div><div>(301) 595-7000</div><div><a href="mailto:aaron@iges.org">aaron@iges.org</a></div><div><br class="khtml-block-placeholder"></div><br class="Apple-interchange-newline"></span></div></span></div></span><br class="Apple-interchange-newline"> </div><br></body></html>