<html><head><style type='text/css'>p { margin: 0; }</style><style type='text/css'>body { font-family: 'Times New Roman'; font-size: 12pt; color: #000000}</style></head><body>I think I tried that before and it didn't help, but I will try it again. Thanks for the suggestion.<br><br>-Aaron<br><br>----- Original Message -----<br>From: "Charles Taylor" <taylor@hpc.ufl.edu><br>To: "Aaron S. Knister" <aaron@iges.org><br>Cc: "lustre-discuss" <lustre-discuss@clusterfs.com>, "Thomas Wakefield" <twake@cola.iges.org><br>Sent: Tuesday, March 4, 2008 3:41:04 PM GMT -05:00 US/Canada Eastern<br>Subject: Re: [Lustre-discuss] Cannot send after transport endpoint shutdown (-108)<br><br>We've seen this before as well. Our experience is that the <br>obd_timeout is far too small for large clusters (ours is 400+ <br>nodes) and the only way we avoid these errors is by setting it to <br>1000 which seems high to us but appears to work and puts an end to <br>the transport endpoint shutdowns.<br><br>On the MDS....<br><br>lctl conf_param srn.sys.timeout=1000<br><br>You may have to do this on the OSS's as well unless you restart the <br>OSS's but I could be wrong on that. You should check it everywhere <br>with...<br><br>cat /proc/sys/lustre/timeout<br><br><br>On Mar 4, 2008, at 3:31 PM, Aaron S. Knister wrote:<br><br>> This morning I've had both my infiniband and tcp lustre clients <br>> hiccup. They are evicted from the server presumably as a result of <br>> their high load and consequent timeouts. My question is- why don't <br>> the clients re-connect. The infiniband and tcp clients both give <br>> the following message when I type "df" - Cannot send after <br>> transport endpoint shutdown (-108). I've been battling with this on <br>> and off now for a few months. I've upgraded my infiniband switch <br>> firmware, all the clients and servers are running the latest <br>> version of lustre and the lustre patched kernel. Any ideas?<br>><br>> -Aaron<br>> _______________________________________________<br>> Lustre-discuss mailing list<br>> Lustre-discuss@lists.lustre.org<br>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
<br><br></body></html>