<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi, I think you might hit this: <a href="http://jira.whamcloud.com/browse/LU-952">http://jira.whamcloud.com/browse/LU-952</a> , you can find the patch from this ticket<div><br></div><div>Regards</div><div>Liang<br><div><div><br><div><div>On May 30, 2012, at 11:21 AM, huangql wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Dear all,<br><br>Recently we found the problem in OSS that some threads might be hung when the server got heavy IO load. In this case, some clients will be evicted or refused by some OSTs and got the error messages as following:<br><br>Server side:<br><br>May 30 11:06:31 boss07 kernel: Lustre: Service thread pid 8011 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. D<br>umping the stack trace for debugging purposes: May 30 11:06:31 boss07 kernel: Lustre: Skipped 1 previous similar message<br>May 30 11:06:31 boss07 kernel: Pid: 8011, comm: ll_ost_71 <br>May 30 11:06:31 boss07 kernel: <br>May 30 11:06:31 boss07 kernel: Call Trace:<br>May 30 11:06:31 boss07 kernel: [<ffffffff886f5d0e>] start_this_handle+0x301/0x3cb [jbd2]<br>May 30 11:06:31 boss07 kernel: [<ffffffff800a09ca>] autoremove_wake_function+0x0/0x2e<br>May 30 11:06:31 boss07 kernel: [<ffffffff886f5e83>] jbd2_journal_start+0xab/0xdf [jbd2]<br>May 30 11:06:31 boss07 kernel: [<ffffffff888ce9b2>] fsfilt_ldiskfs_start+0x4c2/0x590 [fsfilt_ldiskfs]<br>May 30 11:06:31 boss07 kernel: [<ffffffff88920551>] filter_version_get_check+0x91/0x2a0 [obdfilter]<br>May 30 11:06:31 boss07 kernel: [<ffffffff80036cf4>] __lookup_hash+0x61/0x12f<br>May 30 11:06:31 boss07 kernel: [<ffffffff8893108d>] filter_setattr_internal+0x90d/0x1de0 [obdfilter]<br>May 30 11:06:31 boss07 kernel: [<ffffffff800e859b>] lookup_one_len+0x53/0x61<br>May 30 11:06:31 boss07 kernel: [<ffffffff88925452>] filter_fid2dentry+0x512/0x740 [obdfilter]<br>May 30 11:06:31 boss07 kernel: [<ffffffff88924e27>] filter_fmd_get+0x2b7/0x320 [obdfilter]<br>May 30 11:06:31 boss07 kernel: [<ffffffff8003027b>] __up_write+0x27/0xf2<br>May 30 11:06:31 boss07 kernel: [<ffffffff88932721>] filter_setattr+0x1c1/0x3b0 [obdfilter]<br>May 30 11:06:31 boss07 kernel: [<ffffffff8882677a>] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc]<br>May 30 11:06:31 boss07 kernel: [<ffffffff8881e658>] ptlrpc_send_reply+0x5c8/0x5e0 [ptlrpc]<br>May 30 11:06:31 boss07 kernel: [<ffffffff88822b05>] lustre_msg_get_version+0x35/0xf0 [ptlrpc]<br>May 30 11:06:31 boss07 kernel: [<ffffffff888b0abb>] ost_handle+0x25db/0x55b0 [ost]<br>May 30 11:06:31 boss07 kernel: [<ffffffff80150d56>] __next_cpu+0x19/0x28<br>May 30 11:06:31 boss07 kernel: [<ffffffff800767ae>] smp_send_reschedule+0x4e/0x53<br>May 30 11:06:31 boss07 kernel: [<ffffffff8883215a>] ptlrpc_server_handle_request+0x97a/0xdf0 [ptlrpc]<br>May 30 11:06:31 boss07 kernel: [<ffffffff888328a8>] ptlrpc_wait_event+0x2d8/0x310 [ptlrpc]<br>May 30 11:06:31 boss07 kernel: [<ffffffff8008b3bd>] __wake_up_common+0x3e/0x68<br>May 30 11:06:31 boss07 kernel: [<ffffffff88833817>] ptlrpc_main+0xf37/0x10f0 [ptlrpc]<br>May 30 11:06:31 boss07 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11<br>May 30 11:06:31 boss07 kernel: [<ffffffff888328e0>] ptlrpc_main+0x0/0x10f0 [ptlrpc]<br>May 30 11:06:31 boss07 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11<br>May 30 11:06:31 boss07 kernel:<br>May 30 11:06:31 boss07 kernel: LustreError: dumping log to /tmp/lustre-log.1338347191.8011<br><br><br>Client side:<br><br>May 30 09:58:36 ccopt kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.123@tcp. The ost_connect operation failed with -16<br><br>When you got this error message, you failed to run "ls", "df" ,"vi", "touch" and so on, which affect us to do anything in the file system.<br>I think the ost_connect failure could report some error messages to users instead of causing any interactive actions stuck.<br><br>Could someone give us some advice or any suggestions to solve this problem?<br><br>Thank you very much in advance.<br><br><br>Best Regards<br>Qiulan Huang<br>2012-05-30<br>====================================================================<br>Computing center,the Institute of High Energy Physics, China<br>Huang, Qiulan Tel: (+86) 10 8823 6010-105<br>P.O. Box 918-7 Fax: (+86) 10 8823 6839<br>Beijing 100049 P.R. China Email: <a href="mailto:huangql@ihep.ac.cn">huangql@ihep.ac.cn</a><br>===================================================================<span class="Apple-tab-span" style="white-space:pre"> </span><br><br><br><br></div></blockquote></div><br></div></div></div></body></html>