<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=us-ascii" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 8.00.6001.18813"><LINK rel=stylesheet
href="BLOCKQUOTE{margin-Top: 0px; margin-Bottom: 0px; margin-Left: 2em}"></HEAD>
<BODY style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt">
<DIV><FONT size=2 face=Verdana>Hi, all</FONT></DIV>
<DIV> </DIV>
<DIV>Our system run well past two weeks, However, we found there are some
computing nodes which has so many threads named "ll_imp_inval", and the load
average of the clients(computing nodes) is up to 28. As a results, Users can't
submit jobs to the clients. I read the source file(import.c) and In my
opinion, when each ptlrpc-connect-import or ptlrpc-import-recovery, the
ll_imp_inval thread is triggered. So if the server or clients have something
wrong, the thread will not exit. Is it right?</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>
<DIV>we run 'ps -aux | grep ll_imp_inval' ,the results as follows:</DIV>
<DIV> </DIV>
<DIV>
<DIV>root 22568 0.0 0.0 0 0 ? D Aug13 0:00 [ll_imp_inval]</DIV>
<DIV>root 22569 0.0 0.0 0 0 ? D Aug13 0:00 [ll_imp_inval]</DIV></DIV>
<DIV>
<DIV>root 22570 0.0 0.0 0 0 ? D Aug13 0:00 [ll_imp_inval]</DIV>
<DIV>root 22571 0.0 0.0 0 0 ? D Aug13 0:00 [ll_imp_inval]</DIV></DIV>
<DIV><FONT size=2 face=Verdana>...</FONT></DIV></DIV>
<DIV> </DIV>
<DIV>We had check out the log, and found the main messages as follows,and in
other nodes we can get the client evicted messages:</DIV>
<DIV> </DIV>
<DIV>
<DIV>Aug 13 08:57:55 bws0211 kernel: Lustre: Request x5103879 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 08:57:55 bws0211 kernel: Lustre: Skipped 6 previous similar messages</DIV>
<DIV>Aug 13 08:58:34 bws0211 kernel: Lustre: testfs-OST0018-osc-f7dcfe00: Connection restored to service testfs-OST0018 using nid 192.168.50.79@tcp.</DIV>
<DIV>Aug 13 09:00:00 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 09:00:00 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 5 previous similar messages</DIV>
<DIV>Aug 13 09:00:00 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 09:00:00 bws0211 kernel: LustreError: Skipped 2 previous similar messages</DIV>
<DIV>Aug 13 09:00:00 bws0211 kernel: LustreError: 167-0: This client was evicted by testfs-OST001c; in progress operations using this service will fail.</DIV>
<DIV>Aug 13 09:00:00 bws0211 kernel: Lustre: testfs-OST001c-osc-f7dcfe00: Connection restored to service testfs-OST001c using nid 192.168.50.80@tcp.</DIV>
<DIV>Aug 13 09:00:00 bws0211 kernel: Lustre: Skipped 2 previous similar messages</DIV>
<DIV>Aug 13 09:02:05 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 09:02:05 bws0211 kernel: LustreError: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 09:06:15 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 09:06:15 bws0211 kernel: LustreError: Skipped 3 previous similar messages</DIV>
<DIV>Aug 13 09:10:25 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 26s</DIV>
<DIV>Aug 13 09:10:25 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 10 previous similar messages</DIV>
<DIV>Aug 13 09:12:30 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 09:12:30 bws0211 kernel: LustreError: Skipped 5 previous similar messages</DIV>
<DIV>Aug 13 09:20:50 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 09:20:50 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 09:22:55 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 09:22:55 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 09:31:15 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 6s</DIV>
<DIV>Aug 13 09:31:15 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 09:33:20 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 09:33:20 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 09:41:40 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 31s</DIV>
<DIV>Aug 13 09:41:40 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 09:43:45 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 09:43:45 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 09:52:05 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 09:52:05 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 09:54:10 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 09:54:10 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 10:02:30 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 6s</DIV>
<DIV>Aug 13 10:02:30 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 10:04:35 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 10:04:35 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 10:12:55 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 31s</DIV>
<DIV>Aug 13 10:12:55 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 10:15:00 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 10:15:00 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 10:23:20 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 10:23:20 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 10:25:25 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 10:25:25 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 10:33:45 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 6s</DIV>
<DIV>Aug 13 10:33:45 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 10:35:50 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 10:35:50 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 10:44:10 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 31s</DIV>
<DIV>Aug 13 10:44:10 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 10:46:15 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 10:46:15 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 10:54:35 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 10:54:35 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 10:56:40 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 10:56:40 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 11:05:00 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 6s</DIV>
<DIV>Aug 13 11:05:00 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 11:07:05 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 11:07:05 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 11:15:25 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 31s</DIV>
<DIV>Aug 13 11:15:25 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 11:17:30 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 11:17:30 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 11:25:50 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 11:25:50 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 11:27:55 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 11:27:55 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 11:36:15 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 6s</DIV>
<DIV>Aug 13 11:36:15 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 11:38:20 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 11:38:20 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 11:46:40 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 31s</DIV>
<DIV>Aug 13 11:46:40 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 11:48:45 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 11:48:45 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 11:57:05 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 11:57:05 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 12:01:15 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The obd_ping operation failed with -107</DIV>
<DIV>Aug 13 12:01:15 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 12:01:15 bws0211 kernel: Lustre: testfs-OST001c-osc-f7dcfe00: Connection to service testfs-OST001c via nid 192.168.50.80@tcp was lost; in progress operations using this service will wait for recovery to complete.</DIV>
<DIV>Aug 13 12:01:15 bws0211 kernel: Lustre: Skipped 2 previous similar messages</DIV>
<DIV>Aug 13 12:01:15 bws0211 kernel: LustreError: 167-0: This client was evicted by testfs-OST001c; in progress operations using this service will fail.</DIV>
<DIV>Aug 13 12:01:15 bws0211 kernel: Lustre: testfs-OST001c-osc-f7dcfe00: Connection restored to service testfs-OST001c using nid 192.168.50.80@tcp.</DIV>
<DIV>Aug 13 12:07:30 bws0211 kernel: Lustre: Request x5245865 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 12:07:30 bws0211 kernel: Lustre: Skipped 2 previous similar messages</DIV>
<DIV>Aug 13 12:09:35 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 12:09:35 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 3 previous similar messages</DIV>
<DIV>Aug 13 12:09:35 bws0211 kernel: Lustre: testfs-OST001c-osc-f7dcfe00: Connection to service testfs-OST001c via nid 192.168.50.80@tcp was lost; in progress operations using this service will wait for recovery to complete.</DIV>
<DIV>Aug 13 12:09:35 bws0211 kernel: LustreError: 167-0: This client was evicted by testfs-OST001c; in progress operations using this service will fail.</DIV>
<DIV>Aug 13 12:09:35 bws0211 kernel: Lustre: testfs-OST001c-osc-f7dcfe00: Connection restored to service testfs-OST001c using nid 192.168.50.80@tcp.</DIV>
<DIV>Aug 13 12:17:55 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The obd_ping operation failed with -107</DIV>
<DIV>Aug 13 12:17:55 bws0211 kernel: LustreError: Skipped 3 previous similar messages</DIV>
<DIV>Aug 13 12:17:55 bws0211 kernel: Lustre: testfs-OST001c-osc-f7dcfe00: Connection to service testfs-OST001c via nid 192.168.50.80@tcp was lost; in progress operations using this service will wait for recovery to complete.</DIV>
<DIV>Aug 13 12:17:55 bws0211 kernel: LustreError: 167-0: This client was evicted by testfs-OST001c; in progress operations using this service will fail.</DIV>
<DIV>Aug 13 12:17:55 bws0211 kernel: Lustre: testfs-OST001c-osc-f7dcfe00: Connection restored to service testfs-OST001c using nid 192.168.50.80@tcp.</DIV>
<DIV>Aug 13 12:20:00 bws0211 kernel: Lustre: Request x5254534 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 12:20:00 bws0211 kernel: Lustre: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 12:22:05 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 11s</DIV>
<DIV>Aug 13 12:22:05 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 3 previous similar messages</DIV>
<DIV>Aug 13 12:28:20 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 12:28:20 bws0211 kernel: LustreError: Skipped 6 previous similar messages</DIV>
<DIV>Aug 13 12:32:30 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 36s</DIV>
<DIV>Aug 13 12:32:30 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 12:38:45 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 12:38:45 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 12:42:55 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 12:42:55 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 12:49:10 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The obd_ping operation failed with -107</DIV>
<DIV>Aug 13 12:49:10 bws0211 kernel: LustreError: Skipped 5 previous similar messages</DIV>
<DIV>Aug 13 12:49:10 bws0211 kernel: Lustre: testfs-OST001c-osc-f7dcfe00: Connection to service testfs-OST001c via nid 192.168.50.80@tcp was lost; in progress operations using this service will wait for recovery to complete.</DIV>
<DIV>Aug 13 12:49:10 bws0211 kernel: LustreError: 167-0: This client was evicted by testfs-OST001c; in progress operations using this service will fail.</DIV>
<DIV>Aug 13 12:49:10 bws0211 kernel: Lustre: testfs-OST001c-osc-f7dcfe00: Connection restored to service testfs-OST001c using nid 192.168.50.80@tcp.</DIV>
<DIV>Aug 13 12:53:20 bws0211 kernel: Lustre: Request x5277671 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 12:53:20 bws0211 kernel: Lustre: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 12:55:25 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 12:55:25 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 3 previous similar messages</DIV>
<DIV>Aug 13 12:57:30 bws0211 kernel: Lustre: testfs-OST001c-osc-f7dcfe00: Connection to service testfs-OST001c via nid 192.168.50.80@tcp was lost; in progress operations using this service will wait for recovery to complete.</DIV>
<DIV>Aug 13 12:57:30 bws0211 kernel: LustreError: 167-0: This client was evicted by testfs-OST001c; in progress operations using this service will fail.</DIV>
<DIV>Aug 13 12:57:30 bws0211 kernel: Lustre: testfs-OST001c-osc-f7dcfe00: Connection restored to service testfs-OST001c using nid 192.168.50.80@tcp.</DIV>
<DIV>Aug 13 13:03:45 bws0211 kernel: Lustre: Request x5283965 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 13:03:45 bws0211 kernel: Lustre: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 13:05:50 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 13:05:50 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 1 previous similar message</DIV>
<DIV>Aug 13 13:05:50 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 13:05:50 bws0211 kernel: LustreError: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 13:16:15 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 26s</DIV>
<DIV>Aug 13 13:16:15 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 13:16:15 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 13:16:15 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 13:26:40 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 13:26:40 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 13:26:40 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 13:26:40 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 13:37:05 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 13:37:05 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 13:37:05 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 13:37:05 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 13:47:30 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 26s</DIV>
<DIV>Aug 13 13:47:30 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 13:47:30 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 13:47:30 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 13:57:55 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 13:57:55 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 13:57:55 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 13:57:55 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 14:08:20 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 6s</DIV>
<DIV>Aug 13 14:08:20 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 14:08:20 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 14:08:20 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 14:18:45 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 31s</DIV>
<DIV>Aug 13 14:18:45 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 14:18:45 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 14:18:45 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 14:29:10 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 51s</DIV>
<DIV>Aug 13 14:29:10 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 14:29:10 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 14:29:10 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 14:39:35 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 6s</DIV>
<DIV>Aug 13 14:39:35 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 14:39:35 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 14:39:35 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 14:50:00 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 31s</DIV>
<DIV>Aug 13 14:50:00 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 14:58:20 bws0211 kernel: Lustre: Request x5363068 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 14:58:20 bws0211 kernel: Lustre: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 14:58:20 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The obd_ping operation failed with -107</DIV>
<DIV>Aug 13 14:58:20 bws0211 kernel: LustreError: Skipped 9 previous similar messages</DIV>
<DIV>Aug 13 14:58:20 bws0211 kernel: Lustre: testfs-OST001c-osc-f7dcfe00: Connection to service testfs-OST001c via nid 192.168.50.80@tcp was lost; in progress operations using this service will wait for recovery to complete.</DIV>
<DIV>Aug 13 14:58:22 bws0211 kernel: Lustre: Request x5363070 sent from testfs-OST001c-osc-f7dcfe00 to NID 192.168.50.80@tcp 502s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 14:58:22 bws0211 kernel: Lustre: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 15:00:25 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 36s</DIV>
<DIV>Aug 13 15:00:25 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 1 previous similar message</DIV>
<DIV>Aug 13 15:00:27 bws0211 kernel: Lustre: Request x5364501 sent from testfs-OST001c-osc-f7dcfe00 to NID 192.168.50.80@tcp 502s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 15:02:34 bws0211 kernel: Lustre: Request x5365911 sent from testfs-OST001c-osc-f7dcfe00 to NID 192.168.50.80@tcp 504s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 15:04:35 bws0211 kernel: Lustre: Request x5367371 sent from testfs-OST001c-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 15:08:45 bws0211 kernel: Lustre: Request x5370730 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 15:10:51 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 41s</DIV>
<DIV>Aug 13 15:10:51 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 1 previous similar message</DIV>
<DIV>Aug 13 15:19:11 bws0211 kernel: Lustre: Request x5377804 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 15:19:11 bws0211 kernel: Lustre: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 15:21:16 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 46s</DIV>
<DIV>Aug 13 15:21:16 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 1 previous similar message</DIV>
<DIV>Aug 13 15:23:20 bws0211 kernel: Lustre: Request x5369047 sent from testfs-OST001c-osc-f7dcfe00 to NID 192.168.50.80@tcp 1500s ago has timed out (limit 1500s).</DIV>
<DIV>Aug 13 15:23:20 bws0211 kernel: Lustre: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 15:29:36 bws0211 kernel: Lustre: Request x5384919 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 15:31:40 bws0211 kernel: Lustre: Request x5386289 sent from testfs-OST001c-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 15:31:40 bws0211 kernel: Lustre: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 15:31:41 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001c-osc-f7dcfe00: tried all connections, increasing latency to 6s</DIV>
<DIV>Aug 13 15:31:41 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 3 previous similar messages</DIV>
<DIV>Aug 13 15:37:56 bws0211 kernel: Lustre: Request x5390694 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 15:40:01 bws0211 kernel: Lustre: Request x5393807 sent from testfs-OST001c-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 15:40:01 bws0211 kernel: Lustre: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 15:42:06 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001c-osc-f7dcfe00: tried all connections, increasing latency to 11s</DIV>
<DIV>Aug 13 15:42:06 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 2 previous similar messages</DIV>
<DIV>Aug 13 15:46:16 bws0211 kernel: Lustre: Request x5412456 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 15:50:26 bws0211 kernel: Lustre: Request x5425369 sent from testfs-OST001c-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 15:50:26 bws0211 kernel: Lustre: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 15:52:31 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001c-osc-f7dcfe00: tried all connections, increasing latency to 16s</DIV>
<DIV>Aug 13 15:52:31 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 2 previous similar messages</DIV>
<DIV>Aug 13 15:56:41 bws0211 kernel: Lustre: Request x5444459 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 16:02:56 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001c-osc-f7dcfe00: tried all connections, increasing latency to 21s</DIV>
<DIV>Aug 13 16:02:56 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 2 previous similar messages</DIV>
<DIV>Aug 13 16:05:01 bws0211 kernel: Lustre: Request x5469816 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 16:05:01 bws0211 kernel: Lustre: Skipped 2 previous similar messages</DIV>
<DIV>Aug 13 16:07:06 bws0211 kernel: LustreError: 4236:0:(import.c:756:ptlrpc_connect_interpret()) testfs-OST001a_UUID went back in time (transno 22127117 was previously committed, server now claims 22127112)! See https://bugzilla.clusterfs.com/long_list.cgi?buglist=9646</DIV>
<DIV>Aug 13 16:07:06 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 16:09:11 bws0211 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.50.80@tcp. The ost_connect operation failed with -19</DIV>
<DIV>Aug 13 16:13:21 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001b-osc-f7dcfe00: tried all connections, increasing latency to 16s</DIV>
<DIV>Aug 13 16:13:21 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 4 previous similar messages</DIV>
<DIV>Aug 13 16:21:41 bws0211 kernel: Lustre: Request x5519899 sent from testfs-OST001b-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 16:21:41 bws0211 kernel: Lustre: Skipped 2 previous similar messages</DIV>
<DIV>Aug 13 16:23:46 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001b-osc-f7dcfe00: tried all connections, increasing latency to 21s</DIV>
<DIV>Aug 13 16:23:46 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 1 previous similar message</DIV>
<DIV>Aug 13 16:32:06 bws0211 kernel: Lustre: Request x5501056 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 1500s ago has timed out (limit 1500s).</DIV>
<DIV>Aug 13 16:32:06 bws0211 kernel: Lustre: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 16:40:26 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 6s</DIV>
<DIV>Aug 13 16:40:26 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 3 previous similar messages</DIV>
<DIV>Aug 13 16:48:46 bws0211 kernel: Lustre: Request x5599100 sent from testfs-OST001a-osc-f7dcfe00 to NID 192.168.50.80@tcp 500s ago has timed out (limit 500s).</DIV>
<DIV>Aug 13 16:48:46 bws0211 kernel: Lustre: Skipped 5 previous similar messages</DIV>
<DIV>Aug 13 16:55:03 bws0211 kernel: Lustre: setting import testfs-OST001b_UUID INACTIVE by administrator request</DIV>
<DIV>Aug 13 16:55:03 bws0211 kernel: Lustre: Skipped 1 previous similar message</DIV>
<DIV>Aug 13 16:55:03 bws0211 kernel: Lustre: testfs-OST001b-osc-f7dcfe00.osc: set parameter active=0</DIV>
<DIV>Aug 13 16:55:03 bws0211 kernel: Lustre: Skipped 8 previous similar messages</DIV>
<DIV>Aug 13 16:57:06 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) testfs-OST001a-osc-f7dcfe00: tried all connections, increasing latency to 16s</DIV>
<DIV>Aug 13 16:57:06 bws0211 kernel: Lustre: 4237:0:(import.c:395:import_select_connection()) Skipped 5 previous similar messages</DIV>
<DIV>Aug 13 16:57:06 bws0211 kernel: LustreError: 167-0: This client was evicted by testfs-OST001a; in progress operations using this service will fail.</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>Thank you for your help in advance and I hope receive your letter as soon
as possible.<IMG src="cid:__0@Foxmail.net"></DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>Best wishes,</DIV>
<DIV>Sarea</DIV>
<DIV> </DIV>
<DIV> </DIV></DIV>
<DIV align=left><FONT color=#c0c0c0 size=2 face=Verdana>2009-08-17
</FONT></DIV><FONT size=2 face=Verdana>
<HR style="WIDTH: 122px; HEIGHT: 2px" align=left SIZE=2>
<DIV><FONT color=#c0c0c0 size=2 face=Verdana><SPAN>huangql</SPAN>
</FONT></DIV></FONT></BODY></HTML>