Feb 10 17:11:18 edclxs204 kernel: Lustre: 8573:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1357189922740381 sent from lustre-OST0005 to NID 192.168.0.100@tcp 7s ago has timed out (7s prior to deadline).
Feb 10 17:11:18 edclxs204 kernel:   req@ffff810346032c00 x1357189922740381/t0 o106->@NET_0x20000c0a80064_UUID:15/16 lens 296/424 e 0 to 1 dl 1297379478 ref 1 fl Rpc:/0/0 rc 0/0
Feb 10 17:11:25 edclxs204 kernel: Lustre: 8573:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1357189922740381 sent from lustre-OST0005 to NID 192.168.0.100@tcp 7s ago has timed out (7s prior to deadline).
Feb 10 17:11:25 edclxs204 kernel:   req@ffff810346032c00 x1357189922740381/t0 o106->@NET_0x20000c0a80064_UUID:15/16 lens 296/424 e 0 to 1 dl 1297379485 ref 1 fl Rpc:/2/0 rc 0/0
Feb 10 17:11:27 edclxs204 kernel: Lustre: 7988:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1357189922740607 sent from lustre-OST0004 to NID 192.168.0.100@tcp 7s ago has timed out (7s prior to deadline).
Feb 10 17:11:27 edclxs204 kernel:   req@ffff810344d2f000 x1357189922740607/t0 o104->@NET_0x20000c0a80064_UUID:15/16 lens 296/384 e 0 to 1 dl 1297379487 ref 1 fl Rpc:N/0/0 rc 0/0
Feb 10 17:11:27 edclxs204 kernel: LustreError: 138-a: lustre-OST0004: A client on nid 192.168.0.100@tcp was evicted due to a lock blocking callback to 192.168.0.100@tcp timed out: rc -107
Feb 10 17:11:27 edclxs204 kernel: LustreError: Skipped 5 previous similar messages
Feb 10 17:11:27 edclxs204 kernel: LustreError: 138-a: lustre-OST0005: A client on nid 192.168.0.100@tcp was evicted due to a lock blocking callback to 192.168.0.100@tcp timed out: rc -107
Feb 10 17:11:32 edclxs204 kernel: Lustre: 8573:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1357189922740381 sent from lustre-OST0005 to NID 192.168.0.100@tcp 7s ago has timed out (7s prior to deadline).
Feb 10 17:11:32 edclxs204 kernel:   req@ffff810346032c00 x1357189922740381/t0 o106->@NET_0x20000c0a80064_UUID:15/16 lens 296/424 e 0 to 1 dl 1297379492 ref 1 fl Rpc:/2/0 rc 0/0
Feb 10 17:11:32 edclxs204 kernel: Lustre: 8573:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Feb 10 17:11:39 edclxs204 kernel: Lustre: 8573:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1357189922740381 sent from lustre-OST0005 to NID 192.168.0.100@tcp 7s ago has timed out (7s prior to deadline).
Feb 10 17:11:39 edclxs204 kernel:   req@ffff810346032c00 x1357189922740381/t0 o106->@NET_0x20000c0a80064_UUID:15/16 lens 296/424 e 0 to 1 dl 1297379499 ref 1 fl Rpc:/2/0 rc 0/0
Feb 10 17:11:39 edclxs204 kernel: Lustre: 8573:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Feb 10 17:11:45 edclxs204 kernel: Lustre: 8583:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1357189922741262 sent from lustre-OST0005 to NID 192.168.0.100@tcp 7s ago has timed out (7s prior to deadline).
Feb 10 17:11:45 edclxs204 kernel:   req@ffff81056dea2400 x1357189922741262/t0 o106->@NET_0x20000c0a80064_UUID:15/16 lens 296/424 e 0 to 1 dl 1297379505 ref 1 fl Rpc:/0/0 rc 0/0
Feb 10 17:11:45 edclxs204 kernel: Lustre: 8583:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Feb 10 17:11:53 edclxs204 kernel: Lustre: 8354:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1357189922741066 sent from lustre-OST0005 to NID 192.168.0.100@tcp 7s ago has timed out (7s prior to deadline).
Feb 10 17:11:53 edclxs204 kernel:   req@ffff81034fe65c00 x1357189922741066/t0 o106->@NET_0x20000c0a80064_UUID:15/16 lens 296/424 e 0 to 1 dl 1297379513 ref 1 fl Rpc:/2/0 rc 0/0
Feb 10 17:11:53 edclxs204 kernel: Lustre: 8354:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Feb 10 17:12:09 edclxs204 kernel: Lustre: 7880:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1357189922742517 sent from lustre-OST0005 to NID 192.168.0.100@tcp 0s ago has failed due to network error (7s prior to deadline).
Feb 10 17:12:09 edclxs204 kernel:   req@ffff8103a90bd800 x1357189922742517/t0 o106->@NET_0x20000c0a80064_UUID:15/16 lens 296/424 e 0 to 1 dl 1297379536 ref 1 fl Rpc:/2/0 rc 0/0
Feb 10 17:12:09 edclxs204 kernel: Lustre: 7880:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 1100924 previous similar messages
Feb 10 17:12:41 edclxs204 kernel: Lustre: 8354:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1357189922741066 sent from lustre-OST0005 to NID 192.168.0.100@tcp 0s ago has failed due to network error (7s prior to deadline).
Feb 10 17:12:41 edclxs204 kernel:   req@ffff81034fe65c00 x1357189922741066/t0 o106->@NET_0x20000c0a80064_UUID:15/16 lens 296/424 e 0 to 1 dl 1297379568 ref 1 fl Rpc:/2/0 rc 0/0
Feb 10 17:12:41 edclxs204 kernel: Lustre: 8354:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 5001139 previous similar messages
Feb 10 17:13:45 edclxs204 kernel: Lustre: 7947:0:(client.c:1476:ptlrpc_expire_one_request()) @@@ Request x1357189922741899 sent from lustre-OST0005 to NID 192.168.0.100@tcp 0s ago has failed due to network error (7s prior to deadline).
Feb 10 17:13:45 edclxs204 kernel:   req@ffff81037590d400 x1357189922741899/t0 o106->@NET_0x20000c0a80064_UUID:15/16 lens 296/424 e 0 to 1 dl 1297379632 ref 1 fl Rpc:/2/0 rc 0/0
Feb 10 17:13:45 edclxs204 kernel: Lustre: 7947:0:(client.c:1476:ptlrpc_expire_one_request()) Skipped 10023404 previous similar messages
Feb 10 17:14:31 edclxs204 kernel: Lustre: Service thread pid 8573 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 10 17:14:31 edclxs204 kernel: Pid: 8573, comm: ll_ost_371
Feb 10 17:14:31 edclxs204 kernel: 
Feb 10 17:14:31 edclxs204 kernel: Call Trace:
Feb 10 17:14:31 edclxs204 kernel:  [<ffffffff8871673a>] _debug_req+0x4ba/0x4d0 [ptlrpc]
Feb 10 17:14:31 edclxs204 kernel:  [<ffffffff8008e8d9>] __cond_resched+0x1c/0x44
Feb 10 17:14:31 edclxs204 kernel:  [<ffffffff800630cb>] cond_resched+0x37/0x42
Feb 10 17:14:31 edclxs204 kernel:  [<ffffffff800db6e3>] __kmalloc+0x3b/0x9f
Feb 10 17:14:31 edclxs204 kernel:  [<ffffffff885ae378>] cfs_alloc+0x68/0xc0 [libcfs]
Feb 10 17:14:31 edclxs204 kernel:  [<ffffffff885fd086>] LNetMEAttach+0x226/0x510 [lnet]
Feb 10 17:14:31 edclxs204 kernel:  [<ffffffff887134b4>] ptl_send_rpc+0x624/0xe00 [ptlrpc]
Feb 10 17:14:31 edclxs204 kernel:  [<ffffffff88703eac>] ptlrpc_unregister_reply+0x13c/0x9c0 [ptlrpc]
Feb 10 17:14:31 edclxs204 kernel:  [<ffffffff887061a6>] ptlrpc_check_reply+0x1c6/0x610 [ptlrpc]
Feb 10 17:14:31 edclxs204 kernel:  [<ffffffff8870d675>] ptlrpc_queue_wait+0x8d5/0x1700 [ptlrpc]
Feb 10 17:14:31 edclxs204 kernel:  [<ffffffff8871bd15>] lustre_msg_set_opc+0x45/0x120 [ptlrpc]
Feb 10 17:14:31 edclxs204 kernel:  [<ffffffff88704de5>] ptlrpc_at_set_req_timeout+0x85/0xd0 [ptlrpc]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff88705615>] ptlrpc_prep_req_pool+0x625/0x6c0 [ptlrpc]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff8008d07b>] default_wake_function+0x0/0xe
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff886f4e56>] ldlm_server_glimpse_ast+0x266/0x3b0 [ptlrpc]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff88701b33>] interval_iterate_reverse+0x73/0x240 [ptlrpc]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff886e99e0>] ldlm_process_extent_lock+0x0/0xad0 [ptlrpc]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff889f95aa>] filter_intent_policy+0x65a/0x760 [obdfilter]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff886063a0>] LNetPut+0x710/0x820 [lnet]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff886d5eb6>] ldlm_lock_enqueue+0x186/0xb20 [ptlrpc]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff886d2380>] ldlm_lock_create+0x540/0x9f0 [ptlrpc]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff889a65f0>] ost_blocking_ast+0x0/0x9b0 [ost]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff886f7849>] ldlm_handle_enqueue+0xbf9/0x1210 [ptlrpc]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff88715a78>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff889ae4b3>] ost_handle+0x4fe3/0x55b0 [ost]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff800dbeec>] __drain_alien_cache+0x51/0x66
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff88722c37>] ptlrpc_server_handle_request+0xaa7/0x1150 [ptlrpc]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff800470f3>] try_to_wake_up+0x472/0x484
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff8003da91>] lock_timer_base+0x1b/0x3c
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff8008b4a5>] __wake_up_common+0x3e/0x68
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff88726698>] ptlrpc_main+0x1258/0x1420 [ptlrpc]
Feb 10 17:14:32 edclxs204 kernel:  [<ffffffff8008d07b>] default_wake_function+0x0/0xe
Feb 10 17:14:33 edclxs204 kernel:  [<ffffffff800b7a9c>] audit_syscall_exit+0x336/0x362
Feb 10 17:14:33 edclxs204 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Feb 10 17:14:33 edclxs204 kernel:  [<ffffffff88725440>] ptlrpc_main+0x0/0x1420 [ptlrpc]
Feb 10 17:14:33 edclxs204 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Feb 10 17:14:33 edclxs204 kernel: 
Feb 10 17:14:33 edclxs204 kernel: LustreError: dumping log to /tmp/lustre-log.1297379671.8573
Feb 10 17:14:40 edclxs204 kernel: Lustre: Service thread pid 7988 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 10 17:14:40 edclxs204 kernel: Pid: 7988, comm: ll_ost_io_17
Feb 10 17:14:40 edclxs204 kernel: 
Feb 10 17:14:40 edclxs204 kernel: Call Trace:
Feb 10 17:14:40 edclxs204 kernel:  [<ffffffff886efc10>] ldlm_expired_completion_wait+0x0/0x250 [ptlrpc]
Feb 10 17:14:40 edclxs204 kernel:  [<ffffffff800638ab>] schedule_timeout+0x8a/0xad
Feb 10 17:14:40 edclxs204 kernel:  [<ffffffff80097d9f>] process_timeout+0x0/0x5
Feb 10 17:14:40 edclxs204 kernel:  [<ffffffff886f1775>] ldlm_completion_ast+0x4d5/0x880 [ptlrpc]
Feb 10 17:14:40 edclxs204 kernel:  [<ffffffff886d6709>] ldlm_lock_enqueue+0x9d9/0xb20 [ptlrpc]
Feb 10 17:14:40 edclxs204 kernel:  [<ffffffff8008d07b>] default_wake_function+0x0/0xe
Feb 10 17:14:40 edclxs204 kernel:  [<ffffffff886d1b6a>] ldlm_lock_addref_internal_nolock+0x3a/0x90 [ptlrpc]
Feb 10 17:14:40 edclxs204 kernel:  [<ffffffff886f02cb>] ldlm_cli_enqueue_local+0x46b/0x520 [ptlrpc]
Feb 10 17:14:40 edclxs204 kernel:  [<ffffffff889ef4e1>] filter_destroy+0x681/0x1fb0 [obdfilter]
Feb 10 17:14:40 edclxs204 kernel:  [<ffffffff886edbc0>] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc]
Feb 10 17:14:40 edclxs204 kernel:  [<ffffffff886f12a0>] ldlm_completion_ast+0x0/0x880 [ptlrpc]
Feb 10 17:14:40 edclxs204 kernel:  [<ffffffff88716924>] lustre_msg_add_version+0x34/0x110 [ptlrpc]
Feb 10 17:14:41 edclxs204 kernel:  [<ffffffff8871962a>] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc]
Feb 10 17:14:41 edclxs204 kernel:  [<ffffffff886d4efc>] __ldlm_handle2lock+0x8c/0x360 [ptlrpc]
Feb 10 17:14:41 edclxs204 kernel:  [<ffffffff88719739>] lustre_pack_reply+0x29/0xb0 [ptlrpc]
Feb 10 17:14:41 edclxs204 kernel:  [<ffffffff889a1050>] ost_destroy+0x660/0x790 [ost]
Feb 10 17:14:41 edclxs204 kernel:  [<ffffffff88715a78>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Feb 10 17:14:42 edclxs204 kernel:  [<ffffffff887158c5>] lustre_msg_get_opc+0x35/0xf0 [ptlrpc]
Feb 10 17:14:42 edclxs204 kernel:  [<ffffffff889aaa26>] ost_handle+0x1556/0x55b0 [ost]
Feb 10 17:14:42 edclxs204 kernel:  [<ffffffff8019d806>] acpi_processor_idle_simple+0x17d/0x30e
Feb 10 17:14:42 edclxs204 kernel:  [<ffffffff886dae29>] ldlm_resource_get+0x1d9/0xa50 [ptlrpc]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff88722c37>] ptlrpc_server_handle_request+0xaa7/0x1150 [ptlrpc]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff800470f3>] try_to_wake_up+0x472/0x484
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff80062ff8>] thread_return+0x62/0xfe
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff8008b4a5>] __wake_up_common+0x3e/0x68
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff88726698>] ptlrpc_main+0x1258/0x1420 [ptlrpc]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff8008d07b>] default_wake_function+0x0/0xe
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff800b7a9c>] audit_syscall_exit+0x336/0x362
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff88725440>] ptlrpc_main+0x0/0x1420 [ptlrpc]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Feb 10 17:14:43 edclxs204 kernel: 
Feb 10 17:14:43 edclxs204 kernel: LustreError: dumping log to /tmp/lustre-log.1297379680.7988
Feb 10 17:14:43 edclxs204 kernel: Pid: 8062, comm: ll_ost_io_91
Feb 10 17:14:43 edclxs204 kernel: 
Feb 10 17:14:43 edclxs204 kernel: Call Trace:
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff886efc10>] ldlm_expired_completion_wait+0x0/0x250 [ptlrpc]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff800638ab>] schedule_timeout+0x8a/0xad
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff80097d9f>] process_timeout+0x0/0x5
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff886f1775>] ldlm_completion_ast+0x4d5/0x880 [ptlrpc]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff886d6709>] ldlm_lock_enqueue+0x9d9/0xb20 [ptlrpc]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff8008d07b>] default_wake_function+0x0/0xe
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff886d1b6a>] ldlm_lock_addref_internal_nolock+0x3a/0x90 [ptlrpc]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff886f02cb>] ldlm_cli_enqueue_local+0x46b/0x520 [ptlrpc]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff889ef4e1>] filter_destroy+0x681/0x1fb0 [obdfilter]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff886edbc0>] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff886f12a0>] ldlm_completion_ast+0x0/0x880 [ptlrpc]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff88716924>] lustre_msg_add_version+0x34/0x110 [ptlrpc]
Feb 10 17:14:43 edclxs204 kernel:  [<ffffffff8871962a>] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff886d4efc>] __ldlm_handle2lock+0x8c/0x360 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff88719739>] lustre_pack_reply+0x29/0xb0 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff889a1050>] ost_destroy+0x660/0x790 [ost]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff88715a78>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff887158c5>] lustre_msg_get_opc+0x35/0xf0 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff889aaa26>] ost_handle+0x1556/0x55b0 [ost]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff886dae29>] ldlm_resource_get+0x1d9/0xa50 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff88722c37>] ptlrpc_server_handle_request+0xaa7/0x1150 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff800470f3>] try_to_wake_up+0x472/0x484
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff80062ff8>] thread_return+0x62/0xfe
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff8008b4a5>] __wake_up_common+0x3e/0x68
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff88726698>] ptlrpc_main+0x1258/0x1420 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff8008d07b>] default_wake_function+0x0/0xe
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff800b7a9c>] audit_syscall_exit+0x336/0x362
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff88725440>] ptlrpc_main+0x0/0x1420 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Feb 10 17:14:44 edclxs204 kernel: 
Feb 10 17:14:44 edclxs204 kernel: Pid: 7992, comm: ll_ost_io_21
Feb 10 17:14:44 edclxs204 kernel: 
Feb 10 17:14:44 edclxs204 kernel: Call Trace:
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff886efc10>] ldlm_expired_completion_wait+0x0/0x250 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff800638ab>] schedule_timeout+0x8a/0xad
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff80097d9f>] process_timeout+0x0/0x5
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff886f1775>] ldlm_completion_ast+0x4d5/0x880 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff886d6709>] ldlm_lock_enqueue+0x9d9/0xb20 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff8008d07b>] default_wake_function+0x0/0xe
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff886d1b6a>] ldlm_lock_addref_internal_nolock+0x3a/0x90 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff886f02cb>] ldlm_cli_enqueue_local+0x46b/0x520 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff889ef4e1>] filter_destroy+0x681/0x1fb0 [obdfilter]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff886edbc0>] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff886f12a0>] ldlm_completion_ast+0x0/0x880 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff88716924>] lustre_msg_add_version+0x34/0x110 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff8871962a>] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff886d4efc>] __ldlm_handle2lock+0x8c/0x360 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff88719739>] lustre_pack_reply+0x29/0xb0 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff889a1050>] ost_destroy+0x660/0x790 [ost]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff88715a78>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff887158c5>] lustre_msg_get_opc+0x35/0xf0 [ptlrpc]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff889aaa26>] ost_handle+0x1556/0x55b0 [ost]
Feb 10 17:14:44 edclxs204 kernel:  [<ffffffff88722c37>] ptlrpc_server_handle_request+0xaa7/0x1150 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff800470f3>] try_to_wake_up+0x472/0x484
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff80062ff8>] thread_return+0x62/0xfe
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff8008b4a5>] __wake_up_common+0x3e/0x68
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff88726698>] ptlrpc_main+0x1258/0x1420 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff8008d07b>] default_wake_function+0x0/0xe
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff800b7a9c>] audit_syscall_exit+0x336/0x362
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff88725440>] ptlrpc_main+0x0/0x1420 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Feb 10 17:14:45 edclxs204 kernel: 
Feb 10 17:14:45 edclxs204 kernel: Pid: 8051, comm: ll_ost_io_80
Feb 10 17:14:45 edclxs204 kernel: 
Feb 10 17:14:45 edclxs204 kernel: Call Trace:
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff886efc10>] ldlm_expired_completion_wait+0x0/0x250 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff800638ab>] schedule_timeout+0x8a/0xad
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff80097d9f>] process_timeout+0x0/0x5
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff886f1775>] ldlm_completion_ast+0x4d5/0x880 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff886d6709>] ldlm_lock_enqueue+0x9d9/0xb20 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff8008d07b>] default_wake_function+0x0/0xe
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff886d1b6a>] ldlm_lock_addref_internal_nolock+0x3a/0x90 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff886f02cb>] ldlm_cli_enqueue_local+0x46b/0x520 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff889ef4e1>] filter_destroy+0x681/0x1fb0 [obdfilter]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff886edbc0>] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff886f12a0>] ldlm_completion_ast+0x0/0x880 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff88716924>] lustre_msg_add_version+0x34/0x110 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff8871962a>] lustre_pack_reply_flags+0x86a/0x950 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff886d4efc>] __ldlm_handle2lock+0x8c/0x360 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff88719739>] lustre_pack_reply+0x29/0xb0 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff889a1050>] ost_destroy+0x660/0x790 [ost]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff88715a78>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff887158c5>] lustre_msg_get_opc+0x35/0xf0 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff889aaa26>] ost_handle+0x1556/0x55b0 [ost]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff800dbcaa>] free_block+0x126/0x143
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff886dae29>] ldlm_resource_get+0x1d9/0xa50 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff88722c37>] ptlrpc_server_handle_request+0xaa7/0x1150 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff800470f3>] try_to_wake_up+0x472/0x484
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff8008b4a5>] __wake_up_common+0x3e/0x68
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff88726698>] ptlrpc_main+0x1258/0x1420 [ptlrpc]
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff8008d07b>] default_wake_function+0x0/0xe
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff800b7a9c>] audit_syscall_exit+0x336/0x362
Feb 10 17:14:45 edclxs204 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Feb 10 17:14:46 edclxs204 kernel:  [<ffffffff88725440>] ptlrpc_main+0x0/0x1420 [ptlrpc]
Feb 10 17:14:46 edclxs204 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Feb 10 17:14:46 edclxs204 kernel: 
Feb 10 17:14:46 edclxs204 kernel: LustreError: dumping log to /tmp/lustre-log.1297379681.8051
Feb 10 17:14:46 edclxs204 kernel: LustreError: dumping log to /tmp/lustre-log.1297379681.7992
Feb 10 17:14:46 edclxs204 kernel: LustreError: dumping log to /tmp/lustre-log.1297379682.8062
Feb 10 17:14:46 edclxs204 kernel: Lustre: Service thread pid 8642 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 10 17:14:46 edclxs204 kernel: LustreError: dumping log to /tmp/lustre-log.1297379686.8642
Feb 10 17:14:47 edclxs204 kernel: LustreError: 7988:0:(ldlm_request.c:83:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1297379480, 207s ago); not entering recovery in server code, just going back to sleep ns: filter-lustre-OST0004_UUID lock: ffff810333d85000/0x225009bc51a796d8 lrc: 3/0,1 mode: --/PW res: 48598775/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80004000 remote: 0x0 expref: -99 pid: 7988 timeout 0
Feb 10 17:14:47 edclxs204 kernel: LustreError: dumping log to /tmp/lustre-log.1297379687.7988
Feb 10 17:14:47 edclxs204 kernel: LustreError: 8062:0:(ldlm_request.c:83:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1297379480, 207s ago); not entering recovery in server code, just going back to sleep ns: filter-lustre-OST0005_UUID lock: ffff8103439fa200/0x225009bc51a79717 lrc: 3/0,1 mode: --/PW res: 49383274/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80004000 remote: 0x0 expref: -99 pid: 8062 timeout 0
Feb 10 17:14:52 edclxs204 kernel: Lustre: Service thread pid 8354 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 10 17:14:52 edclxs204 kernel: LustreError: dumping log to /tmp/lustre-log.1297379692.8354
Feb 10 17:14:55 edclxs204 kernel: Lustre: lustre-OST0005: haven't heard from client 217b8c92-def0-236b-3ea3-0c772031d4ed (at 192.168.0.100@tcp) in 227 seconds. I think it's dead, and I am evicting it.
Feb 10 17:14:55 edclxs204 kernel: LustreError: 138-a: lustre-OST0005: A client on nid 192.168.0.100@tcp was evicted due to a lock glimpse callback to 192.168.0.100@tcp timed out: rc -4
Feb 10 17:14:55 edclxs204 kernel: LustreError: 7880:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8103a90bd800 x1357189922742517/t0 o106->@NET_0x20000c0a80064_UUID:15/16 lens 296/424 e 0 to 1 dl 1297379702 ref 1 fl Rpc:XS/2/0 rc -4/0
Feb 10 17:14:55 edclxs204 kernel: LustreError: 7880:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 192.168.0.100@tcp) returned 0 from glimpse AST ns: filter-lustre-OST0005_UUID lock: ffff8101379ae800/0x225009bc51a510af lrc: 24/0,0 mode: PW/PW res: 5094186/0 rrc: 24 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x0 remote: 0xc6f4990e26fd59bf expref: 77042 pid: 8632 timeout 0
Feb 10 17:14:55 edclxs204 kernel: Lustre: Service thread pid 8354 completed after 202.24s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Feb 10 17:14:55 edclxs204 kernel: LustreError: Skipped 13 previous similar messages
Feb 10 17:14:57 edclxs204 kernel: LustreError: 7869:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff810346b03c00 x1357189922746005/t0 o106->@NET_0x20000c0a80064_UUID:15/16 lens 296/424 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0
Feb 10 17:14:57 edclxs204 kernel: LustreError: 7869:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 11 previous similar messages
Feb 10 17:14:57 edclxs204 kernel: LustreError: 7869:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 192.168.0.100@tcp) returned 0 from glimpse AST ns: filter-lustre-OST0004_UUID lock: ffff81015d609200/0x225009bc51a71910 lrc: 3/0,0 mode: PW/PW res: 48644715/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x0 remote: 0xc6f4990e27929041 expref: 49645 pid: 8394 timeout 0
Feb 10 17:14:57 edclxs204 kernel: LustreError: 7869:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) Skipped 11 previous similar messages
Feb 10 17:15:16 edclxs204 kernel: Lustre: Service thread pid 7988 completed after 236.48s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Feb 10 17:15:16 edclxs204 kernel: Lustre: Skipped 2 previous similar messages
Feb 10 17:16:34 edclxs204 kernel: Lustre: Service thread pid 8051 completed after 313.89s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Feb 10 17:23:24 edclxs204 kernel: Lustre: Service thread pid 8062 completed after 723.97s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Feb 10 17:23:36 edclxs204 kernel: Lustre: Service thread pid 7992 completed after 736.32s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Feb 10 19:12:05 edclxs204 kernel: LustreError: 137-5: UUID 'lustre-OST0006_UUID' is not available  for connect (no target)
Feb 10 19:12:05 edclxs204 kernel: LustreError: 8532:0:(ldlm_lib.c:1914:target_send_reply_msg()) @@@ processing error (-19)  req@ffff8105adfff400 x1350138880237571/t0 o8-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1297386825 ref 1 fl Interpret:/0/0 rc -19/0
Feb 10 19:12:05 edclxs204 kernel: LustreError: Skipped 1 previous similar message
