Jun 19 04:02:04 node97 syslogd 1.4.1: restart.
Jun 20 00:33:46 node97 kernel: Lustre: 7570:0:(quota_interface.c:457:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Jun 20 15:25:09 node97 kernel: LustreError: 7384:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 180541475: rc -2
Jun 20 15:51:53 node97 kernel: Lustre: 7529:0:(quota_interface.c:457:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Jun 20 16:03:39 node97 kernel: Lustre: 7527:0:(quota_interface.c:457:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Jun 20 20:11:48 node97 kernel: usb 3-1.4: new low speed USB device using ohci_hcd and address 4
Jun 20 20:11:49 node97 kernel: usb 3-1.4: configuration #1 chosen from 1 choice
Jun 20 20:11:49 node97 kernel: input: ATEN UC-10KM V1.3.124 as /class/input/input5
Jun 20 20:11:49 node97 kernel: input: USB HID v1.10 Keyboard [ATEN UC-10KM V1.3.124] on usb-0000:00:07.1-1.4
Jun 20 20:11:49 node97 kernel: input: ATEN UC-10KM V1.3.124 as /class/input/input6
Jun 20 20:11:49 node97 kernel: input: USB HID v1.10 Mouse [ATEN UC-10KM V1.3.124] on usb-0000:00:07.1-1.4
Jun 20 20:13:55 node97 kernel: usb 3-1.4: USB disconnect, address 4
Jun 20 22:15:37 node97 kernel: Lustre: 7569:0:(quota_interface.c:457:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Jun 21 05:21:38 node97 kernel: Lustre: 7977:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0003: 16efd810-b714-76b4-5aeb-eb8d4b669c00 reconnecting
Jun 21 05:21:38 node97 kernel: Lustre: 7977:0:(ldlm_lib.c:575:target_handle_reconnect()) Skipped 9 previous similar messages
Jun 21 05:30:31 node97 kernel: LustreError: 7165:0:(socklnd_cb.c:1707:ksocknal_recv_hello()) Error -11 reading HELLO from 192.168.111.41
Jun 21 05:30:58 node97 kernel: LustreError: 7166:0:(socklnd_cb.c:1707:ksocknal_recv_hello()) Error -11 reading HELLO from 192.168.111.41
Jun 21 05:31:09 node97 kernel: Lustre: 7690:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0003: 16efd810-b714-76b4-5aeb-eb8d4b669c00 reconnecting
Jun 21 05:31:09 node97 kernel: Lustre: 7690:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0003: refuse reconnection from 16efd810-b714-76b4-5aeb-eb8d4b669c00@11.11.11.41@tcp to 0xffff810111cde600; still busy with 1 active RPCs
Jun 21 05:31:09 node97 kernel: LustreError: 7690:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-16)  req@ffff810206143050 x1371645867869044/t0 o8->16efd810-b714-76b4-5aeb-eb8d4b669c00@:0/0 lens 368/264 e 0 to 0 dl 1308605569 ref 1 fl Interpret:/0/0 rc -16/0
Jun 21 05:31:09 node97 kernel: LustreError: 7690:0:(ldlm_lib.c:1892:target_send_reply_msg()) Skipped 169 previous similar messages
Jun 21 05:31:09 node97 kernel: LustreError: 7575:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff810107f6bc00 x1371645867868977/t0 o3->16efd810-b714-76b4-5aeb-eb8d4b669c00@:0/0 lens 448/400 e 2 to 0 dl 1308605597 ref 1 fl Interpret:/0/0 rc 0/0
Jun 21 05:31:09 node97 kernel: Lustre: 7575:0:(ost_handler.c:886:ost_brw_read()) lustre-OST0003: ignoring bulk IO comm error with 16efd810-b714-76b4-5aeb-eb8d4b669c00@ id 12345-11.11.11.41@tcp - client will retry
Jun 21 05:31:14 node97 kernel: Lustre: 7795:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: 16efd810-b714-76b4-5aeb-eb8d4b669c00 reconnecting
Jun 21 05:31:26 node97 kernel: Lustre: 7898:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0003: 16efd810-b714-76b4-5aeb-eb8d4b669c00 reconnecting
Jun 21 06:00:14 node97 kernel: Lustre: 7363:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: 16efd810-b714-76b4-5aeb-eb8d4b669c00 reconnecting
Jun 21 06:00:14 node97 kernel: Lustre: 7363:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0000: refuse reconnection from 16efd810-b714-76b4-5aeb-eb8d4b669c00@11.11.11.41@tcp to 0xffff81012f8e2e00; still busy with 1 active RPCs
Jun 21 06:00:14 node97 kernel: LustreError: 7363:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-16)  req@ffff8101089c8000 x1371645867872736/t0 o8->16efd810-b714-76b4-5aeb-eb8d4b669c00@:0/0 lens 368/264 e 0 to 0 dl 1308607314 ref 1 fl Interpret:/0/0 rc -16/0
Jun 21 06:00:14 node97 kernel: LustreError: 7510:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff81010d3e8400 x1371645867872727/t0 o3->16efd810-b714-76b4-5aeb-eb8d4b669c00@:0/0 lens 448/400 e 0 to 0 dl 1308607247 ref 1 fl Interpret:/0/0 rc 0/0
Jun 21 06:00:14 node97 kernel: Lustre: 7510:0:(ost_handler.c:886:ost_brw_read()) lustre-OST0000: ignoring bulk IO comm error with 16efd810-b714-76b4-5aeb-eb8d4b669c00@ id 12345-11.11.11.41@tcp - client will retry
Jun 21 06:00:21 node97 kernel: Lustre: 7403:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: 16efd810-b714-76b4-5aeb-eb8d4b669c00 reconnecting
Jun 21 06:00:26 node97 kernel: LustreError: 7707:0:(service.c:651:ptlrpc_check_req()) @@@ DROPPING req from old connection 122 < 124  req@ffff81020d434c50 x1371645867872720/t0 o400->16efd810-b714-76b4-5aeb-eb8d4b669c00@:0/0 lens 192/0 e 0 to 0 dl 0 ref 1 fl New:/0/0 rc 0/0
Jun 21 06:06:44 node97 kernel: Lustre: 7918:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: 16efd810-b714-76b4-5aeb-eb8d4b669c00 reconnecting
Jun 21 06:06:44 node97 kernel: Lustre: 7918:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0000: refuse reconnection from 16efd810-b714-76b4-5aeb-eb8d4b669c00@11.11.11.41@tcp to 0xffff81012f8e2e00; still busy with 2 active RPCs
Jun 21 06:06:44 node97 kernel: LustreError: 7918:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81020626a000 x1371645867873459/t0 o8->16efd810-b714-76b4-5aeb-eb8d4b669c00@:0/0 lens 368/264 e 0 to 0 dl 1308607704 ref 1 fl Interpret:/0/0 rc -16/0
Jun 21 06:06:44 node97 kernel: LustreError: 7572:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff8100cc5c1800 x1371645867873458/t0 o3->16efd810-b714-76b4-5aeb-eb8d4b669c00@:0/0 lens 448/400 e 0 to 0 dl 1308607617 ref 1 fl Interpret:/0/0 rc 0/0
Jun 21 06:06:44 node97 kernel: Lustre: 7572:0:(ost_handler.c:886:ost_brw_read()) lustre-OST0000: ignoring bulk IO comm error with 16efd810-b714-76b4-5aeb-eb8d4b669c00@ id 12345-11.11.11.41@tcp - client will retry
Jun 21 06:06:45 node97 kernel: LustreError: 7548:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff81010be5f400 x1371645867873448/t0 o3->16efd810-b714-76b4-5aeb-eb8d4b669c00@:0/0 lens 448/400 e 1 to 0 dl 1308607637 ref 1 fl Interpret:/0/0 rc 0/0
Jun 21 06:06:45 node97 kernel: Lustre: 7548:0:(ost_handler.c:886:ost_brw_read()) lustre-OST0000: ignoring bulk IO comm error with 16efd810-b714-76b4-5aeb-eb8d4b669c00@ id 12345-11.11.11.41@tcp - client will retry
Jun 21 06:06:46 node97 kernel: Lustre: 7756:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0003: 16efd810-b714-76b4-5aeb-eb8d4b669c00 reconnecting
Jun 21 06:06:47 node97 kernel: LustreError: 7732:0:(service.c:651:ptlrpc_check_req()) @@@ DROPPING req from old connection 123 < 124  req@ffff81010e71c800 x1371645867873443/t0 o400->16efd810-b714-76b4-5aeb-eb8d4b669c00@:0/0 lens 192/0 e 0 to 0 dl 0 ref 1 fl New:/0/0 rc 0/0
Jun 21 06:06:55 node97 kernel: Lustre: 7849:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: 16efd810-b714-76b4-5aeb-eb8d4b669c00 reconnecting
Jun 21 06:14:26 node97 kernel: Lustre: 7769:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: 16efd810-b714-76b4-5aeb-eb8d4b669c00 reconnecting
Jun 21 06:14:26 node97 kernel: LustreError: 7814:0:(service.c:651:ptlrpc_check_req()) @@@ DROPPING req from old connection 126 < 127  req@ffff8101130f4000 x1371645867874457/t0 o400->16efd810-b714-76b4-5aeb-eb8d4b669c00@:0/0 lens 192/0 e 0 to 0 dl 0 ref 1 fl New:/0/0 rc 0/0
Jun 21 06:32:54 node97 kernel: Lustre: lustre-OST0003: haven't heard from client 16efd810-b714-76b4-5aeb-eb8d4b669c00 (at 11.11.11.41@tcp) in 227 seconds. I think it's dead, and I am evicting it.
Jun 21 06:32:57 node97 kernel: Lustre: lustre-OST0000: haven't heard from client 16efd810-b714-76b4-5aeb-eb8d4b669c00 (at 11.11.11.41@tcp) in 230 seconds. I think it's dead, and I am evicting it.
Jun 21 11:33:22 node97 kernel: Lustre: 7513:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858242348637 sent from lustre-OST0000 to NID 11.11.11.46@tcp 7s ago has timed out (7s prior to deadline).
Jun 21 11:33:22 node97 kernel:   req@ffff810205d54400 x1371858242348637/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627202 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:22 node97 kernel: LustreError: 138-a: lustre-OST0000: A client on nid 11.11.11.46@tcp was evicted due to a lock blocking callback to 11.11.11.46@tcp timed out: rc -107
Jun 21 11:33:23 node97 kernel: Lustre: 7541:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858242348639 sent from lustre-OST0003 to NID 11.11.11.46@tcp 7s ago has timed out (7s prior to deadline).
Jun 21 11:33:23 node97 kernel:   req@ffff81021be97000 x1371858242348639/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627203 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:23 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.46@tcp was evicted due to a lock blocking callback to 11.11.11.46@tcp timed out: rc -107
Jun 21 11:33:23 node97 kernel: LustreError: Skipped 4 previous similar messages
Jun 21 11:33:24 node97 kernel: LustreError: 7589:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff810204497800 x1371858242348988/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:24 node97 kernel: LustreError: 7589:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.46@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff81022d3aa000/0x4ebf5f343aae7d5c lrc: 4/0,0 mode: PW/PW res: 179719431/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->20479) flags: 0x10020 remote: 0xd9c1cbe2fab530d2 expref: 6721 pid: 8029 timeout 4616063373
Jun 21 11:33:25 node97 kernel: Lustre: 7590:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858242348662 sent from lustre-OST0003 to NID 11.11.11.46@tcp 7s ago has timed out (7s prior to deadline).
Jun 21 11:33:25 node97 kernel:   req@ffff81010e0d2000 x1371858242348662/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627205 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:25 node97 kernel: Lustre: 7590:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Jun 21 11:33:25 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.46@tcp was evicted due to a lock blocking callback to 11.11.11.46@tcp timed out: rc -107
Jun 21 11:33:25 node97 kernel: LustreError: Skipped 4 previous similar messages
Jun 21 11:33:26 node97 kernel: LustreError: 7579:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8102032ba000 x1371858242349211/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:26 node97 kernel: LustreError: 7579:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.46@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff81022e65d000/0x4ebf5f3433700573 lrc: 4/0,0 mode: PW/PW res: 178238511/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->2097151) flags: 0x10020 remote: 0xd9c1cbe2d3a83294 expref: 6527 pid: 8030 timeout 4616065705
Jun 21 11:33:27 node97 kernel: LustreError: 7501:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff810203999c00 x1371858242349228/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:27 node97 kernel: LustreError: 7501:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.46@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff810083591000/0x4ebf5f3448ea4386 lrc: 4/0,0 mode: PR/PR res: 177944408/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0xd9c1cbe304cafc5b expref: 6354 pid: 7768 timeout 4616066823
Jun 21 11:33:28 node97 kernel: LustreError: 7516:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff810104ce9000 x1371858242349305/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:28 node97 kernel: LustreError: 7516:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 27 previous similar messages
Jun 21 11:33:28 node97 kernel: LustreError: 7516:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.46@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff8101f5108600/0x4ebf5f343342f0f7 lrc: 4/0,0 mode: PR/PR res: 177560404/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0xd9c1cbe2ce6d3ada expref: 6259 pid: 7883 timeout 4616067898
Jun 21 11:33:28 node97 kernel: LustreError: 7516:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) Skipped 27 previous similar messages
Jun 21 11:33:31 node97 kernel: Lustre: 7517:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858242348990 sent from lustre-OST0003 to NID 11.11.11.46@tcp 7s ago has timed out (7s prior to deadline).
Jun 21 11:33:31 node97 kernel:   req@ffff8102032bcc00 x1371858242348990/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627211 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:31 node97 kernel: Lustre: 7517:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Jun 21 11:33:31 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.46@tcp was evicted due to a lock blocking callback to 11.11.11.46@tcp timed out: rc -107
Jun 21 11:33:31 node97 kernel: LustreError: Skipped 2 previous similar messages
Jun 21 11:33:33 node97 kernel: Lustre: 7521:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858242349210 sent from lustre-OST0003 to NID 11.11.11.46@tcp 7s ago has timed out (7s prior to deadline).
Jun 21 11:33:33 node97 kernel:   req@ffff81020dbac400 x1371858242349210/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627213 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:33 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.46@tcp was evicted due to a lock blocking callback to 11.11.11.46@tcp timed out: rc -107
Jun 21 11:33:39 node97 kernel: LustreError: 7492:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8100cc282000 x1371858242349864/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:39 node97 kernel: LustreError: 7492:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.46@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff8101dd53ae00/0x4ebf5f34334b4a43 lrc: 4/0,0 mode: PR/PR res: 177160282/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0xd9c1cbe2cdebb45f expref: 5586 pid: 7935 timeout 4616079111
Jun 21 11:33:40 node97 kernel: LustreError: 7491:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.46@tcp: -113
Jun 21 11:33:40 node97 kernel: LustreError: 7491:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff810201565400 x1371858242349873/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627227 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:40 node97 kernel: Lustre: 7491:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858242349873 sent from lustre-OST0003 to NID 11.11.11.46@tcp 0s ago has failed due to network error (7s prior to deadline).
Jun 21 11:33:40 node97 kernel:   req@ffff810201565400 x1371858242349873/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627227 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:40 node97 kernel: Lustre: 7491:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
Jun 21 11:33:40 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.46@tcp was evicted due to a lock blocking callback to 11.11.11.46@tcp timed out: rc -107
Jun 21 11:33:40 node97 kernel: LustreError: Skipped 21 previous similar messages
Jun 21 11:33:50 node97 kernel: LustreError: 7538:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff81010e0c1000 x1371858242350038/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:50 node97 kernel: LustreError: 7538:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 5 previous similar messages
Jun 21 11:33:50 node97 kernel: LustreError: 7538:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.46@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff8101efb62600/0x4ebf5f343344f70c lrc: 4/0,0 mode: PR/PR res: 177125609/0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0xd9c1cbe2cdbab7d8 expref: 5019 pid: 7662 timeout 4616090035
Jun 21 11:33:50 node97 kernel: LustreError: 7538:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) Skipped 5 previous similar messages
Jun 21 11:33:50 node97 kernel: LustreError: 7528:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.46@tcp: -113
Jun 21 11:33:50 node97 kernel: LustreError: 7528:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff810110847000 x1371858242350044/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627237 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:50 node97 kernel: Lustre: 7528:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858242350044 sent from lustre-OST0003 to NID 11.11.11.46@tcp 0s ago has failed due to network error (7s prior to deadline).
Jun 21 11:33:50 node97 kernel:   req@ffff810110847000 x1371858242350044/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627237 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:50 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.46@tcp was evicted due to a lock blocking callback to 11.11.11.46@tcp timed out: rc -107
Jun 21 11:33:51 node97 kernel: LustreError: 7540:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.46@tcp: -113
Jun 21 11:33:51 node97 kernel: LustreError: 7540:0:(lib-move.c:2441:LNetPut()) Skipped 1 previous similar message
Jun 21 11:33:51 node97 kernel: LustreError: 7540:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff810108060000 x1371858242350066/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627238 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:51 node97 kernel: LustreError: 7540:0:(events.c:66:request_out_callback()) Skipped 1 previous similar message
Jun 21 11:33:56 node97 kernel: LustreError: 7555:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.46@tcp: -113
Jun 21 11:33:56 node97 kernel: LustreError: 7555:0:(lib-move.c:2441:LNetPut()) Skipped 1 previous similar message
Jun 21 11:33:56 node97 kernel: LustreError: 7555:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff8102072bd000 x1371858242350254/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627243 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:56 node97 kernel: LustreError: 7555:0:(events.c:66:request_out_callback()) Skipped 1 previous similar message
Jun 21 11:33:58 node97 kernel: LustreError: 7587:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8101ff7a1c00 x1371858242350268/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:58 node97 kernel: LustreError: 7587:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 6 previous similar messages
Jun 21 11:33:58 node97 kernel: LustreError: 7587:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.46@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff8101fda53200/0x4ebf5f34334040b2 lrc: 4/0,0 mode: PW/PW res: 176365072/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0xd9c1cbe297df33ab expref: 4218 pid: 7653 timeout 4616098092
Jun 21 11:33:58 node97 kernel: LustreError: 7587:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) Skipped 6 previous similar messages
Jun 21 11:33:58 node97 kernel: LustreError: 7484:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.46@tcp: -113
Jun 21 11:33:58 node97 kernel: LustreError: 7484:0:(lib-move.c:2441:LNetPut()) Skipped 3 previous similar messages
Jun 21 11:33:58 node97 kernel: LustreError: 7484:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff81011b575000 x1371858242350274/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627245 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 21 11:33:58 node97 kernel: LustreError: 7484:0:(events.c:66:request_out_callback()) Skipped 3 previous similar messages
Jun 21 11:34:03 node97 kernel: LustreError: 7549:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.46@tcp: -113
Jun 21 11:34:03 node97 kernel: LustreError: 7549:0:(lib-move.c:2441:LNetPut()) Skipped 34 previous similar messages
Jun 21 11:34:03 node97 kernel: LustreError: 7549:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff8102099eac00 x1371858242350483/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308627250 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 21 11:34:03 node97 kernel: LustreError: 7549:0:(events.c:66:request_out_callback()) Skipped 34 previous similar messages
Jun 21 11:34:41 node97 kernel: LustreError: 7742:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff810207c40400 x1371858242350927/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0
Jun 21 11:34:41 node97 kernel: LustreError: 7742:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 65 previous similar messages
Jun 21 11:34:41 node97 kernel: LustreError: 7742:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.46@tcp) returned 0 from glimpse AST ns: filter-lustre-OST0003_UUID lock: ffff8101150a7200/0x4ebf5f3448c89e1b lrc: 3/0,0 mode: PW/PW res: 180672617/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80000000 remote: 0xd9c1cbe304787d69 expref: 3117 pid: 7367 timeout 0
Jun 21 11:34:41 node97 kernel: LustreError: 7742:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) Skipped 65 previous similar messages
Jun 21 13:03:46 node97 kernel: LustreError: 8005:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 179400251: rc -2
Jun 21 13:03:46 node97 kernel: LustreError: 7699:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 179400251: rc -2
Jun 21 15:23:08 node97 kernel: Lustre: 7805:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858242836064 sent from lustre-OST0000 to NID 11.11.11.42@tcp 7s ago has timed out (7s prior to deadline).
Jun 21 15:23:08 node97 kernel:   req@ffff810200c3a000 x1371858242836064/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1308640988 ref 1 fl Rpc:/0/0 rc 0/0
Jun 21 15:23:08 node97 kernel: Lustre: 7805:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 43 previous similar messages
Jun 21 15:23:15 node97 kernel: Lustre: 7805:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858242836064 sent from lustre-OST0000 to NID 11.11.11.42@tcp 7s ago has timed out (7s prior to deadline).
Jun 21 15:23:15 node97 kernel:   req@ffff810200c3a000 x1371858242836064/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1308640995 ref 1 fl Rpc:/2/0 rc 0/0
Jun 21 15:23:15 node97 kernel: Lustre: 7805:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 21 15:23:22 node97 kernel: Lustre: 7805:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858242836064 sent from lustre-OST0000 to NID 11.11.11.42@tcp 7s ago has timed out (7s prior to deadline).
Jun 21 15:23:22 node97 kernel:   req@ffff810200c3a000 x1371858242836064/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1308641002 ref 1 fl Rpc:/2/0 rc 0/0
Jun 21 15:23:22 node97 kernel: Lustre: 7805:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 21 15:23:35 node97 kernel: Lustre: 7529:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858242836367 sent from lustre-OST0000 to NID 11.11.11.42@tcp 7s ago has timed out (7s prior to deadline).
Jun 21 15:23:35 node97 kernel:   req@ffff81021c1d1800 x1371858242836367/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308641015 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 21 15:23:35 node97 kernel: Lustre: 7529:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 21 15:23:35 node97 kernel: LustreError: 138-a: lustre-OST0000: A client on nid 11.11.11.42@tcp was evicted due to a lock blocking callback to 11.11.11.42@tcp timed out: rc -107
Jun 21 15:23:35 node97 kernel: LustreError: Skipped 43 previous similar messages
Jun 21 15:23:50 node97 kernel: LustreError: 7804:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.42@tcp: -113
Jun 21 15:23:50 node97 kernel: LustreError: 7804:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff8101fe463400 x1371858242836065/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1308641037 ref 2 fl Rpc:/2/0 rc 0/0
Jun 21 15:23:51 node97 kernel: Lustre: 7804:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858242836065 sent from lustre-OST0003 to NID 11.11.11.42@tcp 0s ago has failed due to network error (7s prior to deadline).
Jun 21 15:23:51 node97 kernel:   req@ffff8101fe463400 x1371858242836065/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1308641038 ref 1 fl Rpc:/2/0 rc 0/0
Jun 21 15:23:51 node97 kernel: Lustre: 7804:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 27556 previous similar messages
Jun 21 15:23:51 node97 kernel: LustreError: 7804:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.42@tcp: -113
Jun 21 15:23:51 node97 kernel: LustreError: 7804:0:(lib-move.c:2441:LNetPut()) Skipped 28724 previous similar messages
Jun 21 15:23:51 node97 kernel: LustreError: 7804:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff8101fe463400 x1371858242836065/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1308641038 ref 2 fl Rpc:/2/0 rc 0/0
Jun 21 15:23:51 node97 kernel: LustreError: 7804:0:(events.c:66:request_out_callback()) Skipped 28724 previous similar messages
Jun 21 15:23:53 node97 kernel: LustreError: 7804:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff8101fe463400 x1371858242836065/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1308641040 ref 2 fl Rpc:/2/0 rc 0/0
Jun 21 15:23:53 node97 kernel: LustreError: 7804:0:(events.c:66:request_out_callback()) Skipped 115720 previous similar messages
Jun 21 15:23:53 node97 kernel: LustreError: 7804:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.42@tcp: -113
Jun 21 15:23:53 node97 kernel: LustreError: 7804:0:(lib-move.c:2441:LNetPut()) Skipped 115721 previous similar messages
Jun 21 15:23:57 node97 kernel: LustreError: 7804:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.42@tcp: -113
Jun 21 15:23:57 node97 kernel: LustreError: 7804:0:(lib-move.c:2441:LNetPut()) Skipped 232751 previous similar messages
Jun 21 15:23:57 node97 kernel: LustreError: 7804:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff8101fe463400 x1371858242836065/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1308641044 ref 2 fl Rpc:/2/0 rc 0/0
Jun 21 15:23:57 node97 kernel: LustreError: 7804:0:(events.c:66:request_out_callback()) Skipped 232752 previous similar messages
Jun 21 15:24:05 node97 kernel: LustreError: 7804:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.42@tcp: -113
Jun 21 15:24:05 node97 kernel: LustreError: 7804:0:(lib-move.c:2441:LNetPut()) Skipped 467298 previous similar messages
Jun 21 15:24:05 node97 kernel: LustreError: 7804:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff8101fe463400 x1371858242836065/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1308641052 ref 2 fl Rpc:/2/0 rc 0/0
Jun 21 15:24:05 node97 kernel: LustreError: 7804:0:(events.c:66:request_out_callback()) Skipped 467298 previous similar messages
Jun 21 15:24:06 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.42@tcp was evicted due to a lock glimpse callback to 11.11.11.42@tcp timed out: rc -4
Jun 21 15:24:06 node97 kernel: LustreError: Skipped 7 previous similar messages
Jun 21 16:30:49 node97 kernel: LustreError: 7660:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 174318386: rc -2
Jun 21 16:52:51 node97 kernel: LustreError: 7760:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 180783025: rc -2
Jun 21 16:52:51 node97 kernel: LustreError: 7348:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 181851458: rc -2
Jun 21 18:30:22 node97 kernel: LustreError: 7424:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 180425190: rc -2
Jun 21 18:30:24 node97 kernel: LustreError: 7341:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 180413499: rc -2
Jun 21 18:30:24 node97 kernel: LustreError: 7341:0:(ldlm_resource.c:860:ldlm_resource_add()) Skipped 1 previous similar message
Jun 21 18:30:25 node97 kernel: LustreError: 7873:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 181487437: rc -2
Jun 21 18:30:26 node97 kernel: LustreError: 7940:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 181477433: rc -2
Jun 21 18:30:26 node97 kernel: LustreError: 7940:0:(ldlm_resource.c:860:ldlm_resource_add()) Skipped 3 previous similar messages
Jun 21 21:44:27 node97 kernel: Lustre: 7580:0:(quota_interface.c:457:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Jun 22 02:08:34 node97 kernel: LustreError: 7923:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 180577683: rc -2
Jun 22 02:09:40 node97 kernel: LustreError: 7370:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 180577683: rc -2
Jun 22 02:09:49 node97 kernel: LustreError: 7882:0:(ldlm_resource.c:860:ldlm_resource_add()) lvbo_init failed for resource 180577683: rc -2
Jun 22 03:15:29 node97 kernel: Lustre: 8015:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: aa9644e0-b916-739d-f886-1aa15c6ab444 reconnecting
Jun 22 03:15:29 node97 kernel: Lustre: 8015:0:(ldlm_lib.c:575:target_handle_reconnect()) Skipped 1 previous similar message
Jun 22 03:15:29 node97 kernel: Lustre: 8015:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0000: refuse reconnection from aa9644e0-b916-739d-f886-1aa15c6ab444@11.11.11.47@tcp to 0xffff81012bd04600; still busy with 1 active RPCs
Jun 22 03:15:29 node97 kernel: LustreError: 8015:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81020872a400 x1371463420976703/t0 o8->aa9644e0-b916-739d-f886-1aa15c6ab444@:0/0 lens 368/264 e 0 to 0 dl 1308683829 ref 1 fl Interpret:/0/0 rc -16/0
Jun 22 03:15:30 node97 kernel: LustreError: 7550:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff8101ff53e400 x1371463420976701/t0 o3->aa9644e0-b916-739d-f886-1aa15c6ab444@:0/0 lens 448/400 e 0 to 0 dl 1308683736 ref 1 fl Interpret:/0/0 rc 0/0
Jun 22 03:15:30 node97 kernel: Lustre: 7550:0:(ost_handler.c:886:ost_brw_read()) lustre-OST0000: ignoring bulk IO comm error with aa9644e0-b916-739d-f886-1aa15c6ab444@ id 12345-11.11.11.47@tcp - client will retry
Jun 22 03:15:35 node97 kernel: Lustre: 7904:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: aa9644e0-b916-739d-f886-1aa15c6ab444 reconnecting
Jun 22 03:15:35 node97 kernel: Lustre: 7904:0:(ldlm_lib.c:575:target_handle_reconnect()) Skipped 4 previous similar messages
Jun 22 03:26:35 node97 kernel: LustreError: 7169:0:(acceptor.c:455:lnet_acceptor()) Error -11 reading connection request from 192.168.111.47
Jun 22 03:27:07 node97 kernel: Lustre: 7900:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: aa9644e0-b916-739d-f886-1aa15c6ab444 reconnecting
Jun 22 03:27:39 node97 kernel: Lustre: 7687:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: aa9644e0-b916-739d-f886-1aa15c6ab444 reconnecting
Jun 22 03:27:39 node97 kernel: Lustre: 7687:0:(ldlm_lib.c:575:target_handle_reconnect()) Skipped 2 previous similar messages
Jun 22 03:27:39 node97 kernel: Lustre: 7765:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0003: refuse reconnection from aa9644e0-b916-739d-f886-1aa15c6ab444@11.11.11.47@tcp to 0xffff81011841d400; still busy with 4 active RPCs
Jun 22 03:27:39 node97 kernel: Lustre: 7765:0:(ldlm_lib.c:875:target_handle_connect()) Skipped 2 previous similar messages
Jun 22 03:27:39 node97 kernel: LustreError: 7765:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-16)  req@ffff8102062ff000 x1371463420979584/t0 o8->aa9644e0-b916-739d-f886-1aa15c6ab444@:0/0 lens 368/264 e 0 to 0 dl 1308684559 ref 1 fl Interpret:/0/0 rc -16/0
Jun 22 03:27:39 node97 kernel: LustreError: 7765:0:(ldlm_lib.c:1892:target_send_reply_msg()) Skipped 2 previous similar messages
Jun 22 03:27:39 node97 kernel: LustreError: 7560:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff8100cc8d3000 x1371463420979568/t0 o3->aa9644e0-b916-739d-f886-1aa15c6ab444@:0/0 lens 448/400 e 0 to 0 dl 1308684466 ref 1 fl Interpret:/0/0 rc 0/0
Jun 22 03:27:39 node97 kernel: Lustre: 7482:0:(ost_handler.c:886:ost_brw_read()) lustre-OST0003: ignoring bulk IO comm error with aa9644e0-b916-739d-f886-1aa15c6ab444@ id 12345-11.11.11.47@tcp - client will retry
Jun 22 03:27:39 node97 kernel: LustreError: 7560:0:(ost_handler.c:829:ost_brw_read()) Skipped 1 previous similar message
Jun 22 03:27:40 node97 kernel: LustreError: 7484:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff81010dcbf000 x1371463420979566/t0 o3->aa9644e0-b916-739d-f886-1aa15c6ab444@:0/0 lens 448/400 e 0 to 0 dl 1308684466 ref 1 fl Interpret:/0/0 rc 0/0
Jun 22 03:27:40 node97 kernel: LustreError: 7484:0:(ost_handler.c:829:ost_brw_read()) Skipped 1 previous similar message
Jun 22 03:27:40 node97 kernel: Lustre: 7484:0:(ost_handler.c:886:ost_brw_read()) lustre-OST0003: ignoring bulk IO comm error with aa9644e0-b916-739d-f886-1aa15c6ab444@ id 12345-11.11.11.47@tcp - client will retry
Jun 22 03:27:40 node97 kernel: Lustre: 7484:0:(ost_handler.c:886:ost_brw_read()) Skipped 2 previous similar messages
Jun 22 03:27:59 node97 kernel: Lustre: 7786:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0003: aa9644e0-b916-739d-f886-1aa15c6ab444 reconnecting
Jun 22 03:33:43 node97 kernel: Lustre: Service thread pid 7584 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jun 22 03:33:43 node97 kernel: Lustre: 0:0:(linux-debug.c:264:libcfs_debug_dumpstack()) showing stack for process 7584
Jun 22 03:33:43 node97 kernel: ll_ost_io_120 S ffff81000101d7a0     0  7584      1          7585  7583 (L-TLB)
Jun 22 03:33:43 node97 kernel:  ffff810110a17a00 0000000000000046 0000000000000001 ffffffff885582c1
Jun 22 03:33:43 node97 kernel:  ffff81010dc0f000 000000000000000a ffff810118f48100 ffff81022feac0c0
Jun 22 03:33:43 node97 kernel:  000157ff1442b790 00000000000014dd ffff810118f482e8 0000000600000000
Jun 22 03:33:43 node97 kernel: Call Trace:
Jun 22 03:33:43 node97 kernel:  [<ffffffff885582c1>] :lnet:LNetMDBind+0x301/0x450
Jun 22 03:33:43 node97 kernel:  [<ffffffff800638ab>] schedule_timeout+0x8a/0xad
Jun 22 03:33:43 node97 kernel:  [<ffffffff800974a4>] process_timeout+0x0/0x5
Jun 22 03:33:43 node97 kernel:  [<ffffffff888f71fc>] :ost:ost_brw_read+0x127c/0x1a70
Jun 22 03:33:43 node97 kernel:  [<ffffffff8008c86b>] default_wake_function+0x0/0xe
Jun 22 03:33:43 node97 kernel:  [<ffffffff8866e788>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Jun 22 03:33:43 node97 kernel:  [<ffffffff888feee3>] :ost:ost_handle+0x2e73/0x53d0
Jun 22 03:33:43 node97 kernel:  [<ffffffff800dae73>] free_block+0x126/0x143
Jun 22 03:33:44 node97 kernel:  [<ffffffff8867b9b5>] :ptlrpc:ptlrpc_server_handle_request+0xaa5/0x1140
Jun 22 03:33:44 node97 kernel:  [<ffffffff80047152>] try_to_wake_up+0x472/0x484
Jun 22 03:33:44 node97 kernel:  [<ffffffff8008ac95>] __wake_up_common+0x3e/0x68
Jun 22 03:33:44 node97 kernel:  [<ffffffff8867f408>] :ptlrpc:ptlrpc_main+0x1258/0x1420
Jun 22 03:33:44 node97 kernel:  [<ffffffff8008c86b>] default_wake_function+0x0/0xe
Jun 22 03:33:44 node97 kernel:  [<ffffffff800b7076>] audit_syscall_exit+0x336/0x362
Jun 22 03:33:44 node97 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jun 22 03:33:44 node97 kernel:  [<ffffffff8867e1b0>] :ptlrpc:ptlrpc_main+0x0/0x1420
Jun 22 03:33:44 node97 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jun 22 03:33:44 node97 kernel: 
Jun 22 03:33:44 node97 kernel: LustreError: dumping log to /tmp/lustre-log.1308684823.7584
Jun 22 03:34:04 node97 kernel: Lustre: lustre-OST0003: haven't heard from client aa9644e0-b916-739d-f886-1aa15c6ab444 (at 11.11.11.47@tcp) in 227 seconds. I think it's dead, and I am evicting it.
Jun 22 03:34:10 node97 kernel: Lustre: lustre-OST0000: haven't heard from client aa9644e0-b916-739d-f886-1aa15c6ab444 (at 11.11.11.47@tcp) in 227 seconds. I think it's dead, and I am evicting it.
Jun 22 03:34:10 node97 kernel: LustreError: 7584:0:(ost_handler.c:825:ost_brw_read()) @@@ Eviction on bulk PUT  req@ffff81020863ec00 x1371463420980216/t0 o3->aa9644e0-b916-739d-f886-1aa15c6ab444@:0/0 lens 448/400 e 2 to 0 dl 1308684995 ref 1 fl Interpret:/0/0 rc 0/0
Jun 22 03:34:10 node97 kernel: Lustre: 7584:0:(ost_handler.c:886:ost_brw_read()) lustre-OST0000: ignoring bulk IO comm error with aa9644e0-b916-739d-f886-1aa15c6ab444@ id 12345-11.11.11.47@tcp - client will retry
Jun 22 03:34:10 node97 kernel: Lustre: Service thread pid 7584 completed after 227.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Jun 22 03:53:00 node97 kernel: Lustre: 7376:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: aa9644e0-b916-739d-f886-1aa15c6ab444 reconnecting
Jun 22 03:53:00 node97 kernel: Lustre: 7989:0:(ldlm_lib.c:804:target_handle_connect()) lustre-OST0003: exp ffff81020d8db400 already connecting
Jun 22 03:53:00 node97 kernel: LustreError: 7989:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-114)  req@ffff81020a524c50 x1371463420980278/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1308686080 ref 1 fl Interpret:/0/0 rc -114/0
Jun 22 03:53:00 node97 kernel: LustreError: 7989:0:(ldlm_lib.c:1892:target_send_reply_msg()) Skipped 1 previous similar message
Jun 22 03:53:00 node97 kernel: LustreError: 7376:0:(ldlm_lib.c:997:target_handle_connect()) aa9644e0-b916-739d-f886-1aa15c6ab444: 11.11.11.47@tcp already connected at higher conn_cnt: 140 > 139
Jun 22 03:53:00 node97 kernel: Lustre: 8030:0:(ldlm_lib.c:804:target_handle_connect()) lustre-OST0003: exp ffff81020d8db400 already connecting
Jun 22 03:53:00 node97 kernel: Lustre: 8030:0:(ldlm_lib.c:804:target_handle_connect()) Skipped 2 previous similar messages
Jun 22 03:53:10 node97 kernel: Lustre: 7807:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: aa9644e0-b916-739d-f886-1aa15c6ab444 reconnecting
Jun 22 03:53:10 node97 kernel: Lustre: 7807:0:(ldlm_lib.c:575:target_handle_reconnect()) Skipped 70 previous similar messages
Jun 22 09:09:33 node97 kernel: Lustre: lustre-OST0000: slow setattr 30s due to heavy IO load
Jun 22 10:34:13 node97 kernel: Lustre: 7682:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858245691565 sent from lustre-OST0000 to NID 11.11.11.41@tcp 7s ago has timed out (7s prior to deadline).
Jun 22 10:34:13 node97 kernel:   req@ffff81020892a000 x1371858245691565/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308710053 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 10:34:13 node97 kernel: Lustre: 7682:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 863491 previous similar messages
Jun 22 10:34:13 node97 kernel: LustreError: 138-a: lustre-OST0000: A client on nid 11.11.11.41@tcp was evicted due to a lock blocking callback to 11.11.11.41@tcp timed out: rc -107
Jun 22 10:34:14 node97 kernel: LustreError: 7668:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff81020b36d400 x1371858245692104/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 10:34:14 node97 kernel: LustreError: 7668:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.41@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff8101bf164e00/0x4ebf5f345596830c lrc: 4/0,0 mode: PW/PW res: 145210912/0 rrc: 2 type: EXT [0->18446744073709551615] (req 24576->28671) flags: 0x20 remote: 0xfa4d47bd3630e1d5 expref: 9314 pid: 7884 timeout 4698914233
Jun 22 10:35:05 node97 kernel: LustreError: 7524:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.41@tcp: -113
Jun 22 10:35:05 node97 kernel: LustreError: 7524:0:(lib-move.c:2441:LNetPut()) Skipped 46543 previous similar messages
Jun 22 10:35:05 node97 kernel: LustreError: 7524:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff81011211b800 x1371858245695835/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308710112 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 22 10:35:05 node97 kernel: LustreError: 7524:0:(events.c:66:request_out_callback()) Skipped 46543 previous similar messages
Jun 22 10:35:05 node97 kernel: Lustre: 7524:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858245695835 sent from lustre-OST0003 to NID 11.11.11.41@tcp 0s ago has failed due to network error (7s prior to deadline).
Jun 22 10:35:05 node97 kernel:   req@ffff81011211b800 x1371858245695835/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308710112 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 10:35:05 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.41@tcp was evicted due to a lock blocking callback to 11.11.11.41@tcp timed out: rc -107
Jun 22 10:35:05 node97 kernel: LustreError: 7480:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff81011a1d9800 x1371858245695839/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 10:35:05 node97 kernel: LustreError: 7480:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.41@tcp) returned 0 from blocking AST ns: filter-lustre-OST0003_UUID lock: ffff8101f52e5800/0x4ebf5f3455737281 lrc: 4/0,0 mode: PR/PR res: 182264363/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0xfa4d47bd35ee4072 expref: 10728 pid: 7879 timeout 4698965169
Jun 22 11:01:19 node97 kernel: Lustre: 7557:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858245797502 sent from lustre-OST0000 to NID 11.11.11.50@tcp 7s ago has timed out (7s prior to deadline).
Jun 22 11:01:19 node97 kernel:   req@ffff8102096a9800 x1371858245797502/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711679 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:19 node97 kernel: Lustre: 7557:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 22 11:01:19 node97 kernel: LustreError: 138-a: lustre-OST0000: A client on nid 11.11.11.50@tcp was evicted due to a lock blocking callback to 11.11.11.50@tcp timed out: rc -107
Jun 22 11:01:19 node97 kernel: LustreError: Skipped 1 previous similar message
Jun 22 11:01:19 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.50@tcp was evicted due to a lock blocking callback to 11.11.11.50@tcp timed out: rc -107
Jun 22 11:01:19 node97 kernel: LustreError: Skipped 124 previous similar messages
Jun 22 11:01:20 node97 kernel: LustreError: 7500:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff810108272800 x1371858245797639/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:20 node97 kernel: LustreError: 7500:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 782 previous similar messages
Jun 22 11:01:20 node97 kernel: LustreError: 7500:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.50@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff8101f7ee6200/0x4ebf5f3455ab161c lrc: 4/0,0 mode: PW/PW res: 181780898/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x10020 remote: 0x9dab7525ec3a6363 expref: 8510 pid: 8019 timeout 4700539669
Jun 22 11:01:20 node97 kernel: LustreError: 7500:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) Skipped 782 previous similar messages
Jun 22 11:01:22 node97 kernel: LustreError: 7548:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff810050bf5400 x1371858245797652/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:22 node97 kernel: LustreError: 7548:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
Jun 22 11:01:22 node97 kernel: LustreError: 7548:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.50@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff8101c11f8400/0x4ebf5f3455aafa69 lrc: 4/0,0 mode: PW/PW res: 181780708/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x10020 remote: 0x9dab7525ec39bf53 expref: 8146 pid: 7424 timeout 4700541924
Jun 22 11:01:22 node97 kernel: LustreError: 7548:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) Skipped 2 previous similar messages
Jun 22 11:01:22 node97 kernel: Lustre: 3965:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858245797631 sent from lustre-OST0003 to NID 11.11.11.50@tcp 7s ago has timed out (7s prior to deadline).
Jun 22 11:01:22 node97 kernel:   req@ffff81020a218000 x1371858245797631/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711682 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:22 node97 kernel: Lustre: 3965:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 67 previous similar messages
Jun 22 11:01:22 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.50@tcp was evicted due to a lock blocking callback to 11.11.11.50@tcp timed out: rc -107
Jun 22 11:01:26 node97 kernel: LustreError: 7474:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff810108aad400 x1371858245797675/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:26 node97 kernel: LustreError: 7474:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 7 previous similar messages
Jun 22 11:01:26 node97 kernel: LustreError: 7474:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.50@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff8101eb684800/0x4ebf5f3455aafd95 lrc: 4/0,0 mode: PW/PW res: 181780728/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x10020 remote: 0x9dab7525ec39d7cc expref: 7527 pid: 7827 timeout 4700545511
Jun 22 11:01:26 node97 kernel: LustreError: 7474:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) Skipped 7 previous similar messages
Jun 22 11:01:27 node97 kernel: Lustre: 7500:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858245797640 sent from lustre-OST0003 to NID 11.11.11.50@tcp 7s ago has timed out (7s prior to deadline).
Jun 22 11:01:27 node97 kernel:   req@ffff81011a113000 x1371858245797640/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711687 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:27 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.50@tcp was evicted due to a lock blocking callback to 11.11.11.50@tcp timed out: rc -107
Jun 22 11:01:29 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.50@tcp was evicted due to a lock blocking callback to 11.11.11.50@tcp timed out: rc -107
Jun 22 11:01:29 node97 kernel: LustreError: Skipped 3 previous similar messages
Jun 22 11:01:31 node97 kernel: LustreError: 7464:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff81011e1dc000 x1371858245797689/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:31 node97 kernel: LustreError: 7464:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 4 previous similar messages
Jun 22 11:01:31 node97 kernel: LustreError: 7464:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.50@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff81004d8a5600/0x4ebf5f3455aaf1b7 lrc: 4/0,0 mode: PW/PW res: 181780639/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x10020 remote: 0x9dab7525ec3992cf expref: 6580 pid: 8034 timeout 4700550508
Jun 22 11:01:31 node97 kernel: LustreError: 7464:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) Skipped 4 previous similar messages
Jun 22 11:01:31 node97 kernel: LustreError: 7464:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.50@tcp: -113
Jun 22 11:01:31 node97 kernel: LustreError: 7464:0:(lib-move.c:2441:LNetPut()) Skipped 1 previous similar message
Jun 22 11:01:31 node97 kernel: LustreError: 7464:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff81005a36d800 x1371858245797690/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711698 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:31 node97 kernel: LustreError: 7464:0:(events.c:66:request_out_callback()) Skipped 1 previous similar message
Jun 22 11:01:32 node97 kernel: LustreError: 7510:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.50@tcp: -113
Jun 22 11:01:32 node97 kernel: LustreError: 7510:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff810104e5e800 x1371858245797692/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711699 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:32 node97 kernel: Lustre: 7510:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858245797692 sent from lustre-OST0003 to NID 11.11.11.50@tcp 0s ago has failed due to network error (7s prior to deadline).
Jun 22 11:01:32 node97 kernel:   req@ffff810104e5e800 x1371858245797692/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711699 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:32 node97 kernel: Lustre: 7510:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Jun 22 11:01:32 node97 kernel: LustreError: 7557:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.50@tcp: -113
Jun 22 11:01:32 node97 kernel: LustreError: 7557:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff8101fe21a000 x1371858245797694/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711699 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:33 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.50@tcp was evicted due to a lock blocking callback to 11.11.11.50@tcp timed out: rc -107
Jun 22 11:01:33 node97 kernel: LustreError: Skipped 13 previous similar messages
Jun 22 11:01:33 node97 kernel: LustreError: 7535:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.50@tcp: -113
Jun 22 11:01:33 node97 kernel: LustreError: 7535:0:(lib-move.c:2441:LNetPut()) Skipped 2 previous similar messages
Jun 22 11:01:33 node97 kernel: LustreError: 7535:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff8101fe89fc00 x1371858245797699/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711700 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:33 node97 kernel: LustreError: 7535:0:(events.c:66:request_out_callback()) Skipped 2 previous similar messages
Jun 22 11:01:35 node97 kernel: LustreError: 7590:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.50@tcp: -113
Jun 22 11:01:35 node97 kernel: LustreError: 7590:0:(lib-move.c:2441:LNetPut()) Skipped 5 previous similar messages
Jun 22 11:01:35 node97 kernel: LustreError: 7590:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff810034f92000 x1371858245797709/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711702 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:35 node97 kernel: LustreError: 7590:0:(events.c:66:request_out_callback()) Skipped 5 previous similar messages
Jun 22 11:01:40 node97 kernel: LustreError: 7583:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff81020ba26c00 x1371858245797810/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:40 node97 kernel: LustreError: 7583:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 12 previous similar messages
Jun 22 11:01:40 node97 kernel: LustreError: 7583:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.50@tcp) returned 0 from blocking AST ns: filter-lustre-OST0000_UUID lock: ffff81019c08da00/0x4ebf5f3455ab3f6d lrc: 4/0,0 mode: PW/PW res: 181781215/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x10020 remote: 0x9dab7525ec3afac3 expref: 4818 pid: 7905 timeout 4700559353
Jun 22 11:01:40 node97 kernel: LustreError: 7583:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) Skipped 12 previous similar messages
Jun 22 11:01:40 node97 kernel: LustreError: 7583:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.50@tcp: -113
Jun 22 11:01:40 node97 kernel: LustreError: 7583:0:(lib-move.c:2441:LNetPut()) Skipped 4 previous similar messages
Jun 22 11:01:40 node97 kernel: LustreError: 7583:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff810201e3ac00 x1371858245797811/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711707 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:40 node97 kernel: LustreError: 7583:0:(events.c:66:request_out_callback()) Skipped 4 previous similar messages
Jun 22 11:01:41 node97 kernel: Lustre: 7502:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858245797813 sent from lustre-OST0003 to NID 11.11.11.50@tcp 0s ago has failed due to network error (7s prior to deadline).
Jun 22 11:01:41 node97 kernel:   req@ffff810031d0b400 x1371858245797813/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711708 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:41 node97 kernel: Lustre: 7502:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
Jun 22 11:01:42 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.50@tcp was evicted due to a lock blocking callback to 11.11.11.50@tcp timed out: rc -107
Jun 22 11:01:42 node97 kernel: LustreError: Skipped 18 previous similar messages
Jun 22 11:01:48 node97 kernel: LustreError: 7579:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-11.11.11.50@tcp: -113
Jun 22 11:01:48 node97 kernel: LustreError: 7579:0:(lib-move.c:2441:LNetPut()) Skipped 15 previous similar messages
Jun 22 11:01:48 node97 kernel: LustreError: 7579:0:(events.c:66:request_out_callback()) @@@ type 4, status -113  req@ffff810204a38800 x1371858245797847/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711715 ref 2 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:48 node97 kernel: LustreError: 7579:0:(events.c:66:request_out_callback()) Skipped 15 previous similar messages
Jun 22 11:01:57 node97 kernel: Lustre: 3964:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1371858245797879 sent from lustre-OST0003 to NID 11.11.11.50@tcp 0s ago has failed due to network error (7s prior to deadline).
Jun 22 11:01:57 node97 kernel:   req@ffff81005a36c800 x1371858245797879/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 1308711724 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 11:01:57 node97 kernel: Lustre: 3964:0:(client.c:1463:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
Jun 22 11:02:00 node97 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 11.11.11.50@tcp was evicted due to a lock blocking callback to 11.11.11.50@tcp timed out: rc -107
Jun 22 11:02:00 node97 kernel: LustreError: Skipped 27 previous similar messages
Jun 22 11:02:05 node97 kernel: LustreError: 7481:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8102047a2800 x1371858245797911/t0 o104->@:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jun 22 11:02:05 node97 kernel: LustreError: 7481:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 5 previous similar messages
Jun 22 11:02:05 node97 kernel: LustreError: 7481:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) ### client (nid 11.11.11.50@tcp) returned 0 from blocking AST ns: filter-lustre-OST0003_UUID lock: ffff8101f670b000/0x4ebf5f345588087e lrc: 4/0,0 mode: PR/PR res: 182169923/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0x9dab7525ec06021f expref: 8775 pid: 7739 timeout 4700584737
Jun 22 11:02:05 node97 kernel: LustreError: 7481:0:(ldlm_lockd.c:595:ldlm_handle_ast_error()) Skipped 5 previous similar messages
Jun 22 16:23:42 node97 kernel: Lustre: lustre-OST0000: slow setattr 40s due to heavy IO load
Jun 22 16:46:53 node97 syslogd 1.4.1: restart.
Jun 22 16:46:53 node97 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jun 22 16:46:53 node97 kernel: Linux version 2.6.18-164.11.1.el5_lustre.1.8.3 (lbuild@x86-build-1) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Fri Apr 9 18:00:39 MDT 2010
Jun 22 16:46:53 node97 kernel: Command line: ro root=LABEL=/ rhgb quiet
Jun 22 16:46:53 node97 kernel: BIOS-provided physical RAM map:
Jun 22 16:46:53 node97 kernel:  BIOS-e820: 0000000000010000 - 000000000009f400 (usable)
Jun 22 16:46:53 node97 kernel:  BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
Jun 22 16:46:53 node97 kernel:  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Jun 22 16:46:53 node97 kernel:  BIOS-e820: 0000000000100000 - 00000000cfe3e000 (usable)
Jun 22 16:46:53 node97 kernel:  BIOS-e820: 00000000cfe3e000 - 00000000cfe46000 (ACPI data)
Jun 22 16:46:53 node97 kernel:  BIOS-e820: 00000000cfe46000 - 00000000cfe47000 (usable)
Jun 22 16:46:53 node97 kernel:  BIOS-e820: 00000000cfe47000 - 00000000e0000000 (reserved)
Jun 22 16:46:53 node97 kernel:  BIOS-e820: 00000000fec00000 - 00000000fee10000 (reserved)
Jun 22 16:46:53 node97 kernel:  BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
Jun 22 16:46:53 node97 kernel:  BIOS-e820: 0000000100000000 - 000000022ffff000 (usable)
Jun 22 16:46:53 node97 kernel: DMI 2.4 present.
Jun 22 16:46:54 node97 rpc.statd[6883]: Version 1.0.9 Starting
Jun 22 16:46:54 node97 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jun 22 16:46:54 node97 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0
Jun 22 16:46:54 node97 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jun 22 16:46:54 node97 kernel: SRAT: PXM 0 -> APIC 3 -> Node 0
Jun 22 16:46:54 node97 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jun 22 16:46:54 node97 kernel: SRAT: PXM 0 -> APIC 5 -> Node 0
Jun 22 16:46:54 node97 kernel: SRAT: PXM 1 -> APIC 8 -> Node 1
Jun 22 16:46:54 node97 kernel: SRAT: PXM 1 -> APIC 9 -> Node 1
Jun 22 16:46:54 node97 kernel: SRAT: PXM 1 -> APIC 10 -> Node 1
Jun 22 16:46:54 node97 kernel: SRAT: PXM 1 -> APIC 11 -> Node 1
Jun 22 16:46:54 node97 kernel: SRAT: PXM 1 -> APIC 12 -> Node 1
Jun 22 16:46:54 node97 kernel: SRAT: PXM 1 -> APIC 13 -> Node 1
Jun 22 16:46:54 node97 kernel: SRAT: Node 0 PXM 0 0-a0000
Jun 22 16:46:54 node97 rpc.statd[6883]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user 
Jun 22 16:46:54 node97 kernel: SRAT: Node 0 PXM 0 0-d0000000
Jun 22 16:46:54 node97 kernel: SRAT: Node 0 PXM 0 0-130000000
Jun 22 16:46:54 node97 kernel: SRAT: Node 1 PXM 1 130000000-230000000
Jun 22 16:46:54 node97 kernel: Bootmem setup node 0 0000000000000000-0000000130000000
Jun 22 16:46:54 node97 kernel: Bootmem setup node 1 0000000130000000-000000022ffff000
Jun 22 16:46:54 node97 kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range
Jun 22 16:46:54 node97 kernel: disabling kdump
Jun 22 16:46:54 node97 kernel: ACPI: PM-Timer IO Port: 0x920
Jun 22 16:46:54 node97 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Jun 22 16:46:54 node97 kernel: Processor #0 0:8 APIC version 16
Jun 22 16:46:54 node97 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x08] enabled)
Jun 22 16:46:55 node97 kernel: Processor #8 0:8 APIC version 16
Jun 22 16:46:55 node97 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Jun 22 16:46:55 node97 kernel: Processor #1 0:8 APIC version 16
Jun 22 16:46:55 node97 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x09] enabled)
Jun 22 16:46:55 node97 kernel: Processor #9 0:8 APIC version 16
Jun 22 16:46:55 node97 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Jun 22 16:46:55 node97 kernel: Processor #2 0:8 APIC version 16
Jun 22 16:46:55 node97 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x0a] enabled)
Jun 22 16:46:55 node97 kernel: Processor #10 0:8 APIC version 16
Jun 22 16:46:55 node97 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
Jun 22 16:46:55 node97 kernel: Processor #3 0:8 APIC version 16
Jun 22 16:46:55 node97 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x0b] enabled)
Jun 22 16:46:55 node97 kernel: Processor #11 0:8 APIC version 16
Jun 22 16:46:55 node97 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
Jun 22 16:46:55 node97 kernel: Processor #4 0:8 APIC version 16
Jun 22 16:46:55 node97 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0c] enabled)
Jun 22 16:46:55 node97 kernel: Processor #12 0:8 APIC version 16
Jun 22 16:46:55 node97 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled)
Jun 22 16:46:55 node97 kernel: Processor #5 0:8 APIC version 16
Jun 22 16:46:55 node97 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0d] enabled)
Jun 22 16:46:55 node97 kernel: Processor #13 0:8 APIC version 16
Jun 22 16:46:55 node97 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jun 22 16:46:56 node97 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
Jun 22 16:46:56 node97 kernel: IOAPIC[0]: apic_id 8, version 17, address 0xfec00000, GSI 0-15
Jun 22 16:46:56 node97 kernel: ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[16])
Jun 22 16:46:56 node97 kernel: IOAPIC[1]: apic_id 9, version 17, address 0xfec01000, GSI 16-31
Jun 22 16:46:56 node97 kernel: ACPI: IOAPIC (id[0x0a] address[0xfec02000] gsi_base[32])
Jun 22 16:46:56 node97 kernel: IOAPIC[2]: apic_id 10, version 17, address 0xfec02000, GSI 32-47
Jun 22 16:46:56 node97 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
Jun 22 16:46:56 node97 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
Jun 22 16:46:56 node97 kernel: Setting APIC routing to physical flat
Jun 22 16:46:56 node97 kernel: ACPI: HPET id: 0x1166a201 base: 0xfed00000
Jun 22 16:46:56 node97 kernel: Using ACPI (MADT) for SMP configuration information
Jun 22 16:46:56 node97 kernel: Nosave address range: 000000000009f000 - 00000000000a0000
Jun 22 16:46:56 node97 kernel: Nosave address range: 00000000000a0000 - 00000000000f0000
Jun 22 16:46:56 node97 kernel: Nosave address range: 00000000000f0000 - 0000000000100000
Jun 22 16:46:56 node97 kernel: Nosave address range: 00000000cfe3e000 - 00000000cfe46000
Jun 22 16:46:56 node97 kernel: Nosave address range: 00000000cfe47000 - 00000000e0000000
Jun 22 16:46:57 node97 kernel: Nosave address range: 00000000e0000000 - 00000000fec00000
Jun 22 16:46:57 node97 kernel: Nosave address range: 00000000fec00000 - 00000000fee10000
Jun 22 16:46:57 node97 kernel: Nosave address range: 00000000fee10000 - 00000000ffc00000
Jun 22 16:46:57 node97 kernel: Nosave address range: 00000000ffc00000 - 0000000100000000
Jun 22 16:46:57 node97 kernel: Allocating PCI resources starting at e2000000 (gap: e0000000:1ec00000)
Jun 22 16:46:57 node97 kernel: SMP: Allowing 12 CPUs, 0 hotplug CPUs
Jun 22 16:46:57 node97 kernel: Built 2 zonelists.  Total pages: 2063935
Jun 22 16:46:57 node97 kernel: Kernel command line: ro root=LABEL=/ rhgb quiet
Jun 22 16:46:57 node97 kernel: Initializing CPU#0
Jun 22 16:46:57 node97 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)
Jun 22 16:46:57 node97 kernel: Console: colour VGA+ 80x25
Jun 22 16:46:57 node97 kernel: Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Jun 22 16:46:57 node97 kernel: Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jun 22 16:46:57 node97 kernel: Checking aperture...
Jun 22 16:46:57 node97 kernel: CPU 0: aperture @ 8c14000000 size 32 MB
Jun 22 16:46:57 node97 kernel: Aperture too small (32 MB)
Jun 22 16:46:57 node97 kernel: No AGP bridge found
Jun 22 16:46:57 node97 kernel: Your BIOS doesn't leave a aperture memory hole
Jun 22 16:46:57 node97 kernel: Please enable the IOMMU option in the BIOS setup
Jun 22 16:46:57 node97 kernel: This costs you 64 MB of RAM
Jun 22 16:46:57 node97 kernel: Mapping aperture over 65536 KB of RAM @ 4000000
Jun 22 16:46:57 node97 kernel: Nosave address range: 0000000004000000 - 0000000008000000
Jun 22 16:46:57 node97 kernel: ACPI: DMAR not present
Jun 22 16:46:57 node97 kernel: Memory: 8174352k/9175036k available (2553k kernel code, 212004k reserved, 1293k data, 208k init)
Jun 22 16:46:57 node97 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 5200.18 BogoMIPS (lpj=2600090)
Jun 22 16:46:58 node97 kernel: Security Framework v1.0.0 initialized
Jun 22 16:46:58 node97 kernel: SELinux:  Initializing.
Jun 22 16:46:58 node97 kernel: selinux_register_security:  Registering secondary module capability
Jun 22 16:46:58 node97 kernel: Capability LSM initialized as secondary
Jun 22 16:46:58 node97 kernel: Mount-cache hash table entries: 256
Jun 22 16:46:58 node97 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Jun 22 16:46:58 node97 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Jun 22 16:46:58 node97 kernel: CPU 0/0 -> Node 0
Jun 22 16:46:58 node97 kernel: CPU: Physical Processor ID: 0
Jun 22 16:46:58 node97 kernel: CPU: Processor Core ID: 0
Jun 22 16:46:58 node97 kernel: SMP alternatives: switching to UP code
Jun 22 16:46:58 node97 kernel: ACPI: Core revision 20060707
Jun 22 16:46:58 node97 kernel: Using local APIC timer interrupts.
Jun 22 16:46:58 node97 kernel: Detected 12.500 MHz APIC timer.
Jun 22 16:46:58 node97 kernel: SMP alternatives: switching to SMP code
Jun 22 16:46:58 node97 kernel: Booting processor 1/12 APIC 0x8
Jun 22 16:46:58 node97 kernel: Initializing CPU#1
Jun 22 16:46:58 node97 kernel: Calibrating delay using timer specific routine.. 5200.25 BogoMIPS (lpj=2600128)
Jun 22 16:46:58 node97 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Jun 22 16:46:58 node97 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Jun 22 16:46:58 node97 kernel: CPU 1/8 -> Node 1
Jun 22 16:46:58 node97 kernel: CPU: Physical Processor ID: 1
Jun 22 16:46:58 node97 kernel: CPU: Processor Core ID: 0
Jun 22 16:46:58 node97 kernel: Six-Core AMD Opteron(tm) Processor 2435 stepping 00
Jun 22 16:46:58 node97 kernel: SMP alternatives: switching to SMP code
Jun 22 16:46:58 node97 kernel: Booting processor 2/12 APIC 0x1
Jun 22 16:46:58 node97 kernel: Initializing CPU#2
Jun 22 16:46:58 node97 kernel: Calibrating delay using timer specific routine.. 5096.09 BogoMIPS (lpj=2548046)
Jun 22 16:46:58 node97 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Jun 22 16:46:58 node97 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Jun 22 16:46:58 node97 kernel: CPU 2/1 -> Node 0
Jun 22 16:46:59 node97 kernel: CPU: Physical Processor ID: 0
Jun 22 16:46:59 node97 kernel: CPU: Processor Core ID: 1
Jun 22 16:46:59 node97 kernel: Six-Core AMD Opteron(tm) Processor 2435 stepping 00
Jun 22 16:46:59 node97 kernel: SMP alternatives: switching to SMP code
Jun 22 16:46:59 node97 kernel: Booting processor 3/12 APIC 0x9
Jun 22 16:46:59 node97 kernel: Initializing CPU#3
Jun 22 16:46:59 node97 kernel: Calibrating delay using timer specific routine.. 5204.51 BogoMIPS (lpj=2602256)
Jun 22 16:46:59 node97 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Jun 22 16:46:59 node97 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Jun 22 16:46:59 node97 kernel: CPU 3/9 -> Node 1
Jun 22 16:46:59 node97 kernel: CPU: Physical Processor ID: 1
Jun 22 16:46:59 node97 kernel: CPU: Processor Core ID: 1
Jun 22 16:46:59 node97 kernel: Six-Core AMD Opteron(tm) Processor 2435 stepping 00
Jun 22 16:46:59 node97 kernel: SMP alternatives: switching to SMP code
Jun 22 16:46:59 node97 kernel: Booting processor 4/12 APIC 0x2
Jun 22 16:46:59 node97 kernel: Initializing CPU#4
Jun 22 16:46:59 node97 kernel: Calibrating delay using timer specific routine.. 5203.67 BogoMIPS (lpj=2601838)
Jun 22 16:46:59 node97 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Jun 22 16:46:59 node97 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Jun 22 16:46:59 node97 kernel: CPU 4/2 -> Node 0
Jun 22 16:46:59 node97 kernel: CPU: Physical Processor ID: 0
Jun 22 16:46:59 node97 kernel: CPU: Processor Core ID: 2
Jun 22 16:46:59 node97 kernel: Six-Core AMD Opteron(tm) Processor 2435 stepping 00
Jun 22 16:46:59 node97 kernel: SMP alternatives: switching to SMP code
Jun 22 16:46:59 node97 kernel: Booting processor 5/12 APIC 0xa
Jun 22 16:46:59 node97 kernel: Initializing CPU#5
Jun 22 16:46:59 node97 kernel: Calibrating delay using timer specific routine.. 5203.42 BogoMIPS (lpj=2601714)
Jun 22 16:46:59 node97 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Jun 22 16:46:59 node97 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Jun 22 16:46:59 node97 kernel: CPU 5/a -> Node 1
Jun 22 16:46:59 node97 kernel: CPU: Physical Processor ID: 1
Jun 22 16:46:59 node97 kernel: CPU: Processor Core ID: 2
Jun 22 16:46:59 node97 kernel: Six-Core AMD Opteron(tm) Processor 2435 stepping 00
Jun 22 16:46:59 node97 kernel: SMP alternatives: switching to SMP code
Jun 22 16:47:00 node97 kernel: Booting processor 6/12 APIC 0x3
Jun 22 16:47:00 node97 kernel: Initializing CPU#6
Jun 22 16:47:00 node97 kernel: Calibrating delay using timer specific routine.. 5171.00 BogoMIPS (lpj=2585502)
Jun 22 16:47:00 node97 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Jun 22 16:47:00 node97 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Jun 22 16:47:00 node97 kernel: CPU 6/3 -> Node 0
Jun 22 16:47:00 node97 kernel: CPU: Physical Processor ID: 0
Jun 22 16:47:00 node97 kernel: CPU: Processor Core ID: 3
Jun 22 16:47:00 node97 kernel: Six-Core AMD Opteron(tm) Processor 2435 stepping 00
Jun 22 16:47:00 node97 kernel: SMP alternatives: switching to SMP code
Jun 22 16:47:00 node97 kernel: Booting processor 7/12 APIC 0xb
Jun 22 16:47:00 node97 kernel: Initializing CPU#7
Jun 22 16:47:00 node97 kernel: Calibrating delay using timer specific routine.. 5201.64 BogoMIPS (lpj=2600823)
Jun 22 16:47:00 node97 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Jun 22 16:47:00 node97 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Jun 22 16:47:00 node97 kernel: CPU 7/b -> Node 1
Jun 22 16:47:00 node97 kernel: CPU: Physical Processor ID: 1
Jun 22 16:47:00 node97 kernel: CPU: Processor Core ID: 3
Jun 22 16:47:00 node97 kernel: Six-Core AMD Opteron(tm) Processor 2435 stepping 00
Jun 22 16:47:00 node97 kernel: SMP alternatives: switching to SMP code
Jun 22 16:47:00 node97 kernel: Booting processor 8/12 APIC 0x4
Jun 22 16:47:00 node97 kernel: Initializing CPU#8
Jun 22 16:47:00 node97 kernel: Calibrating delay using timer specific routine.. 5202.15 BogoMIPS (lpj=2601077)
Jun 22 16:47:00 node97 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Jun 22 16:47:00 node97 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Jun 22 16:47:00 node97 kernel: CPU 8/4 -> Node 0
Jun 22 16:47:00 node97 kernel: CPU: Physical Processor ID: 0
Jun 22 16:47:00 node97 kernel: CPU: Processor Core ID: 4
Jun 22 16:47:00 node97 kernel: Six-Core AMD Opteron(tm) Processor 2435 stepping 00
Jun 22 16:47:00 node97 kernel: SMP alternatives: switching to SMP code
Jun 22 16:47:00 node97 kernel: Booting processor 9/12 APIC 0xc
Jun 22 16:47:00 node97 kernel: Initializing CPU#9
Jun 22 16:47:01 node97 kernel: Calibrating delay using timer specific routine.. 5201.70 BogoMIPS (lpj=2600850)
Jun 22 16:47:01 node97 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Jun 22 16:47:01 node97 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Jun 22 16:47:01 node97 kernel: CPU 9/c -> Node 1
Jun 22 16:47:01 node97 kernel: CPU: Physical Processor ID: 1
Jun 22 16:47:01 node97 kernel: CPU: Processor Core ID: 4
Jun 22 16:47:01 node97 kernel: Six-Core AMD Opteron(tm) Processor 2435 stepping 00
Jun 22 16:47:01 node97 kernel: SMP alternatives: switching to SMP code
Jun 22 16:47:01 node97 kernel: Booting processor 10/12 APIC 0x5
Jun 22 16:47:01 node97 kernel: Initializing CPU#10
Jun 22 16:47:01 node97 kernel: Calibrating delay using timer specific routine.. 5203.66 BogoMIPS (lpj=2601833)
Jun 22 16:47:01 node97 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Jun 22 16:47:01 node97 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Jun 22 16:47:01 node97 kernel: CPU 10/5 -> Node 0
Jun 22 16:47:01 node97 kernel: CPU: Physical Processor ID: 0
Jun 22 16:47:01 node97 kernel: CPU: Processor Core ID: 5
Jun 22 16:47:01 node97 kernel: Six-Core AMD Opteron(tm) Processor 2435 stepping 00
Jun 22 16:47:01 node97 kernel: SMP alternatives: switching to SMP code
Jun 22 16:47:01 node97 kernel: Booting processor 11/12 APIC 0xd
Jun 22 16:47:01 node97 kernel: Initializing CPU#11
Jun 22 16:47:01 node97 kernel: Calibrating delay using timer specific routine.. 5203.55 BogoMIPS (lpj=2601779)
Jun 22 16:47:01 node97 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Jun 22 16:47:01 node97 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Jun 22 16:47:01 node97 kernel: CPU 11/d -> Node 1
Jun 22 16:47:01 node97 kernel: CPU: Physical Processor ID: 1
Jun 22 16:47:01 node97 kernel: CPU: Processor Core ID: 5
Jun 22 16:47:01 node97 kernel: Six-Core AMD Opteron(tm) Processor 2435 stepping 00
Jun 22 16:47:01 node97 kernel: Brought up 12 CPUs
Jun 22 16:47:01 node97 kernel: testing NMI watchdog ... OK.
Jun 22 16:47:02 node97 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer.
Jun 22 16:47:02 node97 kernel: time.c: Detected 2600.104 MHz processor.
Jun 22 16:47:02 node97 kernel: migration_cost=184,146
Jun 22 16:47:02 node97 kernel: checking if image is initramfs... it is
Jun 22 16:47:02 node97 kernel: Freeing initrd memory: 2611k freed
Jun 22 16:47:02 node97 kernel: NET: Registered protocol family 16
Jun 22 16:47:02 node97 kernel: ACPI: bus type pci registered
Jun 22 16:47:02 node97 kernel: PCI: Using MMCONFIG at d0000000
Jun 22 16:47:02 node97 kernel: ACPI: Interpreter enabled
Jun 22 16:47:02 node97 kernel: ACPI: Using IOAPIC for interrupt routing
Jun 22 16:47:02 node97 kernel: ACPI: No dock devices found.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)
Jun 22 16:47:02 node97 kernel: PCI: Enabling HT MSI Mapping on 0000:00:05.0
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IUSB] (IRQs *5)
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [ISF0] (IRQs *14)
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN00] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN01] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN02] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN03] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN04] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN05] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN06] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN07] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN08] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN09] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN10] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN11] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN12] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN13] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN14] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:02 node97 kernel: ACPI: PCI Interrupt Link [IN15] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN16] (IRQs 7 10 *11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN17] (IRQs 7 10 *11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN18] (IRQs 7 10 *11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN19] (IRQs 7 10 *11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN20] (IRQs 7 10 *11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN21] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN22] (IRQs 7 *10 11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN23] (IRQs *7 10 11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN24] (IRQs 7 10 *11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN25] (IRQs 7 10 *11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN26] (IRQs 7 10 *11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN27] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN28] (IRQs *7 10 11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN29] (IRQs 7 10 *11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN30] (IRQs 7 *10 11)
Jun 22 16:47:03 node97 kernel: ACPI: PCI Interrupt Link [IN31] (IRQs 7 10 11) *0, disabled.
Jun 22 16:47:03 node97 kernel: ACPI: PCI Root Bridge [PCI1] (0000:40)
Jun 22 16:47:03 node97 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
Jun 22 16:47:03 node97 kernel: pnp: PnP ACPI init
Jun 22 16:47:03 node97 kernel: pnp: PnP ACPI: found 12 devices
Jun 22 16:47:03 node97 kernel: usbcore: registered new driver usbfs
Jun 22 16:47:03 node97 kernel: usbcore: registered new driver hub
Jun 22 16:47:03 node97 kernel: PCI: Using ACPI for IRQ routing
Jun 22 16:47:03 node97 kernel: PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
Jun 22 16:47:03 node97 kernel: NetLabel: Initializing
Jun 22 16:47:03 node97 kernel: NetLabel:  domain hash size = 128
Jun 22 16:47:03 node97 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jun 22 16:47:03 node97 kernel: NetLabel:  unlabeled traffic allowed by default
Jun 22 16:47:03 node97 kernel: hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0
Jun 22 16:47:03 node97 kernel: hpet0: 3 64-bit timers, 14318180 Hz
Jun 22 16:47:03 node97 kernel: ACPI: DMAR not present
Jun 22 16:47:03 node97 kernel: PCI-DMA: Disabling AGP.
Jun 22 16:47:03 node97 kernel: PCI-DMA: aperture base @ 4000000 size 65536 KB
Jun 22 16:47:03 node97 kernel: PCI-DMA: using GART IOMMU.
Jun 22 16:47:04 node97 kernel: PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
Jun 22 16:47:04 node97 kernel: pnp: 00:01: ioport range 0x408-0x40f has been reserved
Jun 22 16:47:04 node97 kernel: pnp: 00:01: ioport range 0x4d0-0x4d1 has been reserved
Jun 22 16:47:04 node97 kernel: pnp: 00:01: ioport range 0x4d6-0x4d6 has been reserved
Jun 22 16:47:04 node97 kernel: PCI: Bridge: 0000:01:0d.0
Jun 22 16:47:04 node97 kernel:   IO window: disabled.
Jun 22 16:47:04 node97 kernel:   MEM window: disabled.
Jun 22 16:47:04 node97 kernel:   PREFETCH window: disabled.
Jun 22 16:47:04 node97 kernel: PCI: Bridge: 0000:00:05.0
Jun 22 16:47:04 node97 kernel:   IO window: 4000-4fff
Jun 22 16:47:04 node97 kernel:   MEM window: f3f00000-f3ffffff
Jun 22 16:47:04 node97 kernel:   PREFETCH window 0x00000000e2000000-0x00000000e20fffff
Jun 22 16:47:04 node97 kernel: PCI: Bridge: 0000:00:0f.0
Jun 22 16:47:04 node97 kernel:   IO window: disabled.
Jun 22 16:47:04 node97 kernel:   MEM window: disabled.
Jun 22 16:47:04 node97 kernel:   PREFETCH window: disabled.
Jun 22 16:47:04 node97 kernel: PCI: Bridge: 0000:00:10.0
Jun 22 16:47:04 node97 kernel:   IO window: disabled.
Jun 22 16:47:04 node97 kernel:   MEM window: disabled.
Jun 22 16:47:04 node97 kernel:   PREFETCH window: disabled.
Jun 22 16:47:04 node97 kernel: PCI: Bridge: 0000:00:11.0
Jun 22 16:47:04 node97 kernel:   IO window: disabled.
Jun 22 16:47:04 node97 kernel:   MEM window: disabled.
Jun 22 16:47:04 node97 kernel:   PREFETCH window: disabled.
Jun 22 16:47:04 node97 kernel: PCI: Bridge: 0000:00:12.0
Jun 22 16:47:04 node97 kernel:   IO window: disabled.
Jun 22 16:47:04 node97 kernel:   MEM window: f4000000-f7ffffff
Jun 22 16:47:04 node97 kernel:   PREFETCH window 0x00000000e2100000-0x00000000e21fffff
Jun 22 16:47:04 node97 kernel: PCI: Bridge: 0000:00:13.0
Jun 22 16:47:04 node97 kernel:   IO window: disabled.
Jun 22 16:47:04 node97 kernel:   MEM window: disabled.
Jun 22 16:47:04 node97 kernel:   PREFETCH window: disabled.
Jun 22 16:47:04 node97 kernel: GSI 16 sharing vector 0xA9 and IRQ 16
Jun 22 16:47:04 node97 kernel: ACPI: PCI Interrupt 0000:00:0f.0[A] -> GSI 42 (level, low) -> IRQ 169
Jun 22 16:47:05 node97 kernel: GSI 17 sharing vector 0xB1 and IRQ 17
Jun 22 16:47:05 node97 kernel: ACPI: PCI Interrupt 0000:00:10.0[A] -> GSI 38 (level, low) -> IRQ 177
Jun 22 16:47:05 node97 kernel: GSI 18 sharing vector 0xB9 and IRQ 18
Jun 22 16:47:05 node97 kernel: ACPI: PCI Interrupt 0000:00:11.0[A] -> GSI 39 (level, low) -> IRQ 185
Jun 22 16:47:05 node97 kernel: GSI 19 sharing vector 0xC1 and IRQ 19
Jun 22 16:47:05 node97 kernel: ACPI: PCI Interrupt 0000:00:12.0[A] -> GSI 40 (level, low) -> IRQ 193
Jun 22 16:47:05 node97 kernel: GSI 20 sharing vector 0xC9 and IRQ 20
Jun 22 16:47:05 node97 kernel: ACPI: PCI Interrupt 0000:00:13.0[A] -> GSI 41 (level, low) -> IRQ 201
Jun 22 16:47:05 node97 kernel: PCI: Bridge: 0000:40:0f.0
Jun 22 16:47:05 node97 kernel:   IO window: disabled.
Jun 22 16:47:05 node97 kernel:   MEM window: disabled.
Jun 22 16:47:05 node97 kernel:   PREFETCH window: disabled.
Jun 22 16:47:05 node97 kernel: PCI: Bridge: 0000:40:10.0
Jun 22 16:47:05 node97 kernel:   IO window: 5000-5fff
Jun 22 16:47:05 node97 kernel:   MEM window: fdc00000-fdcfffff
Jun 22 16:47:05 node97 kernel:   PREFETCH window 0x00000000e2300000-0x00000000e23fffff
Jun 22 16:47:05 node97 kernel: PCI: Bridge: 0000:40:11.0
Jun 22 16:47:05 node97 kernel:   IO window: 6000-6fff
Jun 22 16:47:05 node97 kernel:   MEM window: fdd00000-fdffffff
Jun 22 16:47:05 node97 kernel:   PREFETCH window 0x00000000e2400000-0x00000000e24fffff
Jun 22 16:47:05 node97 kernel: PCI: Bridge: 0000:40:12.0
Jun 22 16:47:05 node97 kernel:   IO window: disabled.
Jun 22 16:47:05 node97 kernel:   MEM window: f8000000-fbffffff
Jun 22 16:47:05 node97 kernel:   PREFETCH window 0x00000000e2500000-0x00000000e25fffff
Jun 22 16:47:05 node97 kernel: PCI: Bridge: 0000:40:13.0
Jun 22 16:47:05 node97 kernel:   IO window: disabled.
Jun 22 16:47:05 node97 kernel:   MEM window: disabled.
Jun 22 16:47:05 node97 kernel:   PREFETCH window: disabled.
Jun 22 16:47:05 node97 kernel: GSI 21 sharing vector 0xD1 and IRQ 21
Jun 22 16:47:05 node97 kernel: ACPI: PCI Interrupt 0000:40:0f.0[A] -> GSI 36 (level, low) -> IRQ 209
Jun 22 16:47:05 node97 kernel: GSI 22 sharing vector 0xD9 and IRQ 22
Jun 22 16:47:05 node97 kernel: ACPI: PCI Interrupt 0000:40:10.0[A] -> GSI 32 (level, low) -> IRQ 217
Jun 22 16:47:06 node97 kernel: GSI 23 sharing vector 0xE1 and IRQ 23
Jun 22 16:47:06 node97 kernel: ACPI: PCI Interrupt 0000:40:11.0[A] -> GSI 33 (level, low) -> IRQ 225
Jun 22 16:47:06 node97 kernel: GSI 24 sharing vector 0xE9 and IRQ 24
Jun 22 16:47:06 node97 kernel: ACPI: PCI Interrupt 0000:40:12.0[A] -> GSI 34 (level, low) -> IRQ 233
Jun 22 16:47:06 node97 kernel: GSI 25 sharing vector 0x32 and IRQ 25
Jun 22 16:47:06 node97 kernel: ACPI: PCI Interrupt 0000:40:13.0[A] -> GSI 35 (level, low) -> IRQ 50
Jun 22 16:47:06 node97 kernel: NET: Registered protocol family 2
Jun 22 16:47:06 node97 kernel: IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
Jun 22 16:47:06 node97 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
Jun 22 16:47:06 node97 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jun 22 16:47:06 node97 multipathd: dm-0: umount map (uevent) 
Jun 22 16:47:06 node97 kernel: TCP: Hash tables configured (established 262144 bind 65536)
Jun 22 16:47:06 node97 kernel: TCP reno registered
Jun 22 16:47:06 node97 kernel: audit: initializing netlink socket (disabled)
Jun 22 16:47:06 node97 kernel: type=2000 audit(1308761161.586:1): initialized
Jun 22 16:47:06 node97 kernel: Total HugeTLB memory allocated, 0
Jun 22 16:47:06 node97 kernel: VFS: Disk quotas dquot_6.5.1
Jun 22 16:47:06 node97 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jun 22 16:47:06 node97 kernel: Initializing Cryptographic API
Jun 22 16:47:06 node97 kernel: alg: No test for crc32c (crc32c-generic)
Jun 22 16:47:06 node97 kernel: ksign: Installing public key data
Jun 22 16:47:06 node97 kernel: Loading keyring
Jun 22 16:47:06 node97 kernel: io scheduler noop registered
Jun 22 16:47:06 node97 kernel: io scheduler anticipatory registered
Jun 22 16:47:06 node97 kernel: io scheduler deadline registered (default)
Jun 22 16:47:06 node97 kernel: io scheduler cfq registered
Jun 22 16:47:06 node97 kernel: pci 0000:00:04.4: HCRESET not completed yet!
Jun 22 16:47:06 node97 kernel: ACPI: PCI Interrupt 0000:00:0f.0[A] -> GSI 42 (level, low) -> IRQ 169
Jun 22 16:47:06 node97 kernel: assign_interrupt_mode Found MSI capability
Jun 22 16:47:07 node97 kernel: ACPI: PCI Interrupt 0000:00:10.0[A] -> GSI 38 (level, low) -> IRQ 177
Jun 22 16:47:07 node97 kernel: assign_interrupt_mode Found MSI capability
Jun 22 16:47:07 node97 kernel: ACPI: PCI Interrupt 0000:00:11.0[A] -> GSI 39 (level, low) -> IRQ 185
Jun 22 16:47:07 node97 kernel: assign_interrupt_mode Found MSI capability
Jun 22 16:47:07 node97 kernel: ACPI: PCI Interrupt 0000:00:12.0[A] -> GSI 40 (level, low) -> IRQ 193
Jun 22 16:47:07 node97 kernel: assign_interrupt_mode Found MSI capability
Jun 22 16:47:07 node97 kernel: ACPI: PCI Interrupt 0000:00:13.0[A] -> GSI 41 (level, low) -> IRQ 201
Jun 22 16:47:07 node97 kernel: assign_interrupt_mode Found MSI capability
Jun 22 16:47:07 node97 kernel: ACPI: PCI Interrupt 0000:40:0f.0[A] -> GSI 36 (level, low) -> IRQ 209
Jun 22 16:47:07 node97 kernel: assign_interrupt_mode Found MSI capability
Jun 22 16:47:07 node97 kernel: ACPI: PCI Interrupt 0000:40:10.0[A] -> GSI 32 (level, low) -> IRQ 217
Jun 22 16:47:07 node97 kernel: assign_interrupt_mode Found MSI capability
Jun 22 16:47:07 node97 kernel: ACPI: PCI Interrupt 0000:40:11.0[A] -> GSI 33 (level, low) -> IRQ 225
Jun 22 16:47:07 node97 kernel: assign_interrupt_mode Found MSI capability
Jun 22 16:47:07 node97 kernel: ACPI: PCI Interrupt 0000:40:12.0[A] -> GSI 34 (level, low) -> IRQ 233
Jun 22 16:47:07 node97 kernel: assign_interrupt_mode Found MSI capability
Jun 22 16:47:07 node97 kernel: ACPI: PCI Interrupt 0000:40:13.0[A] -> GSI 35 (level, low) -> IRQ 50
Jun 22 16:47:07 node97 kernel: assign_interrupt_mode Found MSI capability
Jun 22 16:47:07 node97 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jun 22 16:47:07 node97 kernel: Real Time Clock Driver v1.12ac
Jun 22 16:47:07 node97 kernel: Non-volatile memory driver v1.2
Jun 22 16:47:07 node97 kernel: Linux agpgart interface v0.101 (c) Dave Jones
Jun 22 16:47:07 node97 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
Jun 22 16:47:07 node97 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jun 22 16:47:07 node97 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jun 22 16:47:07 node97 kernel: 00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jun 22 16:47:07 node97 kernel: brd: module loaded
Jun 22 16:47:07 node97 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
Jun 22 16:47:07 node97 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Jun 22 16:47:07 node97 kernel: ide-floppy driver 0.99.newide
Jun 22 16:47:07 node97 kernel: usbcore: registered new driver hiddev
Jun 22 16:47:07 node97 kernel: usbcore: registered new driver usbhid
Jun 22 16:47:07 node97 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver
Jun 22 16:47:08 node97 kernel: PNP: PS/2 Controller [PNP0303:KBD,PNP0f0e:PS2M] at 0x60,0x64 irq 1,12
Jun 22 16:47:08 node97 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
Jun 22 16:47:08 node97 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12
Jun 22 16:47:08 node97 kernel: mice: PS/2 mouse device common for all mice
Jun 22 16:47:08 node97 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
Jun 22 16:47:08 node97 kernel: md: bitmap version 4.39
Jun 22 16:47:08 node97 kernel: TCP bic registered
Jun 22 16:47:08 node97 kernel: Initializing IPsec netlink socket
Jun 22 16:47:08 node97 kernel: NET: Registered protocol family 1
Jun 22 16:47:08 node97 kernel: NET: Registered protocol family 17
Jun 22 16:47:08 node97 kernel: ACPI: (supports S0 S4 S5)
Jun 22 16:47:08 node97 kernel: Initalizing network drop monitor service
Jun 22 16:47:08 node97 kernel: Freeing unused kernel memory: 208k freed
Jun 22 16:47:08 node97 kernel: Write protecting the kernel read-only data: 498k
Jun 22 16:47:08 node97 kernel: ACPI: PCI Interrupt Link [IUSB] enabled at IRQ 5
Jun 22 16:47:08 node97 kernel: ACPI: PCI Interrupt 0000:00:07.2[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5
Jun 22 16:47:08 node97 kernel: ehci_hcd 0000:00:07.2: EHCI Host Controller
Jun 22 16:47:08 node97 kernel: ehci_hcd 0000:00:07.2: new USB bus registered, assigned bus number 1
Jun 22 16:47:08 node97 kernel: ehci_hcd 0000:00:07.2: irq 5, io mem 0xf3dc0000
Jun 22 16:47:08 node97 kernel: ehci_hcd 0000:00:07.2: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
Jun 22 16:47:08 node97 kernel: usb usb1: configuration #1 chosen from 1 choice
Jun 22 16:47:08 node97 kernel: hub 1-0:1.0: USB hub found
Jun 22 16:47:08 node97 kernel: hub 1-0:1.0: 4 ports detected
Jun 22 16:47:08 node97 kernel: ACPI: PCI Interrupt 0000:00:07.0[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5
Jun 22 16:47:08 node97 kernel: ohci_hcd 0000:00:07.0: OHCI Host Controller
Jun 22 16:47:08 node97 kernel: ohci_hcd 0000:00:07.0: new USB bus registered, assigned bus number 2
Jun 22 16:47:08 node97 kernel: ohci_hcd 0000:00:07.0: irq 5, io mem 0xf3de0000
Jun 22 16:47:08 node97 kernel: usb usb2: configuration #1 chosen from 1 choice
Jun 22 16:47:08 node97 kernel: hub 2-0:1.0: USB hub found
Jun 22 16:47:08 node97 kernel: hub 2-0:1.0: 2 ports detected
Jun 22 16:47:08 node97 kernel: ACPI: PCI Interrupt 0000:00:07.1[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5
Jun 22 16:47:08 node97 kernel: ohci_hcd 0000:00:07.1: OHCI Host Controller
Jun 22 16:47:08 node97 kernel: ohci_hcd 0000:00:07.1: new USB bus registered, assigned bus number 3
Jun 22 16:47:08 node97 kernel: ohci_hcd 0000:00:07.1: irq 5, io mem 0xf3dd0000
Jun 22 16:47:09 node97 kernel: usb usb3: configuration #1 chosen from 1 choice
Jun 22 16:47:09 node97 kernel: hub 3-0:1.0: USB hub found
Jun 22 16:47:09 node97 kernel: hub 3-0:1.0: 2 ports detected
Jun 22 16:47:09 node97 kernel: USB Universal Host Controller Interface driver v3.0
Jun 22 16:47:09 node97 kernel: GSI 26 sharing vector 0x92 and IRQ 26
Jun 22 16:47:09 node97 kernel: ACPI: PCI Interrupt 0000:00:04.4[B] -> GSI 45 (level, low) -> IRQ 146
Jun 22 16:47:09 node97 kernel: uhci_hcd 0000:00:04.4: UHCI Host Controller
Jun 22 16:47:09 node97 kernel: uhci_hcd 0000:00:04.4: new USB bus registered, assigned bus number 4
Jun 22 16:47:09 node97 kernel: uhci_hcd 0000:00:04.4: port count misdetected? forcing to 2 ports
Jun 22 16:47:09 node97 kernel: uhci_hcd 0000:00:04.4: HCRESET not completed yet!
Jun 22 16:47:09 node97 kernel: uhci_hcd 0000:00:04.4: irq 146, io base 0x00001800
Jun 22 16:47:09 node97 kernel: usb usb4: configuration #1 chosen from 1 choice
Jun 22 16:47:09 node97 kernel: hub 4-0:1.0: USB hub found
Jun 22 16:47:09 node97 kernel: hub 4-0:1.0: 2 ports detected
Jun 22 16:47:09 node97 kernel: SCSI subsystem initialized
Jun 22 16:47:09 node97 kernel: HP CISS Driver (v 3.6.20-RH3)
Jun 22 16:47:09 node97 kernel: ACPI: PCI Interrupt 0000:48:00.0[A] -> GSI 33 (level, low) -> IRQ 225
Jun 22 16:47:09 node97 kernel: cciss0: <0x323a> at PCI 0000:48:00.0 IRQ 170 using DAC
Jun 22 16:47:09 node97 kernel:       blocks= 286677120 block_size= 512
Jun 22 16:47:09 node97 kernel:       heads= 255, sectors= 32, cylinders= 35132
Jun 22 16:47:09 node97 kernel: 
Jun 22 16:47:09 node97 kernel:       blocks= 286677120 block_size= 512
Jun 22 16:47:09 node97 kernel:       heads= 255, sectors= 32, cylinders= 35132
Jun 22 16:47:09 node97 kernel: 
Jun 22 16:47:09 node97 kernel:  cciss/c0d0: p1 p2
Jun 22 16:47:09 node97 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Jun 22 16:47:09 node97 kernel: ACPI: PCI Interrupt Link [ISF0] enabled at IRQ 14
Jun 22 16:47:09 node97 kernel: ACPI: PCI Interrupt 0000:01:0e.0[A] -> Link [ISF0] -> GSI 14 (level, low) -> IRQ 14
Jun 22 16:47:09 node97 kernel: scsi0 : sata_svw
Jun 22 16:47:09 node97 kernel: scsi1 : sata_svw
Jun 22 16:47:09 node97 kernel: scsi2 : sata_svw
Jun 22 16:47:09 node97 kernel: scsi3 : sata_svw
Jun 22 16:47:10 node97 kernel: ata1: SATA max UDMA/133 mmio m8192@0xf3ff0000 port 0xf3ff0000 irq 14
Jun 22 16:47:10 node97 kernel: ata2: SATA max UDMA/133 mmio m8192@0xf3ff0000 port 0xf3ff0100 irq 14
Jun 22 16:47:10 node97 kernel: ata3: SATA max UDMA/133 mmio m8192@0xf3ff0000 port 0xf3ff0200 irq 14
Jun 22 16:47:10 node97 kernel: ata4: SATA max UDMA/133 mmio m8192@0xf3ff0000 port 0xf3ff0300 irq 14
Jun 22 16:47:10 node97 kernel: usb 3-1: new full speed USB device using ohci_hcd and address 2
Jun 22 16:47:10 node97 kernel: usb 3-1: not running at top speed; connect to a high speed hub
Jun 22 16:47:10 node97 kernel: usb 3-1: configuration #1 chosen from 1 choice
Jun 22 16:47:10 node97 kernel: hub 3-1:1.0: USB hub found
Jun 22 16:47:10 node97 kernel: hub 3-1:1.0: 4 ports detected
Jun 22 16:47:10 node97 kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jun 22 16:47:10 node97 kernel: ata1.00: ATAPI: hp      DVD RW AD-7586H, KP03, max UDMA/100
Jun 22 16:47:10 node97 kernel: ata1.00: configured for UDMA/100
Jun 22 16:47:10 node97 kernel: usb 4-1: new full speed USB device using uhci_hcd and address 2
Jun 22 16:47:10 node97 kernel: usb 4-1: configuration #1 chosen from 1 choice
Jun 22 16:47:10 node97 kernel: input: HP Virtual Keyboard as /class/input/input0
Jun 22 16:47:10 node97 kernel: input: USB HID v1.01 Keyboard [HP Virtual Keyboard] on usb-0000:00:04.4-1
Jun 22 16:47:10 node97 kernel: input: HP Virtual Keyboard as /class/input/input1
Jun 22 16:47:10 node97 kernel: input: USB HID v1.01 Mouse [HP Virtual Keyboard] on usb-0000:00:04.4-1
Jun 22 16:47:10 node97 kernel: ata2: SATA link down (SStatus 4 SControl 300)
Jun 22 16:47:10 node97 kernel: ata3: SATA link down (SStatus 4 SControl 300)
Jun 22 16:47:10 node97 kernel: ata4: SATA link down (SStatus 4 SControl 300)
Jun 22 16:47:10 node97 kernel:   Vendor: hp        Model: DVD RW AD-7586H   Rev: KP03
Jun 22 16:47:10 node97 kernel:   Type:   CD-ROM                             ANSI SCSI revision: 05
Jun 22 16:47:10 node97 kernel: device-mapper: uevent: version 1.0.3
Jun 22 16:47:10 node97 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com
Jun 22 16:47:10 node97 kernel: device-mapper: dm-raid45: initialized v0.2594l
Jun 22 16:47:10 node97 kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
Jun 22 16:47:10 node97 kernel: EXT3-fs: write access will be enabled during recovery.
Jun 22 16:47:10 node97 kernel: kjournald starting.  Commit interval 5 seconds
Jun 22 16:47:10 node97 kernel: EXT3-fs: recovery complete.
Jun 22 16:47:10 node97 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Jun 22 16:47:10 node97 kernel: SELinux:  Disabled at runtime.
Jun 22 16:47:10 node97 kernel: type=1404 audit(1308761187.212:2): selinux=0 auid=4294967295 ses=4294967295
Jun 22 16:47:10 node97 kernel: input: PC Speaker as /class/input/input2
Jun 22 16:47:10 node97 kernel: ACPI: PCI Interrupt 0000:00:04.2[B] -> GSI 45 (level, low) -> IRQ 146
Jun 22 16:47:11 node97 kernel: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.9.3 (March 17, 2009)
Jun 22 16:47:11 node97 kernel: ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 40 (level, low) -> IRQ 193
Jun 22 16:47:11 node97 kernel: eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f6000000, IRQ 193, node addr 0025b321f362
Jun 22 16:47:11 node97 kernel: ACPI: PCI Interrupt 0000:03:00.1[B] -> GSI 39 (level, low) -> IRQ 185
Jun 22 16:47:11 node97 kernel: eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f4000000, IRQ 185, node addr 0025b321f364
Jun 22 16:47:11 node97 kernel: ACPI: PCI Interrupt 0000:41:00.0[A] -> GSI 34 (level, low) -> IRQ 233
Jun 22 16:47:11 node97 kernel: eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem fa000000, IRQ 233, node addr 0025b321f35e
Jun 22 16:47:11 node97 kernel: ACPI: PCI Interrupt 0000:41:00.1[B] -> GSI 33 (level, low) -> IRQ 225
Jun 22 16:47:11 node97 kernel: eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f8000000, IRQ 225, node addr 0025b321f360
Jun 22 16:47:11 node97 kernel: Floppy drive(s): fd0 is 1.44M
Jun 22 16:47:11 node97 kernel: piix4_smbus 0000:00:06.0: Found 0000:00:06.0 device
Jun 22 16:47:11 node97 kernel: QLogic Fibre Channel HBA Driver
Jun 22 16:47:11 node97 kernel: ACPI: PCI Interrupt 0000:45:00.0[A] -> GSI 32 (level, low) -> IRQ 217
Jun 22 16:47:11 node97 kernel: qla2xxx 0000:45:00.0: Found an ISP2432, irq 217, iobase 0xffffc2000002e000
Jun 22 16:47:11 node97 kernel: qla2xxx 0000:45:00.0: Configuring PCI space...
Jun 22 16:47:11 node97 kernel: qla2xxx 0000:45:00.0: Configure NVRAM parameters...
Jun 22 16:47:11 node97 kernel: qla2xxx 0000:45:00.0: Verifying loaded RISC code...
Jun 22 16:47:11 node97 kernel: qla2xxx 0000:45:00.0: Allocated (64 KB) for EFT...
Jun 22 16:47:11 node97 kernel: qla2xxx 0000:45:00.0: Allocated (1413 KB) for firmware dump...
Jun 22 16:47:11 node97 kernel: scsi4 : qla2xxx
Jun 22 16:47:11 node97 kernel: qla2xxx 0000:45:00.0: 
Jun 22 16:47:11 node97 kernel:  QLogic Fibre Channel HBA Driver: 8.03.00.1.05.05-k
Jun 22 16:47:11 node97 kernel:   QLogic HPAE311A - PCI-Express 4Gb Fibre Channel HBA
Jun 22 16:47:11 node97 kernel:   ISP2432: PCIe (2.5Gb/s x4) @ 0000:45:00.0 hdma+, host#=4, fw=4.04.09 (486)
Jun 22 16:47:11 node97 kernel: scsi 0:0:0:0: Attached scsi generic sg0 type 5
Jun 22 16:47:11 node97 kernel: sr0: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
Jun 22 16:47:11 node97 kernel: Uniform CD-ROM driver Revision: 3.20
Jun 22 16:47:11 node97 kernel: qla2xxx 0000:45:00.0: LIP reset occured (f700).
Jun 22 16:47:11 node97 kernel: qla2xxx 0000:45:00.0: LOOP UP detected (4 Gbps).
Jun 22 16:47:11 node97 kernel:   Vendor: HP        Model: HSV300            Rev: 0953
Jun 22 16:47:11 node97 kernel:   Type:   RAID                               ANSI SCSI revision: 05
Jun 22 16:47:11 node97 kernel: scsi 4:0:0:0: Attached scsi generic sg1 type 12
Jun 22 16:47:11 node97 kernel:   Vendor: HP        Model: HSV300            Rev: 0953
Jun 22 16:47:11 node97 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Jun 22 16:47:12 node97 kernel: sda : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:12 node97 kernel: SCSI device sda: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:12 node97 kernel: sda: Write Protect is off
Jun 22 16:47:12 node97 kernel: SCSI device sda: drive cache: write through w/ FUA
Jun 22 16:47:12 node97 kernel: sda : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:12 node97 kernel: SCSI device sda: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:12 node97 kernel: sda: Write Protect is off
Jun 22 16:47:12 node97 kernel: SCSI device sda: drive cache: write through w/ FUA
Jun 22 16:47:12 node97 kernel:  sda: unknown partition table
Jun 22 16:47:12 node97 kernel: sd 4:0:0:1: Attached scsi disk sda
Jun 22 16:47:12 node97 kernel: sd 4:0:0:1: Attached scsi generic sg2 type 0
Jun 22 16:47:12 node97 kernel:   Vendor: HP        Model: HSV300            Rev: 0953
Jun 22 16:47:12 node97 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Jun 22 16:47:12 node97 kernel: sdb : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:12 node97 kernel: SCSI device sdb: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:12 node97 kernel: sdb: Write Protect is off
Jun 22 16:47:12 node97 kernel: SCSI device sdb: drive cache: write through w/ FUA
Jun 22 16:47:12 node97 kernel: sdb : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:12 node97 kernel: SCSI device sdb: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:12 node97 kernel: sdb: Write Protect is off
Jun 22 16:47:12 node97 kernel: SCSI device sdb: drive cache: write through w/ FUA
Jun 22 16:47:12 node97 kernel:  sdb: unknown partition table
Jun 22 16:47:12 node97 kernel: sd 4:0:0:2: Attached scsi disk sdb
Jun 22 16:47:12 node97 kernel: sd 4:0:0:2: Attached scsi generic sg3 type 0
Jun 22 16:47:12 node97 kernel:   Vendor: HP        Model: HSV300            Rev: 0953
Jun 22 16:47:12 node97 kernel:   Type:   RAID                               ANSI SCSI revision: 05
Jun 22 16:47:12 node97 kernel: scsi 4:0:1:0: Attached scsi generic sg4 type 12
Jun 22 16:47:12 node97 kernel:   Vendor: HP        Model: HSV300            Rev: 0953
Jun 22 16:47:12 node97 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Jun 22 16:47:12 node97 kernel: sdc : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:12 node97 kernel: SCSI device sdc: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:12 node97 kernel: sdc: Write Protect is off
Jun 22 16:47:13 node97 kernel: SCSI device sdc: drive cache: write through w/ FUA
Jun 22 16:47:13 node97 kernel: sdc : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:13 node97 kernel: SCSI device sdc: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:13 node97 kernel: sdc: Write Protect is off
Jun 22 16:47:13 node97 kernel: SCSI device sdc: drive cache: write through w/ FUA
Jun 22 16:47:13 node97 kernel:  sdc: unknown partition table
Jun 22 16:47:13 node97 kernel: sd 4:0:1:1: Attached scsi disk sdc
Jun 22 16:47:13 node97 kernel: sd 4:0:1:1: Attached scsi generic sg5 type 0
Jun 22 16:47:13 node97 kernel:   Vendor: HP        Model: HSV300            Rev: 0953
Jun 22 16:47:13 node97 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Jun 22 16:47:13 node97 kernel: sdd : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:13 node97 kernel: SCSI device sdd: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:13 node97 kernel: sdd: Write Protect is off
Jun 22 16:47:13 node97 kernel: SCSI device sdd: drive cache: write through w/ FUA
Jun 22 16:47:13 node97 kernel: sdd : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:13 node97 kernel: SCSI device sdd: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:13 node97 kernel: sdd: Write Protect is off
Jun 22 16:47:13 node97 kernel: SCSI device sdd: drive cache: write through w/ FUA
Jun 22 16:47:13 node97 kernel:  sdd: unknown partition table
Jun 22 16:47:13 node97 kernel: sd 4:0:1:2: Attached scsi disk sdd
Jun 22 16:47:13 node97 kernel: sd 4:0:1:2: Attached scsi generic sg6 type 0
Jun 22 16:47:13 node97 kernel:   Vendor: HP        Model: HSV300            Rev: 0953
Jun 22 16:47:13 node97 kernel:   Type:   RAID                               ANSI SCSI revision: 05
Jun 22 16:47:13 node97 kernel: scsi 4:0:2:0: Attached scsi generic sg7 type 12
Jun 22 16:47:13 node97 kernel:   Vendor: HP        Model: HSV300            Rev: 0953
Jun 22 16:47:13 node97 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Jun 22 16:47:13 node97 kernel: sde : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:13 node97 kernel: SCSI device sde: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:13 node97 kernel: sde: Write Protect is off
Jun 22 16:47:13 node97 kernel: SCSI device sde: drive cache: write through w/ FUA
Jun 22 16:47:13 node97 kernel: sde : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:13 node97 kernel: SCSI device sde: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:13 node97 kernel: sde: Write Protect is off
Jun 22 16:47:14 node97 kernel: SCSI device sde: drive cache: write through w/ FUA
Jun 22 16:47:14 node97 kernel:  sde: unknown partition table
Jun 22 16:47:14 node97 kernel: sd 4:0:2:1: Attached scsi disk sde
Jun 22 16:47:14 node97 kernel: sd 4:0:2:1: Attached scsi generic sg8 type 0
Jun 22 16:47:14 node97 kernel:   Vendor: HP        Model: HSV300            Rev: 0953
Jun 22 16:47:14 node97 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Jun 22 16:47:14 node97 kernel: sdf : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:14 node97 kernel: SCSI device sdf: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:14 node97 kernel: sdf: Write Protect is off
Jun 22 16:47:14 node97 kernel: SCSI device sdf: drive cache: write through w/ FUA
Jun 22 16:47:14 node97 kernel: sdf : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:14 node97 kernel: SCSI device sdf: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:14 node97 kernel: sdf: Write Protect is off
Jun 22 16:47:14 node97 kernel: SCSI device sdf: drive cache: write through w/ FUA
Jun 22 16:47:14 node97 kernel:  sdf: unknown partition table
Jun 22 16:47:14 node97 kernel: sd 4:0:2:2: Attached scsi disk sdf
Jun 22 16:47:14 node97 kernel: sd 4:0:2:2: Attached scsi generic sg9 type 0
Jun 22 16:47:14 node97 kernel:   Vendor: HP        Model: HSV300            Rev: 0953
Jun 22 16:47:14 node97 kernel:   Type:   RAID                               ANSI SCSI revision: 05
Jun 22 16:47:14 node97 kernel: scsi 4:0:3:0: Attached scsi generic sg10 type 12
Jun 22 16:47:14 node97 kernel:   Vendor: HP        Model: HSV300            Rev: 0953
Jun 22 16:47:14 node97 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Jun 22 16:47:14 node97 kernel: sdg : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:14 node97 kernel: SCSI device sdg: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:14 node97 kernel: sdg: Write Protect is off
Jun 22 16:47:14 node97 kernel: SCSI device sdg: drive cache: write through w/ FUA
Jun 22 16:47:14 node97 kernel: sdg : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:14 node97 kernel: SCSI device sdg: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:14 node97 kernel: sdg: Write Protect is off
Jun 22 16:47:14 node97 kernel: SCSI device sdg: drive cache: write through w/ FUA
Jun 22 16:47:14 node97 kernel:  sdg: unknown partition table
Jun 22 16:47:14 node97 kernel: sd 4:0:3:1: Attached scsi disk sdg
Jun 22 16:47:14 node97 kernel: sd 4:0:3:1: Attached scsi generic sg11 type 0
Jun 22 16:47:15 node97 kernel:   Vendor: HP        Model: HSV300            Rev: 0953
Jun 22 16:47:15 node97 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Jun 22 16:47:15 node97 kernel: sdh : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:15 node97 kernel: SCSI device sdh: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:15 node97 kernel: sdh: Write Protect is off
Jun 22 16:47:15 node97 kernel: SCSI device sdh: drive cache: write through w/ FUA
Jun 22 16:47:15 node97 kernel: sdh : very big device. try to use READ CAPACITY(16).
Jun 22 16:47:15 node97 kernel: SCSI device sdh: 8598323200 512-byte hdwr sectors (4402341 MB)
Jun 22 16:47:15 node97 kernel: sdh: Write Protect is off
Jun 22 16:47:15 node97 kernel: SCSI device sdh: drive cache: write through w/ FUA
Jun 22 16:47:15 node97 kernel:  sdh: unknown partition table
Jun 22 16:47:15 node97 kernel: sd 4:0:3:2: Attached scsi disk sdh
Jun 22 16:47:15 node97 kernel: sd 4:0:3:2: Attached scsi generic sg12 type 0
Jun 22 16:47:15 node97 kernel: floppy0: no floppy controllers found
Jun 22 16:47:15 node97 kernel: Floppy drive(s): fd0 is 1.44M
Jun 22 16:47:15 node97 kernel: floppy0: no floppy controllers found
Jun 22 16:47:15 node97 kernel: lp: driver loaded but no devices found
Jun 22 16:47:15 node97 kernel: NET: Registered protocol family 10
Jun 22 16:47:15 node97 kernel: lo: Disabled Privacy Extensions
Jun 22 16:47:15 node97 kernel: IPv6 over IPv4 tunneling driver
Jun 22 16:47:15 node97 kernel: ACPI: Power Button (FF) [PWRF]
Jun 22 16:47:15 node97 kernel: md: Autodetecting RAID arrays.
Jun 22 16:47:15 node97 kernel: md: autorun ...
Jun 22 16:47:15 node97 kernel: md: ... autorun DONE.
Jun 22 16:47:15 node97 kernel: device-mapper: multipath: version 1.0.5 loaded
Jun 22 16:47:15 node97 kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Jun 22 16:47:15 node97 kernel: EXT3 FS on cciss/c0d0p1, internal journal
Jun 22 16:47:15 node97 kernel: Adding 8385920k swap on /dev/cciss/c0d0p2.  Priority:-1 extents:1 across:8385920k
Jun 22 16:47:15 node97 kernel: powernow-k8: Pre-initialization of ACPI failed
Jun 22 16:47:15 node97 kernel: powernow-k8: Found 2 Six-Core AMD Opteron(tm) Processor 2435 processors (12 cpu cores) (version 2.20.00)
Jun 22 16:47:15 node97 kernel: powernow-k8: Your BIOS does not provide _PSS objects.  PowerNow! does not work on SMP systems without _PSS objects.  Complain to your BIOS vendor.
Jun 22 16:47:16 node97 last message repeated 11 times
Jun 22 16:47:16 node97 kernel: bnx2: eth1: using MSIX
Jun 22 16:47:16 node97 kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready
Jun 22 16:47:16 node97 kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex
Jun 22 16:47:16 node97 kernel: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Jun 22 16:47:16 node97 kernel: Lustre: OBD class driver, http://www.lustre.org/
Jun 22 16:47:16 node97 kernel: Lustre:     Lustre Version: 1.8.3
Jun 22 16:47:16 node97 kernel: Lustre:     Build Version: 1.8.3-20100409182943-PRISTINE-2.6.18-164.11.1.el5_lustre.1.8.3
Jun 22 16:47:16 node97 kernel: Lustre: Added LNI 192.168.111.97@tcp [8/256/0/180]
Jun 22 16:47:16 node97 kernel: Lustre: Accept secure, port 988
Jun 22 16:47:16 node97 kernel: Lustre: Lustre Client File System; http://www.lustre.org/
Jun 22 16:47:16 node97 kernel: init dynlocks cache
Jun 22 16:47:16 node97 kernel: ldiskfs created from ext3-2.6-rhel5
Jun 22 16:47:16 node97 kernel: kjournald starting.  Commit interval 5 seconds
Jun 22 16:47:16 node97 kernel: LDISKFS-fs warning: maximal mount count reached, running e2fsck is recommended
Jun 22 16:47:16 node97 kernel: LDISKFS FS on dm-0, internal journal
Jun 22 16:47:16 node97 kernel: LDISKFS-fs: recovery complete.
Jun 22 16:47:16 node97 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.
Jun 22 16:47:16 node97 kernel: kjournald starting.  Commit interval 5 seconds
Jun 22 16:47:16 node97 kernel: LDISKFS-fs warning: maximal mount count reached, running e2fsck is recommended
Jun 22 16:47:16 node97 kernel: LDISKFS FS on dm-0, internal journal
Jun 22 16:47:16 node97 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.
Jun 22 16:47:16 node97 kernel: LDISKFS-fs: file extents enabled
Jun 22 16:47:16 node97 kernel: LDISKFS-fs: mballoc enabled
Jun 22 16:47:16 node97 kernel: Lustre: MGC192.168.111.100@tcp: Reactivating import
Jun 22 16:47:26 node97 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available  for connect (no target)
Jun 22 16:47:26 node97 kernel: LustreError: 7255:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-19)  req@ffff81012b888800 x1371312002115897/t0 o8-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1308732546 ref 1 fl Interpret:/0/0 rc -19/0
Jun 22 16:47:31 node97 kernel: LustreError: 137-5: UUID 'lustre-OST0003_UUID' is not available  for connect (no target)
Jun 22 16:47:31 node97 kernel: LustreError: 7256:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-19)  req@ffff810113247800 x1371312002115898/t0 o8-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1308732551 ref 1 fl Interpret:/0/0 rc -19/0
Jun 22 16:47:34 node97 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available  for connect (no target)
Jun 22 16:47:34 node97 kernel: LustreError: 7257:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-19)  req@ffff81012503f000 x1371314868890750/t0 o8-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1308732554 ref 1 fl Interpret:/0/0 rc -19/0
Jun 22 16:47:36 node97 kernel: Lustre: Filtering OBD driver; http://www.lustre.org/
Jun 22 16:47:36 node97 kernel: Lustre: 7538:0:(filter.c:990:filter_init_server_data()) RECOVERY: service lustre-OST0000, 37 recoverable clients, 0 delayed clients, last_rcvd 257712296235
Jun 22 16:47:36 node97 kernel: Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/mapper/mpath0 with recovery enabled
Jun 22 16:47:36 node97 kernel: Lustre: lustre-OST0000: Will be in recovery for at least 5:00, or until 37 clients reconnect
Jun 22 16:47:36 node97 kernel: LustreError: 7538:0:(obd_config.c:1011:class_process_proc_param()) lustre-OST0000: unknown param mdt.quota_type=ug2
Jun 22 16:47:36 node97 kernel: Lustre: lustre-OST0000.ost: set parameter quota_type=ug2
Jun 22 16:47:39 node97 kernel: Lustre: 7348:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) lustre-OST0000: 36 recoverable clients remain
Jun 22 16:47:39 node97 kernel: kjournald starting.  Commit interval 5 seconds
Jun 22 16:47:39 node97 kernel: LDISKFS-fs warning: maximal mount count reached, running e2fsck is recommended
Jun 22 16:47:39 node97 kernel: LDISKFS FS on dm-1, internal journal
Jun 22 16:47:39 node97 kernel: LDISKFS-fs: recovery complete.
Jun 22 16:47:39 node97 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.
Jun 22 16:47:39 node97 multipathd: dm-1: umount map (uevent) 
Jun 22 16:47:39 node97 kernel: kjournald starting.  Commit interval 5 seconds
Jun 22 16:47:39 node97 kernel: LDISKFS-fs warning: maximal mount count reached, running e2fsck is recommended
Jun 22 16:47:39 node97 kernel: LDISKFS FS on dm-1, internal journal
Jun 22 16:47:39 node97 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.
Jun 22 16:47:39 node97 kernel: LDISKFS-fs: file extents enabled
Jun 22 16:47:39 node97 kernel: LustreError: 137-5: UUID 'lustre-OST0003_UUID' is not available  for connect (no target)
Jun 22 16:47:39 node97 kernel: LustreError: 7855:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-19)  req@ffff81012b449800 x1371314791390292/t0 o8-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1308732559 ref 1 fl Interpret:/0/0 rc -19/0
Jun 22 16:47:39 node97 kernel: LDISKFS-fs: mballoc enabled
Jun 22 16:47:39 node97 kernel: Lustre: 7966:0:(filter.c:990:filter_init_server_data()) RECOVERY: service lustre-OST0003, 37 recoverable clients, 0 delayed clients, last_rcvd 253410481848
Jun 22 16:47:39 node97 kernel: Lustre: lustre-OST0003: Now serving lustre-OST0003 on /dev/mapper/mpath1 with recovery enabled
Jun 22 16:47:39 node97 kernel: Lustre: lustre-OST0003: Will be in recovery for at least 5:00, or until 37 clients reconnect
Jun 22 16:47:39 node97 kernel: Lustre: lustre-OST0003.ost: set parameter quota_type=ug2
Jun 22 16:47:39 node97 kernel: LustreError: 7966:0:(obd_config.c:1011:class_process_proc_param()) lustre-OST0003: unknown param mdt.quota_type=ug2
Jun 22 16:47:39 node97 pcscd: pcscdaemon.c:507:main() pcsc-lite 1.4.4 daemon ready.
Jun 22 16:47:40 node97 kernel: Lustre: 7684:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) lustre-OST0000: 35 recoverable clients remain
Jun 22 16:47:40 node97 pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead.
Jun 22 16:47:40 node97 pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s)
Jun 22 16:47:41 node97 kernel: Lustre: 7265:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) lustre-OST0000: 34 recoverable clients remain
Jun 22 16:47:42 node97 kernel: Lustre: 7255:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) lustre-OST0003: 36 recoverable clients remain
Jun 22 16:47:42 node97 kernel: Lustre: 7255:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) Skipped 1 previous similar message
Jun 22 16:47:44 node97 kernel: Lustre: 7355:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) lustre-OST0000: 31 recoverable clients remain
Jun 22 16:47:44 node97 kernel: Lustre: 7355:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) Skipped 3 previous similar messages
Jun 22 16:47:44 node97 kernel: Bluetooth: Core ver 2.10
Jun 22 16:47:44 node97 kernel: NET: Registered protocol family 31
Jun 22 16:47:44 node97 kernel: Bluetooth: HCI device and connection manager initialized
Jun 22 16:47:44 node97 kernel: Bluetooth: HCI socket layer initialized
Jun 22 16:47:44 node97 kernel: Bluetooth: L2CAP ver 2.8
Jun 22 16:47:44 node97 kernel: Bluetooth: L2CAP socket layer initialized
Jun 22 16:47:44 node97 kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.1
Jun 22 16:47:44 node97 hidd[8099]: Bluetooth HID daemon
Jun 22 16:47:46 node97 automount[8136]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jun 22 16:47:46 node97 gpm[8188]: *** info [startup.c(95)]: 
Jun 22 16:47:46 node97 gpm[8188]: Started gpm successfully. Entered daemon mode.
Jun 22 16:47:46 node97 xinetd[8179]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
Jun 22 16:47:46 node97 xinetd[8179]: Started working: 0 available services
Jun 22 16:47:47 node97 avahi-daemon[8269]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70).
Jun 22 16:47:47 node97 avahi-daemon[8269]: Successfully dropped root privileges.
Jun 22 16:47:47 node97 avahi-daemon[8269]: avahi-daemon 0.6.16 starting up.
Jun 22 16:47:47 node97 avahi-daemon[8269]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Jun 22 16:47:47 node97 avahi-daemon[8269]: Successfully called chroot().
Jun 22 16:47:47 node97 avahi-daemon[8269]: Successfully dropped remaining capabilities.
Jun 22 16:47:47 node97 avahi-daemon[8269]: Loading service file /services/sftp-ssh.service.
Jun 22 16:47:47 node97 avahi-daemon[8269]: New relevant interface eth1.IPv6 for mDNS.
Jun 22 16:47:47 node97 avahi-daemon[8269]: Joining mDNS multicast group on interface eth1.IPv6 with address fe80::225:b3ff:fe21:f364.
Jun 22 16:47:47 node97 avahi-daemon[8269]: New relevant interface eth1.IPv4 for mDNS.
Jun 22 16:47:47 node97 avahi-daemon[8269]: Joining mDNS multicast group on interface eth1.IPv4 with address 192.168.111.97.
Jun 22 16:47:47 node97 avahi-daemon[8269]: Network interface enumeration completed.
Jun 22 16:47:47 node97 avahi-daemon[8269]: Registering new address record for fe80::225:b3ff:fe21:f364 on eth1.
Jun 22 16:47:47 node97 avahi-daemon[8269]: Registering new address record for 192.168.111.97 on eth1.
Jun 22 16:47:47 node97 avahi-daemon[8269]: Registering HINFO record with values 'X86_64'/'LINUX'.
Jun 22 16:47:48 node97 avahi-daemon[8269]: Server startup complete. Host name is node97.local. Local service cookie is 1151745486.
Jun 22 16:47:49 node97 avahi-daemon[8269]: Service "SFTP File Transfer on node97" (/services/sftp-ssh.service) successfully established.
Jun 22 16:47:57 node97 kernel: Lustre: 7706:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) lustre-OST0000: 28 recoverable clients remain
Jun 22 16:47:57 node97 kernel: Lustre: 7706:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) Skipped 3 previous similar messages
Jun 22 16:47:59 node97 kernel: Adding 8191992k swap on /swap.add.  Priority:-2 extents:2382 across:8856828k
Jun 22 16:47:59 node97 avahi-daemon[8269]: Registering new address record for 11.11.11.97 on eth1.
Jun 22 16:47:59 node97 avahi-daemon[8269]: Withdrawing address record for 11.11.11.97 on eth1.
Jun 22 16:47:59 node97 avahi-daemon[8269]: Registering new address record for 11.11.11.97 on eth1.
Jun 22 16:48:01 node97 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
Jun 22 16:48:01 node97 last message repeated 3 times
Jun 22 16:48:05 node97 kernel: Lustre: 7707:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) lustre-OST0000: 26 recoverable clients remain
Jun 22 16:48:05 node97 kernel: Lustre: 7707:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) Skipped 7 previous similar messages
Jun 22 16:48:15 node97 kernel: Lustre: 7838:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0003: 993d4d04-1a19-dd6b-0220-163a6869cb64 reconnecting
Jun 22 16:48:15 node97 kernel: Lustre: 7838:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0003: refuse reconnection from 993d4d04-1a19-dd6b-0220-163a6869cb64@11.11.11.15@tcp to 0xffff81012ca74000; still busy with 264 active RPCs
Jun 22 16:48:15 node97 kernel: LustreError: 7838:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-16)  req@ffff810115eb8800 x1371314807096328/t0 o8->993d4d04-1a19-dd6b-0220-163a6869cb64@NET_0x200000b0b0b0f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308732592 ref 1 fl Interpret:/0/0 rc -16/0
Jun 22 16:48:15 node97 kernel: Lustre: 7606:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0003: 993d4d04-1a19-dd6b-0220-163a6869cb64 reconnecting
Jun 22 16:48:15 node97 kernel: Lustre: 7606:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0003: refuse reconnection from 993d4d04-1a19-dd6b-0220-163a6869cb64@11.11.11.15@tcp to 0xffff81012ca74000; still busy with 4 active RPCs
Jun 22 16:48:22 node97 kernel: Lustre: 7914:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) lustre-OST0000: 15 recoverable clients remain
Jun 22 16:48:22 node97 kernel: Lustre: 7914:0:(ldlm_lib.c:1788:target_queue_last_replay_reply()) Skipped 23 previous similar messages
Jun 22 16:48:23 node97 kernel: Lustre: 7260:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0003: 993d4d04-1a19-dd6b-0220-163a6869cb64 reconnecting
Jun 22 16:48:30 node97 kernel: Lustre: 7273:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: 993d4d04-1a19-dd6b-0220-163a6869cb64 reconnecting
Jun 22 16:48:30 node97 kernel: Lustre: 7273:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0000: refuse reconnection from 993d4d04-1a19-dd6b-0220-163a6869cb64@11.11.11.15@tcp to 0xffff81013388aa00; still busy with 510 active RPCs
Jun 22 16:48:30 node97 kernel: LustreError: 7273:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-16)  req@ffff810219738400 x1371314807119112/t0 o8->993d4d04-1a19-dd6b-0220-163a6869cb64@NET_0x200000b0b0b0f_UUID:0/0 lens 368/264 e 0 to 0 dl 1308732608 ref 1 fl Interpret:/0/0 rc -16/0
Jun 22 16:48:30 node97 kernel: LustreError: 7273:0:(ldlm_lib.c:1892:target_send_reply_msg()) Skipped 1 previous similar message
Jun 22 16:48:37 node97 kernel: Lustre: 7874:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: 1e209d42-9195-820e-f318-877a9e7f08dd reconnecting
Jun 22 16:48:37 node97 kernel: Lustre: 7874:0:(ldlm_lib.c:575:target_handle_reconnect()) Skipped 1 previous similar message
Jun 22 16:48:37 node97 kernel: Lustre: 7808:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0000: refuse reconnection from 993d4d04-1a19-dd6b-0220-163a6869cb64@11.11.11.15@tcp to 0xffff81013388aa00; still busy with 6 active RPCs
Jun 22 16:48:43 node97 kernel: Lustre: 7791:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: 993d4d04-1a19-dd6b-0220-163a6869cb64 reconnecting
Jun 22 16:48:43 node97 kernel: Lustre: 7791:0:(ldlm_lib.c:575:target_handle_reconnect()) Skipped 2 previous similar messages
Jun 22 16:48:55 node97 kernel: Lustre: lustre-OST0000: Recovery period over after 1:17, of 37 clients 37 recovered and 0 were evicted.
Jun 22 16:48:55 node97 kernel: Lustre: lustre-OST0000: sending delayed replies to recovered clients
Jun 22 16:48:55 node97 kernel: Lustre: lustre-OST0000: received MDS connection from 192.168.111.100@tcp
Jun 22 16:48:55 node97 kernel: Lustre: lustre-OST0003: Recovery period over after 1:15, of 37 clients 37 recovered and 0 were evicted.
Jun 22 16:48:55 node97 kernel: Lustre: lustre-OST0003: sending delayed replies to recovered clients
Jun 22 16:48:55 node97 kernel: Lustre: lustre-OST0003: received MDS connection from 192.168.111.100@tcp
Jun 22 16:49:15 node97 kernel: Lustre: 7688:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: 632d5b96-c7de-a5c1-1d90-214ae7a2f32d reconnecting
Jun 22 16:49:15 node97 kernel: Lustre: 7688:0:(ldlm_lib.c:575:target_handle_reconnect()) Skipped 1 previous similar message
Jun 22 16:49:15 node97 kernel: Lustre: 7873:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0003: refuse reconnection from 632d5b96-c7de-a5c1-1d90-214ae7a2f32d@11.11.11.43@tcp to 0xffff810110f65200; still busy with 1 active RPCs
Jun 22 16:49:15 node97 kernel: Lustre: 7328:0:(ldlm_lib.c:804:target_handle_connect()) lustre-OST0003: exp ffff810110f65200 already connecting
Jun 22 16:49:15 node97 kernel: LustreError: 7873:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-16)  req@ffff8101ce611c00 x1371463404703431/t0 o8->632d5b96-c7de-a5c1-1d90-214ae7a2f32d@NET_0x200000b0b0b2b_UUID:0/0 lens 368/264 e 0 to 0 dl 1308732655 ref 1 fl Interpret:/0/0 rc -16/0
Jun 22 16:49:15 node97 kernel: LustreError: 7873:0:(ldlm_lib.c:1892:target_send_reply_msg()) Skipped 2 previous similar messages
Jun 22 16:49:15 node97 kernel: LustreError: 8552:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff81010af3e800 x1371463404703416/t0 o3->632d5b96-c7de-a5c1-1d90-214ae7a2f32d@NET_0x200000b0b0b2b_UUID:0/0 lens 448/400 e 1 to 0 dl 1308732567 ref 1 fl Interpret:/0/0 rc 0/0
Jun 22 16:49:15 node97 kernel: Lustre: 8552:0:(ost_handler.c:886:ost_brw_read()) lustre-OST0000: ignoring bulk IO comm error with 632d5b96-c7de-a5c1-1d90-214ae7a2f32d@NET_0x200000b0b0b2b_UUID id 12345-11.11.11.43@tcp - client will retry
Jun 22 16:49:20 node97 kernel: Lustre: 7927:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0000: refuse reconnection from 632d5b96-c7de-a5c1-1d90-214ae7a2f32d@11.11.11.43@tcp to 0xffff81022ff20600; still busy with 4 active RPCs
Jun 22 16:49:20 node97 kernel: Lustre: 7927:0:(ldlm_lib.c:875:target_handle_connect()) Skipped 1 previous similar message
Jun 22 16:49:42 node97 kernel: Lustre: lustre-OST0000: slow i_mutex 46s due to heavy IO load
Jun 22 16:49:42 node97 kernel: Lustre: lustre-OST0000: slow i_mutex 46s due to heavy IO load
Jun 22 16:49:42 node97 kernel: Lustre: lustre-OST0000: slow journal start 46s due to heavy IO load
Jun 22 16:49:42 node97 kernel: Lustre: lustre-OST0000: slow journal start 46s due to heavy IO load
Jun 22 16:49:42 node97 kernel: Lustre: lustre-OST0000: slow brw_start 46s due to heavy IO load
Jun 22 16:49:42 node97 kernel: Lustre: lustre-OST0000: slow parent lock 46s due to heavy IO load
Jun 22 16:49:42 node97 kernel: Lustre: lustre-OST0000: slow preprw_read setup 46s due to heavy IO load
Jun 22 16:49:42 node97 kernel: Lustre: lustre-OST0000: slow parent lock 46s due to heavy IO load
Jun 22 16:49:42 node97 kernel: Lustre: Skipped 2 previous similar messages
Jun 22 16:49:42 node97 kernel: LustreError: 8525:0:(ost_handler.c:844:ost_brw_read()) @@@ bulk PUT failed: rc -107  req@ffff8101cbc1ec00 x1371463404703377/t0 o3->632d5b96-c7de-a5c1-1d90-214ae7a2f32d@NET_0x200000b0b0b2b_UUID:0/0 lens 448/400 e 2 to 0 dl 1308732619 ref 1 fl Interpret:/0/0 rc 0/0
Jun 22 16:49:42 node97 kernel: LustreError: 8525:0:(ost_handler.c:844:ost_brw_read()) Skipped 1 previous similar message
Jun 22 16:49:42 node97 kernel: Lustre: lustre-OST0000: slow preprw_read setup 46s due to heavy IO load
Jun 22 16:49:42 node97 kernel: Lustre: Skipped 1 previous similar message
Jun 22 16:49:43 node97 kernel: Lustre: 7631:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: 632d5b96-c7de-a5c1-1d90-214ae7a2f32d reconnecting
Jun 22 16:49:43 node97 kernel: Lustre: 7631:0:(ldlm_lib.c:575:target_handle_reconnect()) Skipped 2 previous similar messages
Jun 22 16:49:43 node97 kernel: Lustre: 7631:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0000: refuse reconnection from 632d5b96-c7de-a5c1-1d90-214ae7a2f32d@11.11.11.43@tcp to 0xffff81022ff20600; still busy with 4 active RPCs
Jun 22 16:49:43 node97 kernel: LustreError: 7631:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-16)  req@ffff8102288d3400 x1371463404703504/t0 o8->632d5b96-c7de-a5c1-1d90-214ae7a2f32d@NET_0x200000b0b0b2b_UUID:0/0 lens 368/264 e 0 to 0 dl 1308732683 ref 1 fl Interpret:/0/0 rc -16/0
Jun 22 16:49:43 node97 kernel: LustreError: 7631:0:(ldlm_lib.c:1892:target_send_reply_msg()) Skipped 1 previous similar message
Jun 22 16:49:44 node97 kernel: Lustre: lustre-OST0000: slow i_mutex 49s due to heavy IO load
Jun 22 16:49:44 node97 kernel: Lustre: Skipped 32 previous similar messages
Jun 22 16:49:44 node97 kernel: Lustre: 8513:0:(ost_handler.c:886:ost_brw_read()) lustre-OST0000: ignoring bulk IO comm error with 632d5b96-c7de-a5c1-1d90-214ae7a2f32d@NET_0x200000b0b0b2b_UUID id 12345-11.11.11.43@tcp - client will retry
Jun 22 16:49:44 node97 kernel: Lustre: 8513:0:(ost_handler.c:886:ost_brw_read()) Skipped 1 previous similar message
Jun 22 16:49:46 node97 kernel: Lustre: lustre-OST0000: slow i_mutex 51s due to heavy IO load
Jun 22 16:49:46 node97 kernel: Lustre: lustre-OST0000: slow i_mutex 51s due to heavy IO load
Jun 22 16:49:46 node97 kernel: Lustre: Skipped 4 previous similar messages
Jun 22 16:50:06 node97 kernel: Lustre: 7842:0:(ldlm_lib.c:875:target_handle_connect()) lustre-OST0000: refuse reconnection from 632d5b96-c7de-a5c1-1d90-214ae7a2f32d@11.11.11.43@tcp to 0xffff81022ff20600; still busy with 2 active RPCs
Jun 22 16:50:16 node97 kernel: Lustre: lustre-OST0000: slow i_mutex 81s due to heavy IO load
Jun 22 16:50:16 node97 kernel: Lustre: Skipped 28 previous similar messages
Jun 22 16:50:17 node97 kernel: Lustre: lustre-OST0000: slow journal start 30s due to heavy IO load
Jun 22 16:50:17 node97 kernel: Lustre: lustre-OST0000: slow brw_start 30s due to heavy IO load
Jun 22 16:50:17 node97 kernel: Lustre: Skipped 5 previous similar messages
Jun 22 16:50:17 node97 kernel: Lustre: Skipped 3 previous similar messages
Jun 22 16:50:17 node97 kernel: Lustre: lustre-OST0000: slow journal start 30s due to heavy IO load
Jun 22 16:50:17 node97 kernel: Lustre: Skipped 11 previous similar messages
Jun 22 16:50:17 node97 kernel: Lustre: lustre-OST0000: slow direct_io 81s due to heavy IO load
Jun 22 16:50:17 node97 kernel: Lustre: lustre-OST0000: slow parent lock 32s due to heavy IO load
Jun 22 16:50:17 node97 kernel: Lustre: Skipped 10 previous similar messages
Jun 22 16:50:26 node97 kernel: Lustre: lustre-OST0000: slow i_mutex 90s due to heavy IO load
Jun 22 16:50:26 node97 kernel: Lustre: Skipped 9 previous similar messages
Jun 22 16:50:29 node97 kernel: Lustre: lustre-OST0000: slow i_mutex 93s due to heavy IO load
Jun 22 16:50:29 node97 kernel: Lustre: Skipped 1 previous similar message
Jun 22 16:50:29 node97 kernel: Lustre: lustre-OST0000: slow direct_io 93s due to heavy IO load
Jun 22 16:50:29 node97 kernel: Lustre: Skipped 2 previous similar messages
Jun 22 16:50:29 node97 kernel: Lustre: lustre-OST0000: slow direct_io 46s due to heavy IO load
Jun 22 16:50:29 node97 kernel: Lustre: lustre-OST0000: slow parent lock 42s due to heavy IO load
Jun 22 16:50:29 node97 kernel: Lustre: Skipped 6 previous similar messages
Jun 22 16:50:29 node97 kernel: Lustre: lustre-OST0000: slow direct_io 47s due to heavy IO load
Jun 22 16:50:29 node97 kernel: Lustre: Skipped 6 previous similar messages
Jun 22 16:50:29 node97 kernel: Lustre: lustre-OST0000: slow i_mutex 44s due to heavy IO load
Jun 22 16:50:29 node97 kernel: Lustre: Skipped 10 previous similar messages
Jun 22 16:50:29 node97 kernel: Lustre: 7597:0:(ldlm_lib.c:575:target_handle_reconnect()) lustre-OST0000: 632d5b96-c7de-a5c1-1d90-214ae7a2f32d reconnecting
Jun 22 16:50:29 node97 kernel: Lustre: 7597:0:(ldlm_lib.c:575:target_handle_reconnect()) Skipped 1 previous similar message
Jun 22 16:50:29 node97 kernel: LustreError: 7597:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error (-16)  req@ffff8101cfc78c00 x1371463404703526/t0 o8->632d5b96-c7de-a5c1-1d90-214ae7a2f32d@NET_0x200000b0b0b2b_UUID:0/0 lens 368/264 e 0 to 0 dl 1308732729 ref 1 fl Interpret:/0/0 rc -16/0
Jun 22 16:50:29 node97 kernel: LustreError: 7597:0:(ldlm_lib.c:1892:target_send_reply_msg()) Skipped 1 previous similar message
Jun 22 16:51:40 node97 kernel: Lustre: lustre-OST0000: slow i_mutex 40s due to heavy IO load
Jun 22 16:51:40 node97 kernel: Lustre: lustre-OST0000: slow i_mutex 64s due to heavy IO load
Jun 22 16:51:40 node97 kernel: Lustre: Skipped 28 previous similar messages
Jun 22 16:51:40 node97 kernel: Lustre: lustre-OST0000: slow direct_io 70s due to heavy IO load
Jun 22 16:51:40 node97 kernel: Lustre: lustre-OST0000: slow journal start 70s due to heavy IO load
Jun 22 16:51:40 node97 kernel: Lustre: lustre-OST0000: slow journal start 70s due to heavy IO load
Jun 22 16:51:40 node97 kernel: Lustre: Skipped 1 previous similar message
Jun 22 16:51:40 node97 kernel: Lustre: lustre-OST0000: slow commitrw commit 70s due to heavy IO load
Jun 22 16:51:40 node97 kernel: Lustre: Skipped 1 previous similar message
Jun 22 16:51:40 node97 kernel: Lustre: lustre-OST0000: slow parent lock 70s due to heavy IO load
Jun 22 16:51:40 node97 kernel: Lustre: lustre-OST0000: slow parent lock 70s due to heavy IO load
Jun 22 16:51:40 node97 kernel: Lustre: lustre-OST0000: slow preprw_write setup 70s due to heavy IO load
Jun 22 16:51:40 node97 kernel: Lustre: lustre-OST0000: slow parent lock 40s due to heavy IO load
Jun 22 16:51:40 node97 kernel: Lustre: lustre-OST0000: slow preprw_write setup 40s due to heavy IO load
