Mar 14 14:58:48 cn46 kernel: LustreError: 4593:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16944672: rc -2
Mar 14 15:03:33 cn46 kernel: Lustre: lustre-OST0003: haven't heard from client 295f21a1-f8bf-e743-3e99-b39f1d03ba0a (at 10.128.15.37@tcp) in 677 seconds. I think it's dead, and I am evicting it.
Mar 14 15:09:14 cn46 kernel: LustreError: 4495:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16947151: rc -2
Mar 14 15:29:00 cn46 kernel: Lustre: 4255:0:(router.c:167:lnet_notify()) Ignoring prediction from 10.128.15.46@tcp of 10.128.15.16@tcp down 792495635060 seconds in the future
Mar 14 15:29:37 cn46 kernel: Lustre: lustre-OST0003: haven't heard from client a43b05da-6e99-b2b8-d474-e3323a20d699 (at 10.128.15.16@tcp) in 677 seconds. I think it's dead, and I am evicting it.
Mar 14 15:30:48 cn46 kernel: Lustre: 4655:0:(ldlm_lib.c:519:target_handle_reconnect()) lustre-OST0003: 4600a146-6e5d-72fe-f398-1aca7050cdbf reconnecting
Mar 14 15:42:13 cn46 kernel: LustreError: 4597:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error (-107)  req@ffff8102043da400 x322225707/t0 o101-><?>@<?>:-1 lens 232/0 ref 0 fl Interpret:/0/0 rc -107/0
Mar 14 15:53:27 cn46 kernel: Lustre: 4255:0:(router.c:167:lnet_notify()) Ignoring prediction from 10.128.15.46@tcp of 10.128.15.37@tcp down 793928613593 seconds in the future
Mar 14 15:53:30 cn46 kernel: Lustre: lustre-OST0003: haven't heard from client 295f21a1-f8bf-e743-3e99-b39f1d03ba0a (at 10.128.15.37@tcp) in 677 seconds. I think it's dead, and I am evicting it.
Mar 14 16:04:02 cn46 kernel: LustreError: 4360:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16958773: rc -2
Mar 14 16:09:27 cn46 kernel: Lustre: lustre-OST0003: haven't heard from client 5b3a8229-84bf-1acd-fbb2-17ff6432ba20 (at 10.128.15.40@tcp) in 677 seconds. I think it's dead, and I am evicting it.
Mar 14 16:11:09 cn46 kernel: LustreError: 4564:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16960205: rc -2
Mar 14 16:27:32 cn46 kernel: LustreError: 31235:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error (-107)  req@ffff81021903f800 x201140724/t0 o13-><?>@<?>:-1 lens 128/0 ref 0 fl Interpret:/0/0 rc -107/0
Mar 14 16:31:58 cn46 kernel: Lustre: 4255:0:(router.c:167:lnet_notify()) Ignoring prediction from 10.128.15.46@tcp of 10.128.15.40@tcp down 796796849282 seconds in the future
Mar 14 16:41:18 cn46 kernel: Lustre: lustre-OST0003: haven't heard from client 5b3a8229-84bf-1acd-fbb2-17ff6432ba20 (at 10.128.15.40@tcp) in 677 seconds. I think it's dead, and I am evicting it.
Mar 14 16:47:05 cn46 kernel: LustreError: 4404:0:(client.c:975:ptlrpc_expire_one_request()) @@@ timeout (sent at 1205509605, 20s ago)  req@ffff8102043b3e00 x5425998/t0 o106->@NET_0x200000a800f02_UUID:15 lens 232/168 ref 1 fl Rpc:/0/0 rc 0/-22
Mar 14 16:47:05 cn46 kernel: LustreError: 138-a: lustre-OST0003: A client on nid 10.128.15.2@tcp was evicted due to a lock glimpse callback to 10.128.15.2@tcp timed out: rc -110
Mar 14 16:52:28 cn46 kernel: LustreError: 4658:0:(ldlm_lockd.c:646:ldlm_server_completion_ast()) ### enqueue wait took 8090868725us from 1205501857 ns: filter-lustre-OST0003_UUID lock: ffff8101b6579780/0xe31104df34352fa1 lrc: 2/0,0 mode: PR/PR res: 16939020/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 10020 remote: 0xd1f759c14d5603b6 expref: 299 pid: 4538
Mar 14 16:52:28 cn46 kernel: LustreError: 4849:0:(ldlm_lockd.c:646:ldlm_server_completion_ast()) ### enqueue wait took 3979699997us from 1205505968 ns: filter-lustre-OST0003_UUID lock: ffff810189b70080/0xe31104df34467bc9 lrc: 2/0,0 mode: PR/PR res: 16955674/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 10020 remote: 0xba313509094b9c32 expref: 195 pid: 4558
Mar 14 17:11:37 cn46 kernel: LustreError: 4508:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16971044: rc -2
Mar 14 17:14:00 cn46 kernel: LustreError: 4378:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16971695: rc -2
Mar 14 17:15:00 cn46 kernel: LustreError: 4354:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16971941: rc -2
Mar 14 17:16:43 cn46 kernel: LustreError: 4410:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16972422: rc -2
Mar 14 17:24:58 cn46 kernel: LustreError: 4402:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16974734: rc -2
Mar 14 17:27:17 cn46 kernel: LustreError: 4470:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16975392: rc -2
Mar 14 17:30:45 cn46 kernel: LustreError: 4598:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16976232: rc -2
Mar 14 17:33:31 cn46 kernel: LustreError: 4388:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16976908: rc -2
Mar 14 17:37:45 cn46 kernel: LustreError: 4458:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16977918: rc -2
Mar 14 17:40:16 cn46 kernel: LustreError: 4532:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16978516: rc -2
Mar 14 17:42:25 cn46 kernel: LustreError: 4575:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16979079: rc -2
Mar 14 17:42:33 cn46 kernel: Lustre: lustre-OST0003: haven't heard from client 4600a146-6e5d-72fe-f398-1aca7050cdbf (at 10.128.15.16@tcp) in 677 seconds. I think it's dead, and I am evicting it.
Mar 14 17:47:57 cn46 kernel: LustreError: 4526:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16980493: rc -2
Mar 14 17:47:57 cn46 kernel: LustreError: 4526:0:(ldlm_resource.c:719:ldlm_resource_add()) Skipped 1 previous similar message
Mar 14 18:06:53 cn46 kernel: LustreError: 4425:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16985208: rc -2
Mar 14 18:06:53 cn46 kernel: LustreError: 4425:0:(ldlm_resource.c:719:ldlm_resource_add()) Skipped 4 previous similar messages
Mar 14 18:44:19 cn46 kernel: Lustre: lustre-OST0003: haven't heard from client 1b90c7bb-d7a7-9888-04de-245c38a7aa08 (at 10.128.15.26@tcp) in 677 seconds. I think it's dead, and I am evicting it.
Mar 14 19:07:34 cn46 kernel: LustreError: 4544:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 16997640: rc -2
Mar 14 19:17:49 cn46 kernel: Lustre: lustre-OST0003: haven't heard from client d40aeb1a-17f3-bf85-62f5-0375f9434dfa (at 10.128.15.14@tcp) in 677 seconds. I think it's dead, and I am evicting it.
Mar 14 19:39:51 cn46 kernel: Lustre: lustre-OST0003: haven't heard from client 41e8f551-29dd-dc5c-e7d4-73e3fb01520b (at 10.128.15.11@tcp) in 677 seconds. I think it's dead, and I am evicting it.
Mar 14 21:22:20 cn46 kernel: Lustre: lustre-OST0003: haven't heard from client 43df24d8-c583-c6e1-42b7-496d14ca4740 (at 10.128.15.7@tcp) in 677 seconds. I think it's dead, and I am evicting it.
Mar 14 21:25:38 cn46 kernel: LustreError: 4996:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error (-107)  req@ffff8102043be000 x55016870/t0 o13-><?>@<?>:-1 lens 128/0 ref 0 fl Interpret:/0/0 rc -107/0
Mar 14 21:36:55 cn46 kernel: Lustre: lustre-OST0003: haven't heard from client 41e8f551-29dd-dc5c-e7d4-73e3fb01520b (at 10.128.15.11@tcp) in 677 seconds. I think it's dead, and I am evicting it.
Mar 14 21:40:01 cn46 auditd[2698]: Audit daemon rotating log files
Mar 14 21:41:34 cn46 kernel: LustreError: 4533:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for resource 17017363: rc -2
