[Lustre-discuss] Lustre error

[Ekaterina Popova]

Hello!
We have seen such error on our Lustre client (lustre-2.0.65) today. 
After that we even can't reboot our machine. Only hard reset helped. 
Lustre client is installed
on working node in clustre. Its CPU usage is high (100%). Several jobs 
are working on it.
How can I solve such problem? What can I do ?

Jul 20 11:04:47 wn0197 kernel: LustreError: 
2744:0:(client.c:2570:ptlrpc_replay_interpret()) Skipped 261 previous 
similar messages
Jul 21 09:51:19 wn0197 kernel: LustreError: 
30149:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff81008fcc6c40
Jul 21 09:51:19 wn0197 kernel: LustreError: 
30149:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 
(lihep-MDT0000-mdc-ffff8101ad8e6000), error 301
Jul 21 09:51:19 wn0197 kernel: LustreError: 
30149:0:(llite_lib.c:1384:ll_statfs_internal()) md_statfs fails: rc = 301
Jul 21 09:52:22 wn0197 kernel: LustreError: 
21610:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff810216d924c0
Jul 21 09:52:22 wn0197 kernel: LustreError: 
29193:0:(file.c:2165:ll_inode_revalidate_fini()) failure -4 inode 1769041
Jul 21 09:52:22 wn0197 kernel: LustreError: 
21610:0:(client.c:1896:ptlrpc_interrupted_set()) Skipped 1 previous 
similar message
Jul 21 09:52:22 wn0197 kernel: LustreError: 
21610:0:(file.c:2165:ll_inode_revalidate_fini()) failure -4 inode 1769041
Jul 21 09:52:23 wn0197 kernel: LustreError: 
4329:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff810207c92ec0
Jul 21 09:52:23 wn0197 kernel: LustreError: 
4329:0:(client.c:1896:ptlrpc_interrupted_set()) Skipped 8 previous 
similar messages
Jul 21 09:52:23 wn0197 kernel: LustreError: 
4329:0:(file.c:2165:ll_inode_revalidate_fini()) failure -4 inode 1769041
Jul 21 09:52:23 wn0197 kernel: LustreError: 
4329:0:(file.c:2165:ll_inode_revalidate_fini()) Skipped 8 previous 
similar messages
Jul 21 09:52:38 wn0197 kernel: LustreError: 
22439:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff8101a3fa3e40
Jul 21 09:52:38 wn0197 kernel: LustreError: 
21374:0:(file.c:2165:ll_inode_revalidate_fini()) failure -4 inode 1769041
Jul 21 09:52:38 wn0197 kernel: LustreError: 
21374:0:(file.c:2165:ll_inode_revalidate_fini()) Skipped 13 previous 
similar messages
Jul 21 09:52:38 wn0197 kernel: LustreError: 
22439:0:(client.c:1896:ptlrpc_interrupted_set()) Skipped 14 previous 
similar messages
Jul 21 09:53:03 wn0197 kernel: LustreError: 
21374:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff810184ac42c0
Jul 21 09:53:03 wn0197 kernel: LustreError: 
21374:0:(client.c:1896:ptlrpc_interrupted_set()) Skipped 8 previous 
similar messages
Jul 21 09:53:03 wn0197 kernel: LustreError: 
21374:0:(file.c:2165:ll_inode_revalidate_fini()) failure -4 inode 1769041
Jul 21 09:53:03 wn0197 kernel: LustreError: 
21374:0:(file.c:2165:ll_inode_revalidate_fini()) Skipped 9 previous 
similar messages
Jul 21 09:53:13 wn0197 kernel: LustreError: 
23467:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff81010f5b2b40
Jul 21 09:53:13 wn0197 kernel: LustreError: 
23467:0:(client.c:1896:ptlrpc_interrupted_set()) Skipped 6 previous 
similar messages
Jul 21 09:53:13 wn0197 kernel: LustreError: 
23467:0:(file.c:2165:ll_inode_revalidate_fini()) failure -4 inode 1769041
Jul 21 09:53:13 wn0197 kernel: LustreError: 
23467:0:(file.c:2165:ll_inode_revalidate_fini()) Skipped 6 previous 
similar messages
Jul 21 10:16:08 wn0197 kernel: LustreError: 
7183:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff81019ba27240
Jul 21 10:16:08 wn0197 kernel: LustreError: 
7183:0:(client.c:1896:ptlrpc_interrupted_set()) Skipped 6 previous 
similar messages
Jul 21 10:16:08 wn0197 kernel: LustreError: 
7183:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 
(lihep-MDT0000-mdc-ffff8101ad8e6000), error 301
Jul 21 10:16:08 wn0197 kernel: LustreError: 
7183:0:(llite_lib.c:1384:ll_statfs_internal()) md_statfs fails: rc = 301
Jul 21 10:16:58 wn0197 kernel: LustreError: 
7207:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff81017249d3c0
Jul 21 10:16:58 wn0197 kernel: LustreError: 
7207:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 
(lihep-MDT0000-mdc-ffff8101ad8e6000), error 301
Jul 21 10:16:58 wn0197 kernel: LustreError: 
7207:0:(llite_lib.c:1384:ll_statfs_internal()) md_statfs fails: rc = 301
Jul 21 10:18:53 wn0197 kernel: LustreError: 
7129:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff8101a7a59d40
Jul 21 10:18:53 wn0197 kernel: LustreError: 
7129:0:(file.c:2165:ll_inode_revalidate_fini()) failure -4 inode 1769041
Jul 21 10:18:53 wn0197 kernel: LustreError: 
7129:0:(file.c:2165:ll_inode_revalidate_fini()) Skipped 6 previous 
similar messages
Jul 21 10:19:41 wn0197 kernel: LustreError: 
7129:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff8101f234f840
Jul 21 10:19:41 wn0197 kernel: LustreError: 
7129:0:(file.c:2165:ll_inode_revalidate_fini()) failure -4 inode 1769041
Jul 21 10:25:11 wn0197 kernel: LustreError: 
7323:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff8101a1ad1240
Jul 21 10:25:11 wn0197 kernel: LustreError: 
7323:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 
(lihep-MDT0000-mdc-ffff8101ad8e6000), error 301
Jul 21 10:25:11 wn0197 kernel: LustreError: 
7323:0:(llite_lib.c:1384:ll_statfs_internal()) md_statfs fails: rc = 301
Jul 21 10:26:05 wn0197 kernel: LustreError: 
4022:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff8100af2dba40
Jul 21 10:26:05 wn0197 kernel: LustreError: 
4022:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 
(lihep-MDT0000-mdc-ffff8101ad8e6000), error 301
Jul 21 10:26:05 wn0197 kernel: LustreError: 
4022:0:(llite_lib.c:1384:ll_statfs_internal()) md_statfs fails: rc = 301
Jul 21 10:26:31 wn0197 kernel: LustreError: 
4014:0:(lmv_obd.c:1198:lmv_statfs()) can't stat MDS #0 
(lihep-MDT0000-mdc-ffff8101ad8e6000), error 301
Jul 21 10:26:31 wn0197 kernel: LustreError: 
4014:0:(llite_lib.c:1384:ll_statfs_internal()) md_statfs fails: rc = 301
Jul 21 10:30:15 wn0197 kernel: LustreError: 
7129:0:(client.c:1896:ptlrpc_interrupted_set()) INTERRUPTED SET 
ffff810186ae88c0
Jul 21 10:30:15 wn0197 kernel: LustreError: 
7129:0:(client.c:1896:ptlrpc_interrupted_set()) Skipped 1 previous 
similar message
Jul 21 10:30:15 wn0197 kernel: LustreError: 
7129:0:(file.c:2165:ll_inode_revalidate_fini()) failure -4 inode 1769041

With regards,
Popova Ekaterina