[Lustre-discuss] no handle for file close and tlrpc_import_delay_req

[Wojciech Turek]

Dear All,

I am seeing following errors on MDS:

Nov  1 12:14:13 mds01 kernel: LustreError: 17076:0:(mds_open.c: 
1474:mds_close()) Skipped 139 previous similar messages
Nov  1 12:14:27 mds01 kernel: LustreError: 17088:0:(mds_open.c: 
1474:mds_close()) @@@ no handle for file close ino 26997837: cookie  
0x47a2a9d95b67cfb6  req at 0000010094367c00 x32950/t0 o35->451db1a1-8c58- 
f825-eaab-a1dd24586e93 at NET_0x200000a8f092e_UUID:-1 lens 296/560 ref 0  
fl Interpret:/0/0 rc 0/0
Nov  1 12:14:32 mds01 kernel: LustreError: 17089:0:(mds_open.c: 
1474:mds_close()) @@@ no handle for file close ino 28697676: cookie  
0x47a2a9d964ee6f0e  req at 000001009181d400 x28502/t0 o35->e838fcbc-4b8c- 
f448-a5d2-5e472e474229 at NET_0x200000a8f060b_UUID:-1 lens 296/560 ref 0  
fl Interpret:/0/0 rc 0/0
Nov  1 12:14:32 mds01 kernel: LustreError: 17089:0:(mds_open.c: 
1474:mds_close()) Skipped 4 previous similar messages
Nov  1 12:14:47 mds01 kernel: LustreError: 17061:0:(mds_open.c: 
1474:mds_close()) @@@ no handle for file close ino 28697676: cookie  
0x47a2a9d964ee7ade  req at 00000100cd26ec00 x113640/t0 o35->d774ea81- 
c2ee-01a7-0d28-87054e92c858 at NET_0x200000a8f0510_UUID:-1 lens 296/560  
ref 0 fl Interpret:/0/0 rc 0/0
Nov  1 12:14:47 mds01 kernel: LustreError: 17061:0:(mds_open.c: 
1474:mds_close()) Skipped 2 previous similar messages
Nov  1 12:15:47 mds01 kernel: Lustre: ddn-home-MDT0000: haven't heard  
from client 9e6c2d9a-1649-3c61-0fda-b5052af0e09f (at 10.143.5.11 at tcp)  
in 227 seconds. I think it's dead, and I am evicting it.
Nov  1 12:15:47 mds01 kernel: Lustre: Skipped 33 previous similar  
messagesNov  1 12:15:55 mds01 kernel: LustreError: 17076:0: 
(mds_open.c:1474:mds_close()) @@@ no handle for file close ino  
25301776: cookie 0x47a2a9d95b4c6238  req at 00000100cd217c00 x211014/t0  
o35->31eec1e1-1f7d-a43b-ed8c-9841a694da28 at NET_0x200000a8f0421_UUID:-1  
lens 296/560 ref 0 fl Interpret:/0/0 rc 0/0
Nov  1 12:15:55 mds01 kernel: LustreError: 17076:0:(mds_open.c: 
1474:mds_close()) Skipped 24 previous similar messagesNov  1 12:15:55  
mds01 kernel: LustreError: 17076:0:(ldlm_lib.c: 
1437:target_send_reply_msg()) @@@ processing error (-116)   
req at 00000100cd217c00 x211014/t0 o35->31eec1e1-1f7d-a43b- 
ed8c-9841a694da28 at NET_0x200000a8f0421_UUID:-1 lens 296/560 ref 0 fl  
Interpret:/0/0 rc -116/0
Nov  1 12:15:55 mds01 kernel: LustreError: 17076:0:(ldlm_lib.c: 
1437:target_send_reply_msg()) Skipped 39 previous similar  
messagesNov  1 12:17:00 mds01 kernel: LustreError: 16649:0: 
(mds_open.c:1474:mds_close()) @@@ no handle for file close ino  
28968880: cookie 0x47a2a9d95bcda4b2  req at 00000100c26b4800 x58156/t0  
o35->1ed2c692-aea1-c9b8-ee90-6e8d4269cda8 at NET_0x200000a8f0503_UUID:-1  
lens 296/560 ref 0 fl Interpret:/0/0 rc 0/0
Nov  1 12:17:00 mds01 kernel: LustreError: 16649:0:(mds_open.c: 
1474:mds_close()) Skipped 2 previous similar messages
Nov  1 12:18:00 mds01 kernel: LustreError: 17066:0:(mds_open.c: 
1474:mds_close()) @@@ no handle for file close ino 28968880: cookie  
0x47a2a9d95bcd8dd6  req at 00000100ccf76a00 x42691/t0 o35->0a5afd52- 
cffe-0bff-9c69-e6027f201f5f at NET_0x200000a8f0424_UUID:-1 lens 296/560  
ref 0 fl Interpret:/0/0 rc 0/0
Nov  1 12:18:00 mds01 kernel: LustreError: 17066:0:(mds_open.c: 
1474:mds_close()) Skipped 1 previous similar message

and OSS are showing following errors:

Nov  1 12:21:35 storage10.beowulf.cluster kernel: LustreError:  
23337:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID   
req at 00000100b0211000 x434527/t0 o101->MGS at MGC10.143.245.201@tcp_0:26  
lens 232/240 ref 1 fl Rpc:/0/0 rc 0/0
Nov  1 12:21:35 storage10.beowulf.cluster kernel: LustreError:  
23337:0:(client.c:519:ptlrpc_import_delay_req()) Skipped 119 previous  
similar messages
Nov  1 12:21:59 storage08.beowulf.cluster kernel: LustreError:  
22609:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID   
req at 00000100af90a800 x755145/t0 o101->MGS at MGC10.143.245.201@tcp_0:26  
lens 232/240 ref 1 fl Rpc:/0/0 rc 0/0
Nov  1 12:21:59 storage08.beowulf.cluster kernel: LustreError:  
22609:0:(client.c:519:ptlrpc_import_delay_req()) Skipped 119 previous  
similar messages
Nov  1 12:23:31 storage09.beowulf.cluster kernel: LustreError:  
23045:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID   
req at 00000100c31fc600 x511984/t0 o101->MGS at MGC10.143.245.201@tcp_0:26  
lens 232/240 ref 1 fl Rpc:/0/0 rc 0/0
Nov  1 12:23:31 storage09.beowulf.cluster kernel: LustreError:  
23045:0:(client.c:519:ptlrpc_import_delay_req()) Skipped 119 previous  
similar messages
Nov  1 12:24:32 storage07.beowulf.cluster kernel: LustreError:  
22220:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID   
req at 0000010119e39400 x1064767/t0 o101->MGS at MGC10.143.245.201@tcp_0:26  
lens 232/240 ref 1 fl Rpc:/0/0 rc 0/0
Nov  1 12:24:32 storage07.beowulf.cluster kernel: LustreError:  
22220:0:(client.c:519:ptlrpc_import_delay_req()) Skipped 119 previous  
similar messages

Does anybody has an idea what can be the reason of this errors?

My system consist of 4 OSS, 24 OST, 1 MDS, 585 clients
Lustre version is 1.6.3
Kernel version on the whole cluster is 2.6.9-55.0.9.EL_lustre.1.6.3smp

Thanks for you help!

Mr Wojciech Turek
Assistant System Manager
University of Cambridge
High Performance Computing service
email: wjt27 at cam.ac.uk
tel. +441223763517



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20071101/029fdd89/attachment.htm>