<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Dear All,<div><br class="webkit-block-placeholder"></div><div>I am seeing following errors on MDS:</div><div><br class="webkit-block-placeholder"></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:14:13 mds01 kernel: LustreError: 17076:0:(mds_open.c:1474:mds_close()) Skipped 139 previous similar messages</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:14:27 mds01 kernel: LustreError: 17088:0:(mds_open.c:1474:mds_close()) @@@ no handle for file close ino 26997837: cookie 0x47a2a9d95b67cfb6 req@0000010094367c00 x32950/t0 o35->451db1a1-8c58-f825-eaab-a1dd24586e93@NET_0x200000a8f092e_UUID:-1 lens 296/560 ref 0 fl Interpret:/0/0 rc 0/0</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:14:32 mds01 kernel: LustreError: 17089:0:(mds_open.c:1474:mds_close()) @@@ no handle for file close ino 28697676: cookie 0x47a2a9d964ee6f0e req@000001009181d400 x28502/t0 o35->e838fcbc-4b8c-f448-a5d2-5e472e474229@NET_0x200000a8f060b_UUID:-1 lens 296/560 ref 0 fl Interpret:/0/0 rc 0/0</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:14:32 mds01 kernel: LustreError: 17089:0:(mds_open.c:1474:mds_close()) Skipped 4 previous similar messages</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:14:47 mds01 kernel: LustreError: 17061:0:(mds_open.c:1474:mds_close()) @@@ no handle for file close ino 28697676: cookie 0x47a2a9d964ee7ade req@00000100cd26ec00 x113640/t0 o35->d774ea81-c2ee-01a7-0d28-87054e92c858@NET_0x200000a8f0510_UUID:-1 lens 296/560 ref 0 fl Interpret:/0/0 rc 0/0</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:14:47 mds01 kernel: LustreError: 17061:0:(mds_open.c:1474:mds_close()) Skipped 2 previous similar messages</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:15:47 mds01 kernel: Lustre: ddn-home-MDT0000: haven't heard from client 9e6c2d9a-1649-3c61-0fda-b5052af0e09f (at 10.143.5.11@tcp) in 227 seconds. I think it's dead, and I am evicting it.</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:15:47 mds01 kernel: Lustre: Skipped 33 previous similar messagesNov 1 12:15:55 mds01 kernel: LustreError: 17076:0:(mds_open.c:1474:mds_close()) @@@ no handle for file close ino 25301776: cookie 0x47a2a9d95b4c6238 req@00000100cd217c00 x211014/t0 o35->31eec1e1-1f7d-a43b-ed8c-9841a694da28@NET_0x200000a8f0421_UUID:-1 lens 296/560 ref 0 fl Interpret:/0/0 rc 0/0</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:15:55 mds01 kernel: LustreError: 17076:0:(mds_open.c:1474:mds_close()) Skipped 24 previous similar messagesNov 1 12:15:55 mds01 kernel: LustreError: 17076:0:(ldlm_lib.c:1437:target_send_reply_msg()) @@@ processing error (-116) req@00000100cd217c00 x211014/t0 o35->31eec1e1-1f7d-a43b-ed8c-9841a694da28@NET_0x200000a8f0421_UUID:-1 lens 296/560 ref 0 fl Interpret:/0/0 rc -116/0</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:15:55 mds01 kernel: LustreError: 17076:0:(ldlm_lib.c:1437:target_send_reply_msg()) Skipped 39 previous similar messagesNov 1 12:17:00 mds01 kernel: LustreError: 16649:0:(mds_open.c:1474:mds_close()) @@@ no handle for file close ino 28968880: cookie 0x47a2a9d95bcda4b2 req@00000100c26b4800 x58156/t0 o35->1ed2c692-aea1-c9b8-ee90-6e8d4269cda8@NET_0x200000a8f0503_UUID:-1 lens 296/560 ref 0 fl Interpret:/0/0 rc 0/0</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:17:00 mds01 kernel: LustreError: 16649:0:(mds_open.c:1474:mds_close()) Skipped 2 previous similar messages</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:18:00 mds01 kernel: LustreError: 17066:0:(mds_open.c:1474:mds_close()) @@@ no handle for file close ino 28968880: cookie 0x47a2a9d95bcd8dd6 req@00000100ccf76a00 x42691/t0 o35->0a5afd52-cffe-0bff-9c69-e6027f201f5f@NET_0x200000a8f0424_UUID:-1 lens 296/560 ref 0 fl Interpret:/0/0 rc 0/0</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:18:00 mds01 kernel: LustreError: 17066:0:(mds_open.c:1474:mds_close()) Skipped 1 previous similar message</font><br><br></div><div>and OSS are showing following errors:</div><div><br class="webkit-block-placeholder"></div><div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:21:35 storage10.beowulf.cluster kernel: LustreError: 23337:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@00000100b0211000 x434527/t0 o101-><a href="mailto:MGS@MGC10.143.245.201">MGS@MGC10.143.245.201</a>@tcp_0:26 lens 232/240 ref 1 fl Rpc:/0/0 rc 0/0</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:21:35 storage10.beowulf.cluster kernel: LustreError: 23337:0:(client.c:519:ptlrpc_import_delay_req()) Skipped 119 previous similar messages</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:21:59 storage08.beowulf.cluster kernel: LustreError: 22609:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@00000100af90a800 x755145/t0 o101-><a href="mailto:MGS@MGC10.143.245.201">MGS@MGC10.143.245.201</a>@tcp_0:26 lens 232/240 ref 1 fl Rpc:/0/0 rc 0/0</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:21:59 storage08.beowulf.cluster kernel: LustreError: 22609:0:(client.c:519:ptlrpc_import_delay_req()) Skipped 119 previous similar messages</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:23:31 storage09.beowulf.cluster kernel: LustreError: 23045:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@00000100c31fc600 x511984/t0 o101-><a href="mailto:MGS@MGC10.143.245.201">MGS@MGC10.143.245.201</a>@tcp_0:26 lens 232/240 ref 1 fl Rpc:/0/0 rc 0/0</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:23:31 storage09.beowulf.cluster kernel: LustreError: 23045:0:(client.c:519:ptlrpc_import_delay_req()) Skipped 119 previous similar messages</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:24:32 storage07.beowulf.cluster kernel: LustreError: 22220:0:(client.c:519:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@0000010119e39400 x1064767/t0 o101-><a href="mailto:MGS@MGC10.143.245.201">MGS@MGC10.143.245.201</a>@tcp_0:26 lens 232/240 ref 1 fl Rpc:/0/0 rc 0/0</font></div><div><font class="Apple-style-span" face="Courier New">Nov 1 12:24:32 storage07.beowulf.cluster kernel: LustreError: 22220:0:(client.c:519:ptlrpc_import_delay_req()) Skipped 119 previous similar messages </font></div></div><div><br class="webkit-block-placeholder"></div><div>Does anybody has an idea what can be the reason of this errors?</div><div><br></div><div>My system consist of 4 OSS, 24 OST, 1 MDS, 585 clients </div><div>Lustre version is 1.6.3</div><div>Kernel version on the whole cluster is 2.6.9-55.0.9.EL_lustre.1.6.3smp</div><div><br class="webkit-block-placeholder"></div><div>Thanks for you help!</div><div><br class="webkit-block-placeholder"></div><div><div> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 11.0px Lucida Grande">Mr Wojciech Turek</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Lucida Grande; min-height: 13px; ">Assistant System Manager</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 11.0px Lucida Grande">University of Cambridge</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 11.0px Lucida Grande">High Performance Computing service </font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="'Lucida Grande'" size="3"><span class="Apple-style-span" style="font-size: 11px;">email: <a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a></span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 11.0px Lucida Grande">tel. +441223763517</font></div></div><div><br class="webkit-block-placeholder"></div></span><br class="Apple-interchange-newline"></span></span> </div><br></div></body></html>