<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Arial","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=FR-CA link=blue vlink=purple>
<div class=WordSection1>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'>I have a cluster front node that is configured as a Lustre
server. The mgs, mdt and ost all reside on this system.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'>The mgs and the mdt are located on a RAID-1 disk on the first
RAID controller. The ost is located on a RAID-5 (six 1.0 TB disks)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'>on the second RAID controller. The system is a dual Intel quad-core
Nehalem processor with 6 GB of RAM. Hyper-threading is on.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'>The server is running F7 with patched kernel 2.6.22.14-72 and
Lustre 1.6.7.2. The server is also a client of itself<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'>(mounted on /home1) in addition to the 64 compute nodes and a
few workstations.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'>When I load the server with the command “find /home1 -inum
100”, the following error messages are written to the<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'>server system logfile (/var/log/messages):<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:08:37 fn1
kernel: Lustre: Request x260093 sent from fn1home1-OST0000-osc to NID 0@lo 50s
a<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>go has timed out (limit
50s).<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:08:37 fn1
kernel: Lustre: fn1home1-OST0000-osc: Connection to service fn1home1-OST0000
via<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> nid 0@lo was lost; in
progress operations using this service will wait for recovery to complete. <o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:08:37 fn1
kernel: Lustre: 3580:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00: fn1home1-mdtlov_UUID
reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:08:37 fn1
kernel: Lustre: 3580:0:(ldlm_lib.c:780:target_handle_connect())
fn1home1-OST0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
fn1home1-mdtlov_UUID@0@lo to 0xffff8101e9cba000; still busy with 2 active<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:08:37 fn1
kernel: LustreError: 3580:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@
proces<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sing error (-16)
req@ffff81015e86a800 x260114/t0
o8->fn1home1-mdtlov_UUID@172.17.15.20@tcp:0/0 lens<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> 304/200 e 0 to 0 dl
1286291417 ref 1 fl Interpret:/0/0 rc -16/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:08:37 fn1
kernel: LustreError: 3580:0:(ldlm_lib.c:1643:target_send_reply_msg()) Skipped
19<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:08:37 fn1
kernel: LustreError: 11-0: an error occurred while communicating with 0@lo. The
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>ost_connect operation failed
with -16<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:02 fn1
kernel: Lustre: 3152:0:(import.c:507:import_select_connection())
fn1home1-OST000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>0-osc: tried all
connections, increasing latency to 6s<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:02 fn1
kernel: Lustre: 3521:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00: fn1home1-mdtlov_UUID
reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:02 fn1
kernel: Lustre: 3521:0:(ldlm_lib.c:780:target_handle_connect())
fn1home1-OST0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
fn1home1-mdtlov_UUID@0@lo to 0xffff8101e9cba000; still busy with 2 active<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:02 fn1
kernel: LustreError: 3521:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@
proces<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sing error (-16)
req@ffff810112f3f850 x260116/t0
o8->fn1home1-mdtlov_UUID@172.17.15.20@tcp:0/0 lens<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> 304/200 e 0 to 0 dl
1286291442 ref 1 fl Interpret:/0/0 rc -16/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:02 fn1
kernel: LustreError: 11-0: an error occurred while communicating with 0@lo. The
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>ost_connect operation failed
with -16<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: Lustre: Request x260076 sent from fn1home1-OST0000-osc-ffff8101ee411000
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>to NID 0@lo 100s ago has
timed out (limit 100s).<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: Lustre: fn1home1-OST0000-osc-ffff8101ee411000: Connection to service
fn1<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>home1-OST0000 via nid 0@lo
was lost; in progress operations using this service will wait for recover<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>y to complete. <o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: Lustre: 3591:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
12f4dd8a-1056-861c-2a01-70095593af9f reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: Lustre: 3591:0:(ldlm_lib.c:780:target_handle_connect())
fn1home1-OST0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
12f4dd8a-1056-861c-2a01-70095593af9f@0@lo to 0xffff8101119f6000; still bu<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sy with 3 active RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: LustreError: 11-0: an error occurred while communicating with 0@lo. The
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>ost_connect operation failed
with -16<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: LustreError: 3542:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC:
canceling anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: LustreError: 3542:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: LustreError: 3542:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC:
canceling anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: LustreError: 3542:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: LustreError: 3542:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC:
canceling anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: LustreError: 3542:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Skipped
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>136 previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: LustreError: 3542:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: LustreError: 3542:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
Skipped<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> 136 previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: LustreError: 3355:0:(filter.c:1229:filter_parent_lock())
fn1home1-OST000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>0: slow parent lock 100s<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: LustreError: 3596:0:(filter.c:1229:filter_parent_lock()) fn1home1-OST000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>0: slow parent lock 64s<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: Lustre: 6833:0:(service.c:1317:ptlrpc_server_handle_request()) @@@
Reque<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>st x260093 took longer than
estimated (50+33s); client may timeout. req@ffff810158400400 x260093/t0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> o5->fn1home1-mdtlov_UUID@172.17.15.20@tcp:0/0
lens 336/336 e 0 to 0 dl 1286291317 ref 1 fl Complete<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>:/0/0 rc 0/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:10 fn1
kernel: LustreError: 3596:0:(filter.c:1229:filter_parent_lock()) Skipped 1
previ<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>ous similar message<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:35 fn1
kernel: Lustre: 3152:0:(import.c:507:import_select_connection())
fn1home1-OST000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>0-osc: tried all
connections, increasing latency to 11s<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:35 fn1
kernel: Lustre: 3524:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
12f4dd8a-1056-861c-2a01-70095593af9f reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:35 fn1
kernel: Lustre: fn1home1-OST0000-osc: Connection restored to service
fn1home1-OS<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>T0000 using nid 0@lo.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:35 fn1
kernel: Lustre: 3524:0:(ldlm_lib.c:541:target_handle_reconnect()) Skipped 1
prev<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>ious similar message<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:35 fn1
kernel: Lustre: fn1home1-OST0000: received MDS connection from 0@lo<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:35 fn1
kernel: Lustre: MDS fn1home1-MDT0000: fn1home1-OST0000_UUID now active,
resettin<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>g orphans<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:35 fn1
kernel: Lustre: 3619:0:(filter.c:3077:filter_precreate()) fn1home1-OST0000:
prec<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>reate aborted by destroy<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:09:35 fn1
kernel: Lustre: 3566:0:(filter.c:2849:filter_destroy_precreated())
fn1home1-OST0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>000: deleting orphan objects
from 3096267 to 3096299<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:10 fn1
kernel: Lustre: 3513:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
a80ed6b4-a776-a472-f0cb-43d1af16ebce reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:10 fn1
kernel: Lustre: 3513:0:(ldlm_lib.c:780:target_handle_connect())
fn1home1-OST0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
a80ed6b4-a776-a472-f0cb-43d1af16ebce@172.17.15.18@tcp to 0xffff81011959c0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00; still busy with 2 active
RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:10 fn1
kernel: LustreError: 3513:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@
proces<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sing error (-16) req@ffff8101e919dc00
x1348683852632032/t0 o8->a80ed6b4-a776-a472-f0cb-43d1af16ebce<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>@NET_0x20000ac110f12_UUID:0/0
lens 368/200 e 0 to 0 dl 1286293490 ref 1 fl Interpret:/0/0 rc -16/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:10 fn1
kernel: LustreError: 3513:0:(ldlm_lib.c:1643:target_send_reply_msg()) Skipped 1
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>previous similar message<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:17 fn1
kernel: Lustre: 3564:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
a80ed6b4-a776-a472-f0cb-43d1af16ebce reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:17 fn1
kernel: Lustre: 3564:0:(ldlm_lib.c:780:target_handle_connect())
fn1home1-OST0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
a80ed6b4-a776-a472-f0cb-43d1af16ebce@172.17.15.18@tcp to 0xffff81011959c0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00; still busy with 2 active
RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:17 fn1
kernel: LustreError: 3564:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@
proces<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sing error (-16)
req@ffff8101e97a9400 x1348683852632033/t0
o8->a80ed6b4-a776-a472-f0cb-43d1af16ebce<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>@NET_0x20000ac110f12_UUID:0/0
lens 368/200 e 0 to 0 dl 1286293497 ref 1 fl Interpret:/0/0 rc -16/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:24 fn1
kernel: Lustre: 3539:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
a80ed6b4-a776-a472-f0cb-43d1af16ebce reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:24 fn1
kernel: Lustre: 3539:0:(ldlm_lib.c:780:target_handle_connect())
fn1home1-OST0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
a80ed6b4-a776-a472-f0cb-43d1af16ebce@172.17.15.18@tcp to 0xffff81011959c0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00; still busy with 2 active
RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:24 fn1
kernel: LustreError: 3539:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@
proces<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sing error (-16)
req@ffff8101dd1d1400 x1348683852632034/t0
o8->a80ed6b4-a776-a472-f0cb-43d1af16ebce<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>@NET_0x20000ac110f12_UUID:0/0
lens 368/200 e 0 to 0 dl 1286293504 ref 1 fl Interpret:/0/0 rc -16/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:31 fn1
kernel: Lustre: 3563:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
a80ed6b4-a776-a472-f0cb-43d1af16ebce reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:31 fn1
kernel: Lustre: 3563:0:(ldlm_lib.c:780:target_handle_connect())
fn1home1-OST0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
a80ed6b4-a776-a472-f0cb-43d1af16ebce@172.17.15.18@tcp to 0xffff81011959c0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00; still busy with 2 active
RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:31 fn1
kernel: LustreError: 3563:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@
proces<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sing error (-16)
req@ffff810172c25000 x1348683852632035/t0
o8->a80ed6b4-a776-a472-f0cb-43d1af16ebce<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>@NET_0x20000ac110f12_UUID:0/0
lens 368/200 e 0 to 0 dl 1286293511 ref 1 fl Interpret:/0/0 rc -16/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:38 fn1
kernel: Lustre: 3557:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
a80ed6b4-a776-a472-f0cb-43d1af16ebce reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:38 fn1 kernel:
Lustre: 3557:0:(ldlm_lib.c:780:target_handle_connect()) fn1home1-OST0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
a80ed6b4-a776-a472-f0cb-43d1af16ebce@172.17.15.18@tcp to 0xffff81011959c0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00; still busy with 2 active
RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:45 fn1
kernel: Lustre: 3498:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
a80ed6b4-a776-a472-f0cb-43d1af16ebce reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:45 fn1
kernel: Lustre: 3498:0:(ldlm_lib.c:780:target_handle_connect())
fn1home1-OST0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
a80ed6b4-a776-a472-f0cb-43d1af16ebce@172.17.15.18@tcp to 0xffff81011959c0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00; still busy with 2 active
RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:45 fn1
kernel: LustreError: 3498:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@
proces<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sing error (-16)
req@ffff8101dd1d1c00 x1348683852632039/t0 o8->a80ed6b4-a776-a472-f0cb-43d1af16ebce<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>@NET_0x20000ac110f12_UUID:0/0
lens 368/200 e 0 to 0 dl 1286293525 ref 1 fl Interpret:/0/0 rc -16/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:45 fn1
kernel: LustreError: 3498:0:(ldlm_lib.c:1643:target_send_reply_msg()) Skipped 1
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>previous similar message<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:59 fn1
kernel: Lustre: 3560:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
a80ed6b4-a776-a472-f0cb-43d1af16ebce reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:59 fn1
kernel: Lustre: 3560:0:(ldlm_lib.c:541:target_handle_reconnect()) Skipped 1
prev<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>ious similar message<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:59 fn1
kernel: Lustre: 3560:0:(ldlm_lib.c:780:target_handle_connect())
fn1home1-OST0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
a80ed6b4-a776-a472-f0cb-43d1af16ebce@172.17.15.18@tcp to 0xffff81011959c0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00; still busy with 2 active
RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:43:59 fn1
kernel: Lustre: 3560:0:(ldlm_lib.c:780:target_handle_connect()) Skipped 1
previo<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>us similar message<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:06 fn1
kernel: LustreError: 3563:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@
proces<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sing error (-16)
req@ffff810137723c00 x1348683852632044/t0
o8->a80ed6b4-a776-a472-f0cb-43d1af16ebce<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>@NET_0x20000ac110f12_UUID:0/0
lens 368/200 e 0 to 0 dl 1286293546 ref 1 fl Interpret:/0/0 rc -16/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:06 fn1
kernel: LustreError: 3563:0:(ldlm_lib.c:1643:target_send_reply_msg()) Skipped 2
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>previous similar messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:20 fn1
kernel: Lustre: 3553:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
a80ed6b4-a776-a472-f0cb-43d1af16ebce reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:20 fn1
kernel: Lustre: 3553:0:(ldlm_lib.c:541:target_handle_reconnect()) Skipped 2
prev<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>ious similar messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:20 fn1
kernel: Lustre: 3553:0:(ldlm_lib.c:780:target_handle_connect())
fn1home1-OST0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
a80ed6b4-a776-a472-f0cb-43d1af16ebce@172.17.15.18@tcp to 0xffff81011959c0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00; still busy with 2 active
RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:20 fn1
kernel: Lustre: 3553:0:(ldlm_lib.c:780:target_handle_connect()) Skipped 2
previo<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>us similar messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:41 fn1
kernel: LustreError: 3550:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@
proces<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sing error (-16)
req@ffff81011af54c50 x1348683852632051/t0
o8->a80ed6b4-a776-a472-f0cb-43d1af16ebce<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>@NET_0x20000ac110f12_UUID:0/0
lens 368/200 e 0 to 0 dl 1286293581 ref 1 fl Interpret:/0/0 rc -16/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:41 fn1
kernel: LustreError: 3550:0:(ldlm_lib.c:1643:target_send_reply_msg()) Skipped 4
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>previous similar messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: Lustre: Request x613043 sent from fn1home1-OST0000-osc-ffff8101ee411000
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>to NID 0@lo 100s ago has
timed out (limit 100s).<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: Lustre: Skipped 1 previous similar message<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: Lustre: fn1home1-OST0000-osc-ffff8101ee411000: Connection to service
fn1<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>home1-OST0000 via nid 0@lo
was lost; in progress operations using this service will wait for recover<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>y to complete.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: LustreError: 9759:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC:
canceling anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: LustreError: 11-0: an error occurred while communicating with 0@lo. The
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>ost_connect operation failed
with -16<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: LustreError: 9759:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Skipped
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>124 previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: LustreError: 9759:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: LustreError: 9759:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
Skipped<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> 124 previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: LustreError: 9759:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC:
canceling anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: LustreError: 9759:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Skipped
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>9 previous similar messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: LustreError: 9759:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: LustreError: 9759:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
Skipped<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> 9 previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:43 fn1
kernel: Lustre: Request x613079 sent from fn1home1-OST0000-osc-ffff8101ee411000
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>to NID 0@lo 100s ago has
timed out (limit 100s).<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:44 fn1
kernel: LustreError: 9759:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC:
canceling anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:44 fn1
kernel: LustreError: 9759:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Skipped
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>97 previous similar messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:44 fn1
kernel: LustreError: 9759:0:(ldlm_request.c:1622:ldlm_cli_cancel_list()) ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:44 fn1
kernel: LustreError: 9759:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
Skipped<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> 97 previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:44 fn1
kernel: Lustre: Request x613181 sent from fn1home1-OST0000-osc-ffff8101ee411000
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>to NID 0@lo 100s ago has
timed out (limit 100s).<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:44 fn1
kernel: LustreError: 3609:0:(filter.c:1229:filter_parent_lock())
fn1home1-OST000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>0: slow parent lock 100s<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:44 fn1
kernel: Lustre: 3609:0:(service.c:1317:ptlrpc_server_handle_request()) @@@
Reque<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>st x1348683852632028 took
longer than estimated (6+95s); client may timeout. </span><span
style='font-size:10.0pt;font-family:"Courier New"'>req@ffff810137723000 x<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Courier New"'>1348683852632028/t0
o101->a80ed6b4-a776-a472-f0cb-43d1af16ebce@NET_0x20000ac110f12_UUID:0/0 lens
296<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>/288 e 0 to 0 dl 1286293389
ref 1 fl Complete:/0/0 rc 0/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:44:45 fn1
kernel: Lustre: 3351:0:(service.c:1317:ptlrpc_server_handle_request()) @@@
Reque<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>st x613079 took longer than
estimated (100+2s); client may timeout. </span><span style='font-size:
10.0pt;font-family:"Courier New"'>req@ffff81011d843c00 x613079/t0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Courier New"'> o103->12f4dd8a-1056-861c-2a01-70095593af9f@172.17.15.20@tcp:0/0
lens 688/128 e 0 to 0 dl 1286293483<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Courier New"'> </span><span lang=EN-US style='font-size:10.0pt;
font-family:"Courier New"'>ref 1 fl Complete:/0/0 rc 0/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:45:08 fn1
kernel: Lustre: 3152:0:(import.c:507:import_select_connection())
fn1home1-OST000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>0-osc-ffff8101ee411000:
tried all connections, increasing latency to 6s<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:45:08 fn1
kernel: Lustre: 3152:0:(import.c:507:import_select_connection()) Skipped 1
previ<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>ous similar message<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:45:08 fn1
kernel: Lustre: 3575:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-OST00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
12f4dd8a-1056-861c-2a01-70095593af9f reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:45:08 fn1
kernel: Lustre: 3575:0:(ldlm_lib.c:541:target_handle_reconnect()) Skipped 5
prev<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>ious similar messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:45:08 fn1
kernel: Lustre: fn1home1-OST0000-osc-ffff8101ee411000: Connection restored to
se<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>rvice fn1home1-OST0000 using
nid 0@lo.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 11:45:08 fn1
kernel: Lustre: Skipped 1 previous similar message<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: Lustre: Request x858420 sent from fn1home1-MDT0000-mdc-ffff8101ee411000
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>to NID 0@lo 100s ago has
timed out (limit 100s).<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: Lustre: fn1home1-MDT0000-mdc-ffff8101ee411000: Connection to service
fn1<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>home1-MDT0000 via nid 0@lo
was lost; in progress operations using this service will wait for recover<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>y to complete.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: Lustre: 3419:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-MDT00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
12f4dd8a-1056-861c-2a01-70095593af9f reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: Lustre: 3419:0:(ldlm_lib.c:780:target_handle_connect()) fn1home1-MDT0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
12f4dd8a-1056-861c-2a01-70095593af9f@0@lo to 0xffff8101e9f32000; still bu<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sy with 2 active RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: Lustre: 3419:0:(ldlm_lib.c:780:target_handle_connect()) Skipped 4
previo<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>us similar messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: LustreError: 3419:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@
proces<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sing error (-16)
req@ffff810115e7d450 x858794/t0
o38->12f4dd8a-1056-861c-2a01-70095593af9f@172.17.1<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>5.20@tcp:0/0 lens 304/200 e
0 to 0 dl 1286296096 ref 1 fl Interpret:/0/0 rc -16/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: LustreError: 3419:0:(ldlm_lib.c:1643:target_send_reply_msg()) Skipped 1
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>previous similar message<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC:
canceling anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Skipped
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>219 previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
Skipped<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> 219 previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: LustreError: 11-0: an error occurred while communicating with 0@lo. The
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>mds_connect operation failed
with -16<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC:
canceling anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:36 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:37 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC:
canceling anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:37 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:38 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC:
canceling anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:38 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Skipped
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>2 previous similar messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:38 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:38 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
Skipped<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> 2 previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:40 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC: canceling
anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:40 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Skipped
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>4 previous similar messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:40 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:40 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
Skipped<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> 4 previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:45 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC: canceling
anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:45 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Skipped
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>7 previous similar messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:45 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:45 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
Skipped<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> 7 previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:53 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Got rc
-<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>11 from cancel RPC:
canceling anyway<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:53 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1033:ldlm_cli_cancel_req()) Skipped
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>18 previous similar messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:53 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
ldlm_cl<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>i_cancel_list: -11<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:26:53 fn1
kernel: LustreError: 3417:0:(ldlm_request.c:1622:ldlm_cli_cancel_list())
Skipped<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> 18 previous similar
messages<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:27:01 fn1
kernel: Lustre: 3152:0:(import.c:507:import_select_connection())
fn1home1-MDT000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>0-mdc-ffff8101ee411000:
tried all connections, increasing latency to 6s<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:27:01 fn1
kernel: Lustre: 3408:0:(ldlm_lib.c:541:target_handle_reconnect())
fn1home1-MDT00<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>00:
12f4dd8a-1056-861c-2a01-70095593af9f reconnecting<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:27:01 fn1
kernel: Lustre: 3408:0:(ldlm_lib.c:780:target_handle_connect())
fn1home1-MDT0000<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>: refuse reconnection from
12f4dd8a-1056-861c-2a01-70095593af9f@0@lo to 0xffff8101e9f32000; still bu<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sy with 2 active RPCs<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:27:01 fn1
kernel: LustreError: 3408:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@
proces<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>sing error (-16)
req@ffff810197228400 x858911/t0
o38->12f4dd8a-1056-861c-2a01-70095593af9f@172.17.1<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>5.20@tcp:0/0 lens 304/200 e
0 to 0 dl 1286296121 ref 1 fl Interpret:/0/0 rc -16/0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>Oct 5 12:27:01 fn1
kernel: LustreError: 11-0: an error occurred while communicating with 0@lo. The
<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>mds_connect operation failed
with -16<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'>And so on…<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'>What are these messages trying to tell me?<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:
"Arial","sans-serif";color:navy'>Denis Charland, ing. </span></b><b><span
lang=EN-CA style='font-size:10.0pt;font-family:"Arial","sans-serif";color:navy'>|
P. Eng.</span></b><b><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:navy'><o:p></o:p></span></b></p>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Arial","sans-serif";
color:navy'>Administrateur de Systèmes UNIX | UNIX Systems Administrator<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Arial","sans-serif";
color:navy'>Tél. | tel. (450) 641-5078 Fax (450)
641-5106<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Arial","sans-serif";
color:navy'>Courriel | E-mail :</span><span style='font-size:9.0pt;font-family:
"Arial","sans-serif";color:#1F497D'> </span><span lang=EN-US style='font-size:
10.0pt;font-family:"Arial","sans-serif";color:blue'><a
href="mailto:denis.charland@cnrc-nrc.gc.ca"><span lang=FR-CA style='color:blue'>denis.charland@cnrc-nrc.gc.ca</span></a></span><span
lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'>
</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif";
color:#1F497D'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif";
color:#1F497D'> <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Arial","sans-serif";
color:navy'>Institut des matériaux industriels | Industrial Materials
Institute<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Arial","sans-serif";
color:navy'>Conseil national de recherches Canada | National Research Council
Canada<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:9.0pt;font-family:"Arial","sans-serif";
color:navy'>75, de Mortagne, Boucherville, Québec, Canada, J4B 6Y4<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:"Arial","sans-serif";
color:navy'>Gouvernement du Canada | Government of Canada</span><span
style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D'><o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
</div>
</body>
</html>