<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I am seeing the following error, multiple times on clients
trying to talk to a particular OST. The errors are inter<st1:PersonName
w:st="on">mit</st1:PersonName>tent: I get five to ten every few seconds,
then none for several hours (or even several days), then five to ten again.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>The servers are running Lustre 1.6.6, and are configured for
IB and tcp (over eth0). There are 20 IB clients, and 200 tcp/eth0
clients. The system was relatively quiet while these errors were occurring.
<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>modprobe.conf contains:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>options ib_mthca msi_x=1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>options lnet networks="tcp0(eth0),o2ib(ib0)"<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>options ko2iblnd ipif_name=ib0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Any idea what causes this, and how to resolve it?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Thanks.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:26:39 ts-nrel-01 kernel: LustreError: 11-0: an
error occurred while communicating with 172.16.103.26@tcp. The obd_ping
operation failed with -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:26:39 ts-nrel-01 kernel: Lustre:
lstr-ter-OST0003-osc: Connection to service lstr-ter-OST0003 via nid
172.16.103.26@tcp was lost; in progress operations using this service will wait
for recovery to complete.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:26:39 ts-nrel-01 kernel: LustreError: 167-0: This
client was evicted by lstr-ter-OST0003; in progress operations using this service
will fail.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:26:39 ts-nrel-01 kernel: Lustre:
lstr-ter-OST0003-osc: Connection restored to service lstr-ter-OST0003 using nid
172.16.103.26@tcp.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:26:39 ts-nrel-01 kernel: Lustre: MDS
lstr-ter-MDT0000: lstr-ter-OST0003_UUID now active, resetting orphans<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:30:49 ts-nrel-01 kernel: LustreError: 11-0: an
error occurred while communicating with 172.16.103.26@tcp. The obd_ping
operation failed with -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:30:49 ts-nrel-01 kernel: Lustre:
lstr-ter-OST0003-osc: Connection to service lstr-ter-OST0003 via nid
172.16.103.26@tcp was lost; in progress operations using this service will wait
for recovery to complete.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:30:49 ts-nrel-01 kernel: LustreError: 167-0: This
client was evicted by lstr-ter-OST0003; in progress operations using this
service will fail.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:30:49 ts-nrel-01 kernel: Lustre:
lstr-ter-OST0003-osc: Connection restored to service lstr-ter-OST0003 using nid
172.16.103.26@tcp.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:30:49 ts-nrel-01 kernel: Lustre: MDS
lstr-ter-MDT0000: lstr-ter-OST0003_UUID now active, resetting orphans<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:34:59 ts-nrel-01 kernel: LustreError: 11-0: an
error occurred while communicating with 172.16.103.26@tcp. The obd_ping
operation failed with -107<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:34:59 ts-nrel-01 kernel: Lustre:
lstr-ter-OST0003-osc: Connection to service lstr-ter-OST0003 via nid
172.16.103.26@tcp was lost; in progress operations using this service will wait
for recovery to complete.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:34:59 ts-nrel-01 kernel: LustreError: 167-0: This
client was evicted by lstr-ter-OST0003; in progress operations using this
service will fail.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:34:59 ts-nrel-01 kernel: Lustre:
lstr-ter-OST0003-osc: Connection restored to service lstr-ter-OST0003 using nid
172.16.103.26@tcp.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Mar 18 10:34:59 ts-nrel-01 kernel: Lustre: MDS
lstr-ter-MDT0000: lstr-ter-OST0003_UUID now active, resetting orphans<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Roger Spellman<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Staff Engineer<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Terascala, Inc.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>508-588-1501<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>www.terascala.com <http://www.terascala.com/><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
</div>
</body>
</html>