<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mv="http://macVmlSchemaUri" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#365F91;
font-weight:normal;
font-style:normal;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">The pre-built rpms are most likely compiled against the in-kernel IB drivers. If you’re using the MOFED drivers you’ll need to recompile Lustre. The instructions here may help you out
<a href="http://wiki.lustre.org/Compiling_Lustre">http://wiki.lustre.org/Compiling_Lustre</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Chris Horn<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Ravi Konila <ravibhatk@gmail.com><br>
<b>Reply-To: </b>Ravi Konila <ravibhatk@gmail.com><br>
<b>Date: </b>Thursday, October 12, 2017 at 1:33 PM<br>
<b>To: </b>Chris Horn <hornc@cray.com>, Parag Khuraswar <parag_k@citilindia.com>, 'Lustre User Discussion Mailing List' <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] Lnet not starting<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hi<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">I am using pre-built rpms.<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">Regards</span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><strong><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">Ravi Konila</span></strong><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black">From:</span></b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> Chris Horn
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black">Sent:</span></b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> Thursday, October 12, 2017 10:51 PM<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black">To:</span></b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> Ravi Konila ; Parag Khuraswar ; 'Lustre User Discussion
Mailing List' <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black">Subject:</span></b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> Re: [lustre-discuss] Lnet not starting<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Are you compiling Lustre yourself or using pre-built rpms?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Chris Horn<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Ravi Konila <ravibhatk@gmail.com><br>
<b>Reply-To: </b>Ravi Konila <ravibhatk@gmail.com><br>
<b>Date: </b>Thursday, October 12, 2017 at 11:40 AM<br>
<b>To: </b>Chris Horn <hornc@cray.com>, Parag Khuraswar <parag_k@citilindia.com>, 'Lustre User Discussion Mailing List' <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] Lnet not starting</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hi Chris</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">I installed RHEL 6.7, MLNX_OFED_LINUX-3.4-1.0.0.0-rhel6.7-x86_64 and then Lustre 2.8 in my Lustre MDS/MGT/OSS servers.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">My ib0 is working fine and I can ping other nodes.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">my lustre.conf file has</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">options lnet networks=o2ib(ib0)</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">With this, If I run “service lnet start” it fails with error
</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">LNET configure error 22: Invalid argument</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">dmesg give me output as below (I just captured last line but there are many lines with symbol error or so)</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">LNetError: 16770:0:(api-ni.c:1276:lnet_startup_lndni()) Can't load LND o2ib, module ko2iblnd, rc=256</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">If I specify tcp in lustre.conf,, it works fine.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">I have reinstalled Lustre and then Mellanox OFED driver but still the problem is same, not able to make infiniband up with Lustre Lnet</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">Regards</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><strong><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">Ravi Konila</span></strong><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">Sr. Technical Consultant</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black">From:</span></b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> Chris Horn
</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black">Sent:</span></b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> Thursday, October 12, 2017 9:02 PM</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black">To:</span></b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> Ravi Konila ; Parag Khuraswar ; 'Lustre User Discussion
Mailing List' </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black">Subject:</span></b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> Re: [lustre-discuss] Lnet not starting</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="color:black">dmesg output should provide more information about the “Invalid argument” error that you are seeing, but my guess would be that Lustre was compiled against a different IB stack than what you have installed.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Chris Horn<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Ravi Konila <ravibhatk@gmail.com><br>
<b>Reply-To: </b>Ravi Konila <ravibhatk@gmail.com><br>
<b>Date: </b>Thursday, October 12, 2017 at 7:58 AM<br>
<b>To: </b>Parag Khuraswar <parag_k@citilindia.com>, 'Lustre User Discussion Mailing List' <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] Lnet not starting</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hi Parag</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Even I am facing the same issue with RHEL 6.7 and Lustre 2.8. Now I am also trying with RHEL 7.3 and Lustre 2.10.0.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">I am planning to install Mellanox OFED driver with 3.4 stack. Looks like there is some problem with OFED 4.x stack with Lustre 2.10.0.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Let me try the same and update.
</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">When I start “service lnet start” it gives LNET configure error 22: Invalid argument</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">but it works fine with tcp.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">Regards</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><strong><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">Ravi Konila</span></strong><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">Sr. Technical Consultant</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Maruti Suzuki India Ltd</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black">From:</span></b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> Parag Khuraswar
</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black">Sent:</span></b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> Thursday, October 12, 2017 6:11 PM</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black">To:</span></b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> 'Lustre User Discussion Mailing List'
</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black">Subject:</span></b><span style="font-size:10.0pt;font-family:"tahoma",serif;color:black"> [lustre-discuss] Lnet not starting</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#365F91">Hi,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#365F91">I am installing Lustre 2.10.0 on RHEL 7.3.
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#365F91">IB is working fine but lnet is not coming up. Lustre service is running.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#365F91">Ibstat also show link up and active.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#365F91">Lustre and lnet modules are also loaded.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#365F91">Regards,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:#365F91">Parag</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt;color:black">
<hr size="2" width="100%" align="center">
</span></div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>