<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">You will have to recompile lustre with the patch in LU-8397. The key for us was to look at the contents of /proc/fs/lustre/mgc/*/import. Before the patch, failover_nids from
that file was only showing one NID, despite mkfs.lustre/tunefs.lustre showing multiple service nodes configured. See the mailing list thread and the LU for more details.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Looking back at this, our problems were related to multirail (using both IB and TCP). Based on the mkfs.lustre commands you sent in your original email, that probably isn’t
your issue. Just for reference, this is what the mkfs.lustre command looks like for us.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> mkfs.lustre \<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> --mgsnode=192.52.98.30@tcp0,10.148.0.30@o2ib0 \<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> --mgsnode=192.52.98.31@tcp0,10.148.0.31@o2ib0 \<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> --fsname=testfs \<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> --backfstype=zfs \<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> --reformat \<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> --verbose \<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> --mdt --index=0 \<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> --servicenode=${LUSTRE_LOCAL_TCP_IP}@tcp0,${LUSTRE_LOCAL_IB_IP}@o2ib0 \<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> --servicenode=${LUSTRE_PEER_TCP_IP}@tcp0,${LUSTRE_PEER_IB_IP}@o2ib0 \<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> metadata/meta-test<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Looking at this, you used a single --failover instead of a multiple --servicenode's. The admin manual indicates --servicenode is preferred. You might try that. I still think
looking at the import file I pointed you to above would be instructive regardless.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Ravi Konila <ravibhatk@gmail.com><br>
<b>Reply-To: </b>Ravi Konila <ravibhatk@gmail.com><br>
<b>Date: </b>Thursday, October 26, 2017 at 1:31 AM<br>
<b>To: </b>Darby Vicker <darby.vicker-1@nasa.gov>, "Mannthey, Keith" <keith.mannthey@intel.com>, Lustre Discuss <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] MGS is not working in HA<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Hi
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">I am using Lustre 2.8 on RHEL 6.7.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">As my application requires RHEL 6.7, I had to use Lustre 2.8.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Any suggestions?<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">Regards</span><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><strong><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">Ravi Konila</span></strong><span style="font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"> Vicker, Darby (JSC-EG311)
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Sent:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"> Wednesday, October 25, 2017 11:51 PM<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">To:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"> Mannthey, Keith ; Ravi Konila ; Lustre
Discuss <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:whitesmoke"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Subject:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"> Re: [lustre-discuss] MGS is not working
in HA<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Sorry – I also meant to say that the resolution went off the mailing list and was continued in LU-8397. You can find the patch there.
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Darby Vicker <darby.vicker-1@nasa.gov><br>
<b>Date: </b>Wednesday, October 25, 2017 at 1:17 PM<br>
<b>To: </b>"Mannthey, Keith" <keith.mannthey@intel.com>, Ravi Konila <ravibhatk@gmail.com>, Lustre Discuss <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] MGS is not working in HA<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Which version of lustre are you using? We initially has problem with this too when using failover with lustre 2.8 and 2.9. We got a patch that fixed it and recent
versions work fine for us. We have a combined MGS/MDS so our scenario is a little different but this sounds very similar to our issue.
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2017-January/014125.html">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2017-January/014125.html</a></span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of "Mannthey, Keith" <keith.mannthey@intel.com><br>
<b>Date: </b>Wednesday, October 25, 2017 at 11:30 AM<br>
<b>To: </b>Ravi Konila <ravibhatk@gmail.com>, Lustre Discuss <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] MGS is not working in HA<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Kavi,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> You may want to open a jira ticket with this error. It looks like the mount command is only trying only the first nid of the mount command.
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Jira is
<a href="https://jira.hpdd.intel.com">https://jira.hpdd.intel.com</a> “LU” project.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I have seen Lustre Servers first mount behave like this but not client mounts. It should try the first server, timeout and try the 2<sup>nd</sup> server.
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Keith
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span></a><span style="color:black"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> lustre-discuss [mailto:lustre-discuss-bounces@lists.lustre.org]
<b>On Behalf Of </b>Ravi Konila<br>
<b>Sent:</b> Wednesday, October 25, 2017 5:07 AM<br>
<b>To:</b> Lustre Discuss <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> [lustre-discuss] MGS is not working in HA</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Hi
</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">I have two servers for MGS/MDS and have configured it pacemaker for HA.
</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">The command which I gave on first MGS/MDS mds01 is</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">mkfs.lustre --mgs --failnode 192.168.0.51@o2ib --backfstype=ldiskfs /dev/mapper/mpathd</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Next I created lustre filesystem for MDT</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">mkfs.lustre --mdt --fsname lhome --index 0 --mgsnode 192.168.0.50@o2ib --mgsnode 192.168.0.51@o2ib --servicenode 192.168.0.50@o2ib --servicenode 192.168.0.51@o2ib --backfstype=ldiskfs
/dev/mapper/mpathb</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Now, in my client, If I give</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">mount –t lustre 192.168.0.50@o2ib:192.168.0.51@o2ib:/lhome /home, it does not work and asks if MGS is running.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">But if I give mount –t lustre 192.168.0.50@o2ib:/lhome /home it works fine.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Also when my first MDS (mds01) is down, my client is not mounting lustre from 2nd MGS.
</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">It says check if MGS is running?</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Any help will be highly appreciated.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">Regards</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><strong><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:black">Ravi Konila</span></strong><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>