<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Apple Color Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Arial",sans-serif;
color:#000040;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US">There are, to my knowledge, a couple of open bugs related to the “lctl replace_nids” command that you should review prior to committing
to a change:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US"><a href="https://jira.hpdd.intel.com/browse/LU-8948">https://jira.hpdd.intel.com/browse/LU-8948</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US"><a href="https://jira.hpdd.intel.com/browse/LU-10384">https://jira.hpdd.intel.com/browse/LU-10384</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US">Some time ago, I wrote a d[r]aft guide on how to manage relatively complex LNet server configs, including the long-hand method for changing
server NIDs. I thought this had made it onto the community wiki but I appear to be mistaken. I don’t have time to make a mediawiki version, but I’ve uploaded a PDF version here:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US"><a href="http://wiki.lustre.org/File:Defining_Multiple_LNet_Interfaces_for_Multi-homed_Servers,_v1.pdf">http://wiki.lustre.org/File:Defining_Multiple_LNet_Interfaces_for_Multi-homed_Servers,_v1.pdf</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US">YMMV, there’s no warranty, whether express or implied, and I assume no liability, etc.
</span><span style="font-size:11.0pt;font-family:"Apple Color Emoji";color:#000040;mso-fareast-language:EN-US">☺</span><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US">Nevertheless, I hope this helps, at least as a cross-reference.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US">Malcolm.</span><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of "Vicker, Darby (JSC-EG311)" <darby.vicker-1@nasa.gov><br>
<b>Date: </b>Saturday, 6 January 2018 at 11:11 am<br>
<b>To: </b>Lustre discussion <lustre-discuss@lists.lustre.org><br>
<b>Cc: </b>"Kirk, Benjamin (JSC-EG311)" <benjamin.kirk@nasa.gov><br>
<b>Subject: </b>Re: [lustre-discuss] Adding a new NID<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">Sorry – one other question. We are configured for failover too. Will the "lctl replace_nids" do the right thing or should I do the tunefs to make sure all the failover pairs get
updated properly? This is what our tunefs command would look like for an OST:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> tunefs.lustre \</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> --dry-run \</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> --verbose \</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> --writeconf \</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> --erase-param \</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> --mgsnode=192.52.98.30@tcp0,10.148.0.30@o2ib0,10.150.100.30@o2ib1 \</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> --mgsnode=192.52.98.31@tcp0,10.148.0.31@o2ib0,10.150.100.31@o2ib1 \</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> --servicenode=${LUSTRE_LOCAL_TCP_IP}@tcp0,${LUSTRE_LOCAL_IB_L1_IP}@o2ib0,${LUSTRE_LOCAL_IB_EUROPA_IP}@o2ib1 \</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> --servicenode=${LUSTRE_PEER_TCP_IP}@tcp0,${LUSTRE_PEER_IB_L1_IP}@o2ib0,${LUSTRE_PEER_IB_EUROPA_IP}@o2ib1 \</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> $pool/ost-fsl</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">Our original mkfs.lustre options looked about like that, sans the o2ib1 NIDs. I'm worried that the "lctl repalce_nids" command won't know how to update the mgsnode and servicenode
properly. Is replace_nids smart enough for this? </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Darby Vicker <darby.vicker-1@nasa.gov><br>
<b>Date: </b>Friday, January 5, 2018 at 5:16 PM<br>
<b>To: </b>Lustre discussion <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[non-nasa source] [lustre-discuss] Adding a new NID</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">Hello everyone,</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">We have an existing LFS that is dual-homed on ethernet (mainly for our workstations) and IB (for the computational cluster), ZFS backend for the MDT and OST's. We just got a new
computational cluster and need to add another IB NID. The procedure for doing this is straight forward (14.5 in the admin manual) and amounts to:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">Unmount the clients</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">Unmount the MDT</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">Unmount all OSTs</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">mount -t lustre MDT partition -o nosvc mount_point</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">lctl replace_nids devicename nid1[,nid2,nid3 ...]</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">We haven't had to update a NID in a while so I was happy to see you can do this with "lctl replace_nids" instead of "tunsfs.lustre --writeconf". </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">I know this is dangerous, but we will sometime make minor changes to the servers by unmounting lustre on the servers (but leaving the clients up), make the changes, then remount
the servers. If we are confident we can do this quickly, the clients recover just fine. </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">While this isn't such a minor change, I'm a little tempted to do that in this case since nothing will really change for the existing clients – they don't need the new NID. Am I
asking for trouble here or do you think I can get away with this? I'm not too concerned about the possibility of it taking too long and getting the existing clients evicted. I'm (obviously) more concerned about doing something that would lead to corrupting
the FS. I should probably schedule an outage and do this right but... :)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">Darby</span><o:p></o:p></p>
</div>
</body>
</html>