<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
Hi,<div><br class="webkit-block-placeholder"></div><div>Thanks for that. Actually I have a little more complex situation here. I have two sets of clients. First set is working in 10.142.10.0/24 network and the second set is working in 10.143.0.0/16 network.</div><div>Each server has two NIC's. </div><div>NIC1 = ETH0 10.143.0.0/16 and NIC2= ETH1 10.142.10.0/24 </div><div>lnet configures network in the following manner:</div><div>eth0 = <ip>@tcp0</div><div>eth1 = <ip>@tcp1</div><div><br class="webkit-block-placeholder"></div><div>I am going to change lustre configuration in order to introduce failover features.</div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">MGS is cobined with with mdt01=/dev/dm-0</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br class="webkit-block-placeholder"></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">on mds01</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-0</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-1</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">on oss1</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.8@tcp0,10.142.10.8@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-0</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.8@tcp0,10.142.10.8@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-1</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.8@tcp0,10.142.10.8@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-2</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.8@tcp0,10.142.10.8@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-3</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.8@tcp0,10.142.10.8@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-4</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.8@tcp0,10.142.10.8@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-5</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">on oss2</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.7@tcp0,10.142.10.7@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-6</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.7@tcp0,10.142.10.7@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-7</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.7@tcp0,10.142.10.7@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-8</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.7@tcp0,10.142.10.7@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-9</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.7@tcp0,10.142.10.7@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-10</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.7@tcp0,10.142.10.7@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-11</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">on oss3</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.10@tcp0,10.142.10.10@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-0</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.10@tcp0,10.142.10.10@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-1</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.10@tcp0,10.142.10.10@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-2</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.10@tcp0,10.142.10.10@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-3</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.10@tcp0,10.142.10.10@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-4</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.10@tcp0,10.142.10.10@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-5</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">on oss4</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.9@tcp0,10.142.10.9@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-6</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.9@tcp0,10.142.10.9@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-7</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.9@tcp0,10.142.10.9@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-8</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.9@tcp0,10.142.10.9@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-9</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.9@tcp0,10.142.10.9@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-10</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica">tunefs.lustre --erase-params --writeconf --failnode=10.143.245.9@tcp0,10.142.10.9@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-11</font></div> <br class="webkit-block-placeholder"></div><div>Will above be correct?</div><div><br class="webkit-block-placeholder"></div><div>Cheers,</div><div><br class="webkit-block-placeholder"></div><div>Wojciech Turek</div><div><br class="webkit-block-placeholder"></div><div><br></div><div><div><div>On 12 Nov 2007, at 21:36, Robert LeBlanc wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"> <font size="5"><font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:9pt">You should just unmount all the clients, all OSTs and then:<br> <br> tunefs.lustre —failnode 10.0.0.2@tcp —writeconf /dev/shared/disk<br> <br> If your volume is already on the shared disk, them mount everything and you should be good to go. You can also do it on a live mounted system by using lctl, but I’m not exactly sure how to do that.<br> <br> Robert<br> <br> On 11/12/07 2:24 PM, "Wojciech Turek" <<a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a>> wrote:<br> <br> </span></font></font><blockquote type="cite"><font size="5"><font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:9pt">Hi,<br> <br> How will look my tunefs.lustre command line if I would like to configure failnode for my MDS. I have two MDT's and MGS is on the same block device that one of MDT's ? I have also two servers connected to share matadata storage.<br> <br> Thanks,<br> <br> Wojciech <br> On 12 Nov 2007, at 20:49, Nathan Rutman wrote:<br> <br> </span></font></font><blockquote type="cite"><font size="5"><font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:9pt">Robert LeBlanc wrote:<br> <br> </span></font></font><blockquote type="cite"><font size="5"><font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:9pt">Ok, I feel really stupid. I've done this before without any problem, but I<br> can't seem to get it to work and I can't find my notes from the last time I<br> did it. We have separate MGS and MDTs. I can't seem to get our MGS to<br> failover correctly after reformatting it.<br> <br> mkfs.lustre --mkfsoptions="-O dir_index" --reformat --mgs<br> --failnode=192.168.1.253@o2ib /dev/mapper/ldiskc-part1<br> <br> <br> <br> </span></font></font></blockquote><font size="5"><font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:9pt">The MGS doesn't actually use the --failnode option (although it won't <br> hurt). You actually have to tell the other nodes<br> in the system (servers and clients) about the failover options for the <br> MGS (use the --mgsnode parameter on servers, and mount address for <br> clients). The reason is because the servers must contact the MGS for <br> the configuration information, and they can't ask the MGS where its <br> failover partner is if e.g. the failover partner is the one that's running.<br> <br> <br> </span></font></font><blockquote type="cite"><font size="5"><font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:9pt">We are running this on Debian, using the Lustre 1.6.3 debs from svn on Lenny<br> with 2.6.22.12. I've tried several permutations of the mkfs.lustre command,<br> specifing both nodes as failover, and both nodes as MGS and pretty much<br> every other combination of the above. With the above command tunefs.lustre<br> shows that failnode and mgsnode are the failover node.<br> <br> Thanks,<br> Robert<br> <br> Robert LeBlanc<br> College of Life Sciences Computer Support<br> Brigham Young University<br> <a href="mailto:leblanc@byu.edu">leblanc@byu.edu</a><br> (801)422-1882<br> <br> <br> _______________________________________________<br> Lustre-discuss mailing list<br> <a href="mailto:Lustre-discuss@clusterfs.com">Lustre-discuss@clusterfs.com</a><br> <a href="https://mail.clusterfs.com/mailman/listinfo/lustre-discuss">https://mail.clusterfs.com/mailman/listinfo/lustre-discuss</a><br> <br> <br> <br> _______________________________________________<br> Lustre-discuss mailing list<br> <a href="mailto:Lustre-discuss@clusterfs.com">Lustre-discuss@clusterfs.com</a><br> <a href="https://mail.clusterfs.com/mailman/listinfo/lustre-discuss">https://mail.clusterfs.com/mailman/listinfo/lustre-discuss</a><br> <br> <br> <br> </span></font></font><font size="6"><font face="Lucida Grande"><span style="font-size:10pt">Mr Wojciech Turek<br> </span></font></font><font size="5"><font face="Helvetica, Verdana, Arial"><span style="font-size:9pt">Assistant System Manager<br> </span></font></font><font size="6"><font face="Lucida Grande"><span style="font-size:10pt">University of Cambridge<br> High Performance Computing service <br> </span></font></font><font face="Lucida Grande"><font size="5"><span style="font-size:8.5pt">email: <a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a><br> </span></font><font size="6"><span style="font-size:10pt">tel. +441223763517<br> </span></font></font><font size="5"><font face="Helvetica, Verdana, Arial"><span style="font-size:9pt"><br> <br> <br> </span></font><span style="font-size:9pt"><font face="Calibri, Verdana, Helvetica, Arial"><br> <br> </font></span></font></blockquote></blockquote></blockquote><font size="5"><span style="font-size:9pt"><font face="Calibri, Verdana, Helvetica, Arial"><br> <br> Robert LeBlanc<br> College of Life Sciences Computer Support<br> Brigham Young University<br> <a href="mailto:leblanc@byu.edu">leblanc@byu.edu</a><br> (801)422-1882<br> <br> </font></span></font> </blockquote></div><br><div> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 11.0px Lucida Grande">Mr Wojciech Turek</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Lucida Grande; min-height: 13px; ">Assistant System Manager</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 11.0px Lucida Grande">University of Cambridge</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 11.0px Lucida Grande">High Performance Computing service </font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="'Lucida Grande'" size="3"><span class="Apple-style-span" style="font-size: 11px;">email: <a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a></span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 11.0px Lucida Grande">tel. +441223763517</font></div></div><div><br class="webkit-block-placeholder"></div></span><br class="Apple-interchange-newline"></span></span> </div><br></div></body></html>