<HTML>
<HEAD>
<TITLE>Re: [Lustre-discuss] Failover for MGS</TITLE>
</HEAD>
<BODY>
<FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'>Since you are only adding parameters you don’t need the —erase-params option. I think.<BR>
<BR>
Robert<BR>
<BR>
<BR>
On 11/12/07 3:23 PM, "Wojciech Turek" <wjt27@cam.ac.uk> wrote:<BR>
<BR>
</SPAN></FONT></FONT><BLOCKQUOTE><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'>Hi,<BR>
<BR>
Thanks for that. Actually I have a little more complex situation here. I have two sets of clients. First set is working in 10.142.10.0/24 network and the second set is working in 10.143.0.0/16 network.<BR>
Each server has two NIC's. <BR>
NIC1 = ETH0 10.143.0.0/16 and NIC2= ETH1 10.142.10.0/24 <BR>
lnet configures network in the following manner:<BR>
eth0 = <ip>@tcp0<BR>
eth1 = <ip>@tcp1<BR>
<BR>
I am going to change lustre configuration in order to introduce failover features.<BR>
MGS is cobined with with mdt01=/dev/dm-0<BR>
<BR>
</SPAN></FONT></FONT><FONT SIZE="6"><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:10pt'>on mds01<BR>
</SPAN></FONT></FONT><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'>tunefs.lustre --erase-params --writeconf --failnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-0<BR>
</SPAN></FONT></FONT><FONT SIZE="6"><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:10pt'>tunefs.lustre --erase-params --writeconf --failnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-1<BR>
</SPAN></FONT></FONT><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'><BR>
</SPAN></FONT></FONT><FONT SIZE="6"><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:10pt'>on oss1<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.8@tcp0,10.142.10.8@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-0<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.8@tcp0,10.142.10.8@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-1<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.8@tcp0,10.142.10.8@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-2<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.8@tcp0,10.142.10.8@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-3<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.8@tcp0,10.142.10.8@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-4<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.8@tcp0,10.142.10.8@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-5<BR>
</SPAN></FONT></FONT><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'><BR>
</SPAN></FONT></FONT><FONT SIZE="6"><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:10pt'>on oss2<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.7@tcp0,10.142.10.7@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-6<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.7@tcp0,10.142.10.7@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-7<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.7@tcp0,10.142.10.7@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-8<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.7@tcp0,10.142.10.7@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-9<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.7@tcp0,10.142.10.7@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-10<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.7@tcp0,10.142.10.7@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-11<BR>
</SPAN></FONT></FONT><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'><BR>
</SPAN></FONT></FONT><FONT SIZE="6"><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:10pt'>on oss3<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.10@tcp0,10.142.10.10@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-0<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.10@tcp0,10.142.10.10@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-1<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.10@tcp0,10.142.10.10@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-2<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.10@tcp0,10.142.10.10@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-3<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.10@tcp0,10.142.10.10@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-4<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.10@tcp0,10.142.10.10@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-5<BR>
</SPAN></FONT></FONT><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'><BR>
</SPAN></FONT></FONT><FONT SIZE="6"><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:10pt'>on oss4<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.9@tcp0,10.142.10.9@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-6<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.9@tcp0,10.142.10.9@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-7<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.9@tcp0,10.142.10.9@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-8<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.9@tcp0,10.142.10.9@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-9<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.9@tcp0,10.142.10.9@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-10<BR>
tunefs.lustre --erase-params --writeconf --failnode=10.143.245.9@tcp0,10.142.10.9@tcp1 --mgsnode=10.143.245.201@tcp0,10.142.10.201@tcp1 --mgsnode=10.143.245.202@tcp0,10.142.10.202@tcp1 /dev/dm-11<BR>
</SPAN></FONT></FONT><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'> <BR>
Will above be correct?<BR>
<BR>
Cheers,<BR>
<BR>
Wojciech Turek<BR>
<BR>
<BR>
On 12 Nov 2007, at 21:36, Robert LeBlanc wrote:<BR>
<BR>
</SPAN></FONT></FONT><BLOCKQUOTE><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'> You should just unmount all the clients, all OSTs and then:<BR>
<BR>
tunefs.lustre —failnode 10.0.0.2@tcp —writeconf /dev/shared/disk<BR>
<BR>
If your volume is already on the shared disk, them mount everything and you should be good to go. You can also do it on a live mounted system by using lctl, but I’m not exactly sure how to do that.<BR>
<BR>
Robert<BR>
<BR>
On 11/12/07 2:24 PM, "Wojciech Turek" <wjt27@cam.ac.uk> wrote:<BR>
<BR>
<BR>
</SPAN></FONT></FONT><BLOCKQUOTE><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'>Hi,<BR>
<BR>
How will look my tunefs.lustre command line if I would like to configure failnode for my MDS. I have two MDT's and MGS is on the same block device that one of MDT's ? I have also two servers connected to share matadata storage.<BR>
<BR>
Thanks,<BR>
<BR>
Wojciech <BR>
On 12 Nov 2007, at 20:49, Nathan Rutman wrote:<BR>
<BR>
<BR>
</SPAN></FONT></FONT><BLOCKQUOTE><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'>Robert LeBlanc wrote:<BR>
<BR>
<BR>
</SPAN></FONT></FONT><BLOCKQUOTE><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'>Ok, I feel really stupid. I've done this before without any problem, but I<BR>
can't seem to get it to work and I can't find my notes from the last time I<BR>
did it. We have separate MGS and MDTs. I can't seem to get our MGS to<BR>
failover correctly after reformatting it.<BR>
<BR>
mkfs.lustre --mkfsoptions="-O dir_index" --reformat --mgs<BR>
--failnode=192.168.1.253@o2ib /dev/mapper/ldiskc-part1<BR>
<BR>
<BR>
<BR>
<BR>
</SPAN></FONT></FONT></BLOCKQUOTE><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'>The MGS doesn't actually use the --failnode option (although it won't <BR>
hurt). You actually have to tell the other nodes<BR>
in the system (servers and clients) about the failover options for the <BR>
MGS (use the --mgsnode parameter on servers, and mount address for <BR>
clients). The reason is because the servers must contact the MGS for <BR>
the configuration information, and they can't ask the MGS where its <BR>
failover partner is if e.g. the failover partner is the one that's running.<BR>
<BR>
<BR>
<BR>
</SPAN></FONT></FONT><BLOCKQUOTE><FONT SIZE="5"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:9pt'>We are running this on Debian, using the Lustre 1.6.3 debs from svn on Lenny<BR>
with 2.6.22.12. I've tried several permutations of the mkfs.lustre command,<BR>
specifing both nodes as failover, and both nodes as MGS and pretty much<BR>
every other combination of the above. With the above command tunefs.lustre<BR>
shows that failnode and mgsnode are the failover node.<BR>
<BR>
Thanks,<BR>
Robert<BR>
<BR>
Robert LeBlanc<BR>
College of Life Sciences Computer Support<BR>
Brigham Young University<BR>
leblanc@byu.edu<BR>
(801)422-1882<BR>
<BR>
<BR>
_______________________________________________<BR>
Lustre-discuss mailing list<BR>
Lustre-discuss@clusterfs.com<BR>
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss<BR>
<BR>
<BR>
<BR>
_______________________________________________<BR>
Lustre-discuss mailing list<BR>
Lustre-discuss@clusterfs.com<BR>
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss<BR>
<BR>
<BR>
<BR>
</SPAN></FONT></FONT><FONT SIZE="6"><FONT FACE="Lucida Grande"><SPAN STYLE='font-size:10pt'>Mr Wojciech Turek<BR>
</SPAN></FONT></FONT><FONT SIZE="5"><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:9pt'>Assistant System Manager<BR>
</SPAN></FONT></FONT><FONT SIZE="6"><FONT FACE="Lucida Grande"><SPAN STYLE='font-size:10pt'>University of Cambridge<BR>
High Performance Computing service <BR>
</SPAN></FONT></FONT><FONT FACE="Lucida Grande"><FONT SIZE="4"><SPAN STYLE='font-size:8pt'>email: wjt27@cam.ac.uk<BR>
</SPAN></FONT><FONT SIZE="6"><SPAN STYLE='font-size:10pt'>tel. +441223763517<BR>
<BR>
</SPAN></FONT></FONT><FONT SIZE="5"><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:9pt'> <BR>
<BR>
<BR>
</SPAN></FONT><SPAN STYLE='font-size:9pt'><FONT FACE="Calibri, Verdana, Helvetica, Arial"> <BR>
<BR>
<BR>
<BR>
Robert LeBlanc<BR>
College of Life Sciences Computer Support<BR>
Brigham Young University<BR>
leblanc@byu.edu<BR>
(801)422-1882<BR>
<BR>
<BR>
<BR>
<BR>
</FONT></SPAN></FONT><FONT SIZE="6"><FONT FACE="Lucida Grande"><SPAN STYLE='font-size:10pt'>Mr Wojciech Turek<BR>
</SPAN></FONT></FONT><FONT SIZE="5"><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:9pt'>Assistant System Manager<BR>
</SPAN></FONT></FONT><FONT SIZE="6"><FONT FACE="Lucida Grande"><SPAN STYLE='font-size:10pt'>University of Cambridge<BR>
High Performance Computing service <BR>
</SPAN></FONT></FONT><FONT FACE="Lucida Grande"><FONT SIZE="5"><SPAN STYLE='font-size:8.5pt'>email: wjt27@cam.ac.uk<BR>
</SPAN></FONT><FONT SIZE="6"><SPAN STYLE='font-size:10pt'>tel. +441223763517<BR>
</SPAN></FONT></FONT><FONT SIZE="5"><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:9pt'><BR>
<BR>
<BR>
</SPAN></FONT><SPAN STYLE='font-size:9pt'><FONT FACE="Calibri, Verdana, Helvetica, Arial"><BR>
<BR>
</FONT></SPAN></FONT></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE><FONT SIZE="5"><SPAN STYLE='font-size:9pt'><FONT FACE="Calibri, Verdana, Helvetica, Arial"><BR>
<BR>
Robert LeBlanc<BR>
College of Life Sciences Computer Support<BR>
Brigham Young University<BR>
leblanc@byu.edu<BR>
(801)422-1882<BR>
<BR>
</FONT></SPAN></FONT>
</BODY>
</HTML>