<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
pre
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1><pre><font size=2 face="Courier New"><span
style='font-size:10.0pt'>I have 2 MDS, configured as an active/standby pair. I have 5 OSTs that are NOT active/standby. I<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>have 5 clients.<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>I am using Lustre 1.6.5, due to <a
href="https://bugzilla.lustre.org/show_bug.cgi?id=18232"
title="NEW - I/O Errors on Failover">bug 18232</a> which only affects 1.6.6. Using Lustre 1.6.5, when I<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>reset my active node, the standby takes over. This is quite reliable.<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>Today, I did the following in this order:<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'> Unmounted all the clients<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'> Rebooted all the clients<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'> Stopped Linux HA from running<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'> Unmounted the OSTs<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'> Unmounted the MDS<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'> Rebooted the OSTs<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'> Rebooted both MDSes<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>When the MDSes started up, Linux HA chose one to be active. That system mounted the MDT.<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>I looked at the file /proc/fs/lustre/mds/tacc-MDT0000/recovery_status, and it showed:<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>[root@ts-tacc-01 ~]# cat /proc/fs/lustre/mds/tacc-MDT0000/recovery_status <o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>status: RECOVERING<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>recovery_start: 0<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>time_remaining: 0<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>connected_clients: 0/5<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>completed_clients: 0/5<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>replayed_requests: 0/??<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>queued_requests: 0<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>next_transno: 17768<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>***** Note that recovery_start and time_remaining are both zero. *****<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>I waited a several minutes, and this file was the same.<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>I was waiting for recovery to complete before trying to mount the OSTs. However, it appears that<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>this would never occur!<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>Does this look like a bug? <o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>---------------------------<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>I format my MDT using the following command. The command is run from 10.2.43.1, and the failnode<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>is 10.2.43.2:<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>mkfs.lustre --reformat --fsname tacc --mdt --mgs --device-size=10000000 --mkfsoptions=' -m 0 -O<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>mmp' --failnode=10.2.43.2@o2ib0 /dev/sdb<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>I format the OSTs using the following command:<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>/usr/bin/time -p mkfs.lustre --reformat --ost --mkfsoptions='-J device=/dev/sdc1 -m 0' --fsname<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>tacc --device-size=400000000 --mgsnode=10.2.43.1@o2ib0 --mgsnode=10.2.43.2@o2ib0 /dev/sdb<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>I mount the clients using:<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span lang=FR style='font-size:10.0pt'>mount -t lustre 10.2.43.1@o2ib:10.2.43.2@o2ib:/tacc /mnt/lustre<o:p></o:p></span></font></pre>
<p class=MsoNormal><font size=2 face=Arial><span lang=FR style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
</div>
</body>
</html>