<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40"
xmlns:ns0="urn:schemas-microsoft-com:office:smarttags">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
@font-face
{font-family:"\@MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>In short, I am working to failover the MDT to another node.
I have activated heartbeat and it appears to be running properly. However, even
if other resources failover, the Lustre filesystem does not appear to. The
mount point on lustre01 (head mdt), does not transfer to lustre02 (slave mdt)
given a failure or a simple ‘/usr/lib/heartbeat/hb_takeover
foreign’ from the backup mdt server.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I am working with 2 nodes, both of which can see the same
device, /dev/sdc1. I ensured that the device could be mounted by either server.
The storage is Fibre Channel, if anybody is curious. Heartbeat was configured
and set up as below<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>/etc/ha.d/authkeys was set up (simple and the same on both
servers). In /usr/lib/ocf/resource.d/heartbeat/Filesystem, I included the
lustre filesystem as follows:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>if [ $blockdevice = "yes" ]; then<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
if [ "$DEVICE" != "/dev/null" -a ! -b
"$DEVICE" ] ; then<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
ocf_log err "Couldn't find device [$DEVICE]. Expected /dev/??? to
exist"<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
exit $OCF_ERR_ARGS<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
fi<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
if<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
case $FSTYPE in<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
ext3|reiserfs|reiser4|lustre|nss|xfs|jfs|vfat|fat|nfs|cifs|smbfs|ocfs2)
false;;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
*)
true;;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
esac<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
then<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
ocf_log info "Starting filesystem check on $DEVICE"<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
if [ -z "$FSTYPE" ]; then<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>
$FSCK -a $DEVICE<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>---etc<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>(this was the same on both servers)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Nothing was changed in /etc/ha.d/resource.d/Filesystem, as
/usr/lib/ocf/resource.d/heartbeat/Filesystem was used instead.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>/etc/ha.d/haresoureces contains the names and filesystems of
the two servers. lustre01 is the primary mds server and lustre02 is the backup<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>(same on both servers)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>lustre01 192.168.100.1
Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>lustre02 192.168.100.2
Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>The /etc/ha.d/ha.cf file on both servers is:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>debugfile
/var/log/ha-debug<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>logfile
/var/log/ha-log<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>logfacility local0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>keepalive 2<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>deadtime 15<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>initdead 60<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>udpport 694<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>bcast
eth1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>auto_failback off<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>node lustre01<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>node lustre02<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I have tried various orderings of starting heartbeat, but
generally, I first format the lustre01 node using ‘mkfs.lustre --mdt
--mgs --fsname mylustre --failnode=lustre02@tcp --reformat /dev/sdc1 ‘.
This works fine. Following this step, I mount the primary node (as shown on
p.76 of the Lustre 1.6 manual), ‘mount –t lustre /dev/sdc1
/lustremds’. /lustremds exists on both nodes. After this the
‘service heartbeat start’ command is issued on both nodes. The
results are as follows:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<div style='mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;
padding:0in 0in 1.0pt 0in'>
<p class=MsoNormal style='border:none;padding:0in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>Lustre01 (primary mdt)<o:p></o:p></span></font></p>
</div>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3727]: 2008/02/20_16:30:34 info:
**************************<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3727]: 2008/02/20_16:30:34 info: Configuration
validated. Starting heartbeat 2.1.2<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:34 info: heartbeat:
version 2.1.2<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:34 info: Heartbeat
generation: 1200690464<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:34 info: G_main_add_TriggerHandler:
Added signal manual handler<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:34 info:
G_main_add_TriggerHandler: Added signal manual handler<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:34 info: Removing
/var/run/heartbeat/rsctmp failed, recreating.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:34 info: glib: UDP
Broadcast heartbeat started on port 694 (694) interface eth1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:34 info: glib: UDP
Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:34 info:
G_main_add_SignalHandler: Added signal handler for signal 17<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:34 info: Local status now
set to: 'up'<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:35 info: Link
lustre01:eth1 up.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:40 info: Link
lustre02:eth1 up.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:40 info: Status update for
node lustre02: status up<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>harc[3735]: 2008/02/20_16:30:40
info: Running /etc/ha.d/rc.d/status status<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:41 info: Comm_now_up():
updating status to active<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:41 info: Local status now
set to: 'active'<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:41 WARN:
G_CH_dispatch_int: Dispatch function for read child took too long to execute:
210 ms (> 50 ms) (GSource: 0x8432df8)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:41 info: Status update for
node lustre02: status active<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>harc[3752]: 2008/02/20_16:30:41
info: Running /etc/ha.d/rc.d/status status<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:52 info: remote resource
transition completed.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:52 info: remote resource
transition completed.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:30:52 info: Initial resource
acquisition complete (T_RESOURCES(us))<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[3804]: 2008/02/20_16:30:52 INFO:
Resource is stopped<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3768]: 2008/02/20_16:30:52 info: Local Resource
acquisition completed.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>harc[3843]: 2008/02/20_16:30:52
info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ip-request-resp[3843]: 2008/02/20_16:30:52 received
ip-request-resp 192.168.100.1 OK yes<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[3864]: 2008/02/20_16:30:52 info:
Acquiring resource group: lustre01 192.168.100.1
Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[3891]: 2008/02/20_16:30:52 INFO:
Resource is stopped<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[3864]: 2008/02/20_16:30:52 info:
Running /etc/ha.d/resource.d/IPaddr 192.168.100.1 start<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[3967]: 2008/02/20_16:30:52 INFO: Using
calculated nic for 192.168.100.1: eth1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[3967]: 2008/02/20_16:30:52 INFO: Using
calculated netmask for 192.168.100.1: 255.255.255.0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[3967]: 2008/02/20_16:30:52 INFO: eval
ifconfig eth1:0 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[3967]: 2008/02/20_16:30:52 ERROR: Could
not add 192.168.100.1 to eth1: 255<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[3950]: 2008/02/20_16:30:52 ERROR:
Unknown error: 255<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[3864]: 2008/02/20_16:30:52 ERROR:
Return code 1 from /etc/ha.d/resource.d/IPaddr<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[3864]: 2008/02/20_16:30:52 CRIT:
Giving up resources due to failure of 192.168.100.1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[3864]: 2008/02/20_16:30:52 info:
Releasing resource group: lustre01 192.168.100.1
Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[3864]: 2008/02/20_16:30:52 info:
Running /etc/ha.d/resource.d/Filesystem /dev/sdc /lustremds lustre stop<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Filesystem[4103]:
2008/02/20_16:30:52 INFO: Running stop for /dev/sdc on /lustremds<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Filesystem[4092]:
2008/02/20_16:30:52 INFO: Success<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[3864]: 2008/02/20_16:30:52 info:
Running /etc/ha.d/resource.d/IPaddr 192.168.100.1 stop<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4176]: 2008/02/20_16:30:52 INFO:
Success<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:31:22 info: lustre02 wants to
go standby [foreign]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>hb_standby[4227]:
2008/02/20_16:31:23 Going standby [foreign].<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:31:23 WARN: Standby in
progress- new request from lustre01 ignored [3600 seconds left]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:31:23 info: standby: acquire
[foreign] resources from lustre02<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[4241]: 2008/02/20_16:31:23 info: acquire local HA
resources (standby).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4254]: 2008/02/20_16:31:23 info:
Acquiring resource group: lustre01 192.168.100.1
Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4281]: 2008/02/20_16:31:23 INFO:
Resource is stopped<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4254]: 2008/02/20_16:31:23 info:
Running /etc/ha.d/resource.d/IPaddr 192.168.100.1 start<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4357]: 2008/02/20_16:31:23 INFO: Using
calculated nic for 192.168.100.1: eth1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4357]: 2008/02/20_16:31:23 INFO: Using
calculated netmask for 192.168.100.1: 255.255.255.0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4357]: 2008/02/20_16:31:23 INFO: eval
ifconfig eth1:1 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4357]: 2008/02/20_16:31:23 ERROR: Could
not add 192.168.100.1 to eth1: 255<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4340]: 2008/02/20_16:31:23 ERROR:
Unknown error: 255<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4254]: 2008/02/20_16:31:24 ERROR:
Return code 1 from /etc/ha.d/resource.d/IPaddr<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4254]: 2008/02/20_16:31:24 CRIT:
Giving up resources due to failure of 192.168.100.1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4254]: 2008/02/20_16:31:24 info:
Releasing resource group: lustre01 192.168.100.1 Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4254]: 2008/02/20_16:31:24 info:
Running /etc/ha.d/resource.d/Filesystem /dev/sdc /lustremds lustre stop<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Filesystem[4491]:
2008/02/20_16:31:24 INFO: Running stop for /dev/sdc on /lustremds<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Filesystem[4480]: 2008/02/20_16:31:24
INFO: Success<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4254]: 2008/02/20_16:31:24 info:
Running /etc/ha.d/resource.d/IPaddr 192.168.100.1 stop<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4564]: 2008/02/20_16:31:24 INFO:
Success<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[4241]: 2008/02/20_16:31:24 info: local HA resource
acquisition completed (standby).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:31:24 info: Standby resource
acquisition done [foreign].<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:31:24 info: remote resource
transition completed.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>hb_standby[4621]:
2008/02/20_16:31:54 Going standby [foreign].<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:31:54 info: lustre01 wants to
go standby [foreign]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:31:54 info: standby: lustre02
can take our foreign resources<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[4635]: 2008/02/20_16:31:54 info: give up foreign
HA resources (standby).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4648]: 2008/02/20_16:31:55 info:
Releasing resource group: lustre02 192.168.100.2
Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4648]: 2008/02/20_16:31:55 info:
Running /etc/ha.d/resource.d/Filesystem /dev/sdc /lustremds lustre stop<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Filesystem[4697]:
2008/02/20_16:31:55 INFO: Running stop for /dev/sdc on /lustremds<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Filesystem[4686]:
2008/02/20_16:31:55 INFO: Success<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4648]: 2008/02/20_16:31:55 info:
Running /etc/ha.d/resource.d/IPaddr 192.168.100.2 stop<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4770]: 2008/02/20_16:31:55 INFO:
Success<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[4635]: 2008/02/20_16:31:55 info: foreign HA
resource release completed (standby).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:31:55 info: Local standby
process completed [foreign].<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:31:56 WARN: 1 lost packet(s)
for [lustre02] [58:60]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:31:56 info: remote resource
transition completed.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:31:56 info: No pkts missing
from lustre02!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:31:56 info: Other node
completed standby takeover of foreign resources.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:32:26 info: lustre02 wants to
go standby [foreign]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:32:27 info: standby: acquire
[foreign] resources from lustre02<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[4811]: 2008/02/20_16:32:27 info: acquire local HA
resources (standby).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4824]: 2008/02/20_16:32:27 info:
Acquiring resource group: lustre01 192.168.100.1
Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4851]: 2008/02/20_16:32:27 INFO:
Resource is stopped<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4824]: 2008/02/20_16:32:27 info:
Running /etc/ha.d/resource.d/IPaddr 192.168.100.1 start<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4927]: 2008/02/20_16:32:27 INFO: Using
calculated nic for 192.168.100.1: eth1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4927]: 2008/02/20_16:32:27 INFO: Using
calculated netmask for 192.168.100.1: 255.255.255.0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4927]: 2008/02/20_16:32:27 INFO: eval
ifconfig eth1:2 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4927]: 2008/02/20_16:32:27 ERROR: Could
not add 192.168.100.1 to eth1: 255<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4910]: 2008/02/20_16:32:27 ERROR:
Unknown error: 255<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4824]: 2008/02/20_16:32:27 ERROR:
Return code 1 from /etc/ha.d/resource.d/IPaddr<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4824]: 2008/02/20_16:32:27 CRIT:
Giving up resources due to failure of 192.168.100.1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4824]: 2008/02/20_16:32:27 info:
Releasing resource group: lustre01 192.168.100.1
Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4824]: 2008/02/20_16:32:27 info:
Running /etc/ha.d/resource.d/Filesystem /dev/sdc /lustremds lustre stop<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Filesystem[5061]:
2008/02/20_16:32:27 INFO: Running stop for /dev/sdc on /lustremds<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Filesystem[5050]:
2008/02/20_16:32:27 INFO: Success<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[4824]: 2008/02/20_16:32:27 info:
Running /etc/ha.d/resource.d/IPaddr 192.168.100.1 stop<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[5134]: 2008/02/20_16:32:27 INFO:
Success<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[4811]: 2008/02/20_16:32:27 info: local HA resource
acquisition completed (standby).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:32:27 info: Standby resource
acquisition done [foreign].<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[3728]: 2008/02/20_16:32:28 info: remote resource
transition completed.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<div style='mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;
padding:0in 0in 1.0pt 0in'>
<p class=MsoNormal style='border:none;padding:0in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>Lustre02 (secondary mdt)<o:p></o:p></span></font></p>
</div>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[4833]: 2008/02/20_16:39:24 info: lustre01 wants to
go standby [foreign]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[4833]: 2008/02/20_16:39:25 info: standby: acquire
[foreign] resources from lustre01<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>heartbeat[6658]: 2008/02/20_16:39:25 info: acquire local HA
resources (standby).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>ResourceManager[6671]: 2008/02/20_16:39:25 info:
Acquiring resource group: lustre02 192.168.100.2
Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>However, I cannot see Lustre mounted on either device. Does
anybody know what is the issue here? This statement concerns me:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4357]: 2008/02/20_16:31:23 ERROR: Could
not add 192.168.100.1 to eth1: 255<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>IPaddr[4340]: 2008/02/20_16:31:23 ERROR:
Unknown error: 255<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>BTW, 192.168.100.1 is the eth1 address on lustre01 (main
mdt)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Thanks<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
</div>
</body>
</html>