<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>I've never used heartbeat before but just from reading what you wrote I see a couple things that could be wrong. The first is in <span class="Apple-style-span" style="font-family: Arial; font-size: 13px; ">/etc/ha.d/haresoureces (pasted lines below)</span></div><div><span class="Apple-style-span" style="font-family: Arial; font-size: 13px; "> </span></div><blockquote type="cite"><span class="Apple-style-span" style="color: rgb(0, 0, 0); "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">lustre01 192.168.100.1 Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">lustre02 192.168.100.2 Filesystem::/dev/sdc::/lustremds::lustre</span></font></div></span></blockquote><div><br class="webkit-block-placeholder"></div>If I understand this correctly, you're telling it to look for the filesystem on /dev/sdc...but your mkfs command created the filesystem on /dev/sdc1. I don't think thats' the main problem here, however. I believe the problem is that you have 192.168.100.1 assigned to eth1 already. It's trying to re-assign it to eth1:0 which will cause it to fail. Try giving another ip to eth1 <br><br><div><div>On Feb 20, 2008, at 5:53 PM, Chadha, Narjit wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div lang="EN-US" link="blue" vlink="purple"><div class="Section1"><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">In short, I am working to failover the MDT to another node. I have activated heartbeat and it appears to be running properly. However, even if other resources failover, the Lustre filesystem does not appear to. The mount point on lustre01 (head mdt), does not transfer to lustre02 (slave mdt) given a failure or a simple ‘/usr/lib/heartbeat/hb_takeover foreign’ from the backup mdt server.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">I am working with 2 nodes, both of which can see the same device, /dev/sdc1. I ensured that the device could be mounted by either server. The storage is Fibre Channel, if anybody is curious. Heartbeat was configured and set up as below<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">/etc/ha.d/authkeys was set up (simple and the same on both servers). In /usr/lib/ocf/resource.d/heartbeat/Filesystem, I included the lustre filesystem as follows:<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">if [ $blockdevice = "yes" ]; then<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> if [ "$DEVICE" != "/dev/null" -a ! -b "$DEVICE" ] ; then<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> ocf_log err "Couldn't find device [$DEVICE]. Expected /dev/??? to exist"<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> exit $OCF_ERR_ARGS<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> fi<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> if<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> case $FSTYPE in<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> ext3|reiserfs|reiser4|lustre|nss|xfs|jfs|vfat|fat|nfs|cifs|smbfs|ocfs2) false;;<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> *) true;;<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> esac<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> then<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> ocf_log info "Starting filesystem check on $DEVICE"<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> if [ -z "$FSTYPE" ]; then<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "> $FSCK -a $DEVICE<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">---etc<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">(this was the same on both servers)<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Nothing was changed in /etc/ha.d/resource.d/Filesystem, as /usr/lib/ocf/resource.d/heartbeat/Filesystem was used instead.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">/etc/ha.d/haresoureces contains the names and filesystems of the two servers. lustre01 is the primary mds server and lustre02 is the backup<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">(same on both servers)<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">lustre01 192.168.100.1 Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">lustre02 192.168.100.2 Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">The /etc/ha.d/ha.cf file on both servers is:<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">debugfile /var/log/ha-debug<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">logfile /var/log/ha-log<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">logfacility local0<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">keepalive 2<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">deadtime 15<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">initdead 60<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">udpport 694<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">bcast eth1<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">auto_failback off<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">node lustre01<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">node lustre02<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">I have tried various orderings of starting heartbeat, but generally, I first format the lustre01 node using ‘mkfs.lustre --mdt --mgs --fsname mylustre --failnode=lustre02@tcp --reformat /dev/sdc1 ‘. This works fine. Following this step, I mount the primary node (as shown on p.76 of the Lustre 1.6 manual), ‘mount –t lustre /dev/sdc1 /lustremds’. /lustremds exists on both nodes. After this the ‘service heartbeat start’ command is issued on both nodes. The results are as follows:<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="border-top-style: none; border-right-style: none; border-left-style: none; border-width: initial; border-color: initial; border-bottom-style: solid; border-bottom-color: windowtext; border-bottom-width: 1pt; padding-top: 0in; padding-right: 0in; padding-bottom: 1pt; padding-left: 0in; "><div style="border-top-style: none; border-right-style: none; border-bottom-style: none; border-left-style: none; border-width: initial; border-color: initial; padding-top: 0in; padding-right: 0in; padding-bottom: 0in; padding-left: 0in; margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Lustre01 (primary mdt)<o:p></o:p></span></font></div></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3727]: 2008/02/20_16:30:34 info: **************************<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3727]: 2008/02/20_16:30:34 info: Configuration validated. Starting heartbeat 2.1.2<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:34 info: heartbeat: version 2.1.2<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:34 info: Heartbeat generation: 1200690464<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:34 info: G_main_add_TriggerHandler: Added signal manual handler<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:34 info: G_main_add_TriggerHandler: Added signal manual handler<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:34 info: Removing /var/run/heartbeat/rsctmp failed, recreating.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:34 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:34 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:34 info: G_main_add_SignalHandler: Added signal handler for signal 17<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:34 info: Local status now set to: 'up'<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:35 info: Link lustre01:eth1 up.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:40 info: Link lustre02:eth1 up.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:40 info: Status update for node lustre02: status up<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">harc[3735]: 2008/02/20_16:30:40 info: Running /etc/ha.d/rc.d/status status<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:41 info: Comm_now_up(): updating status to active<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:41 info: Local status now set to: 'active'<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:41 WARN: G_CH_dispatch_int: Dispatch function for read child took too long to execute: 210 ms (> 50 ms) (GSource: 0x8432df8)<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:41 info: Status update for node lustre02: status active<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">harc[3752]: 2008/02/20_16:30:41 info: Running /etc/ha.d/rc.d/status status<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:52 info: remote resource transition completed.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:52 info: remote resource transition completed.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:30:52 info: Initial resource acquisition complete (T_RESOURCES(us))<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[3804]: 2008/02/20_16:30:52 INFO: Resource is stopped<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3768]: 2008/02/20_16:30:52 info: Local Resource acquisition completed.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">harc[3843]: 2008/02/20_16:30:52 info: Running /etc/ha.d/rc.d/ip-request-resp ip-request-resp<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ip-request-resp[3843]: 2008/02/20_16:30:52 received ip-request-resp 192.168.100.1 OK yes<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[3864]: 2008/02/20_16:30:52 info: Acquiring resource group: lustre01 192.168.100.1 Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[3891]: 2008/02/20_16:30:52 INFO: Resource is stopped<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[3864]: 2008/02/20_16:30:52 info: Running /etc/ha.d/resource.d/IPaddr 192.168.100.1 start<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[3967]: 2008/02/20_16:30:52 INFO: Using calculated nic for 192.168.100.1: eth1<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[3967]: 2008/02/20_16:30:52 INFO: Using calculated netmask for 192.168.100.1: 255.255.255.0<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[3967]: 2008/02/20_16:30:52 INFO: eval ifconfig eth1:0 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[3967]: 2008/02/20_16:30:52 ERROR: Could not add 192.168.100.1 to eth1: 255<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[3950]: 2008/02/20_16:30:52 ERROR: Unknown error: 255<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[3864]: 2008/02/20_16:30:52 ERROR: Return code 1 from /etc/ha.d/resource.d/IPaddr<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[3864]: 2008/02/20_16:30:52 CRIT: Giving up resources due to failure of 192.168.100.1<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[3864]: 2008/02/20_16:30:52 info: Releasing resource group: lustre01 192.168.100.1 Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[3864]: 2008/02/20_16:30:52 info: Running /etc/ha.d/resource.d/Filesystem /dev/sdc /lustremds lustre stop<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Filesystem[4103]: 2008/02/20_16:30:52 INFO: Running stop for /dev/sdc on /lustremds<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Filesystem[4092]: 2008/02/20_16:30:52 INFO: Success<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[3864]: 2008/02/20_16:30:52 info: Running /etc/ha.d/resource.d/IPaddr 192.168.100.1 stop<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4176]: 2008/02/20_16:30:52 INFO: Success<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:31:22 info: lustre02 wants to go standby [foreign]<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">hb_standby[4227]: 2008/02/20_16:31:23 Going standby [foreign].<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:31:23 WARN: Standby in progress- new request from lustre01 ignored [3600 seconds left]<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:31:23 info: standby: acquire [foreign] resources from lustre02<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[4241]: 2008/02/20_16:31:23 info: acquire local HA resources (standby).<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4254]: 2008/02/20_16:31:23 info: Acquiring resource group: lustre01 192.168.100.1 Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4281]: 2008/02/20_16:31:23 INFO: Resource is stopped<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4254]: 2008/02/20_16:31:23 info: Running /etc/ha.d/resource.d/IPaddr 192.168.100.1 start<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4357]: 2008/02/20_16:31:23 INFO: Using calculated nic for 192.168.100.1: eth1<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4357]: 2008/02/20_16:31:23 INFO: Using calculated netmask for 192.168.100.1: 255.255.255.0<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4357]: 2008/02/20_16:31:23 INFO: eval ifconfig eth1:1 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4357]: 2008/02/20_16:31:23 ERROR: Could not add 192.168.100.1 to eth1: 255<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4340]: 2008/02/20_16:31:23 ERROR: Unknown error: 255<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4254]: 2008/02/20_16:31:24 ERROR: Return code 1 from /etc/ha.d/resource.d/IPaddr<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4254]: 2008/02/20_16:31:24 CRIT: Giving up resources due to failure of 192.168.100.1<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4254]: 2008/02/20_16:31:24 info: Releasing resource group: lustre01 192.168.100.1 Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4254]: 2008/02/20_16:31:24 info: Running /etc/ha.d/resource.d/Filesystem /dev/sdc /lustremds lustre stop<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Filesystem[4491]: 2008/02/20_16:31:24 INFO: Running stop for /dev/sdc on /lustremds<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Filesystem[4480]: 2008/02/20_16:31:24 INFO: Success<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4254]: 2008/02/20_16:31:24 info: Running /etc/ha.d/resource.d/IPaddr 192.168.100.1 stop<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4564]: 2008/02/20_16:31:24 INFO: Success<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[4241]: 2008/02/20_16:31:24 info: local HA resource acquisition completed (standby).<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:31:24 info: Standby resource acquisition done [foreign].<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:31:24 info: remote resource transition completed.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">hb_standby[4621]: 2008/02/20_16:31:54 Going standby [foreign].<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:31:54 info: lustre01 wants to go standby [foreign]<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:31:54 info: standby: lustre02 can take our foreign resources<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[4635]: 2008/02/20_16:31:54 info: give up foreign HA resources (standby).<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4648]: 2008/02/20_16:31:55 info: Releasing resource group: lustre02 192.168.100.2 Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4648]: 2008/02/20_16:31:55 info: Running /etc/ha.d/resource.d/Filesystem /dev/sdc /lustremds lustre stop<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Filesystem[4697]: 2008/02/20_16:31:55 INFO: Running stop for /dev/sdc on /lustremds<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Filesystem[4686]: 2008/02/20_16:31:55 INFO: Success<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4648]: 2008/02/20_16:31:55 info: Running /etc/ha.d/resource.d/IPaddr 192.168.100.2 stop<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4770]: 2008/02/20_16:31:55 INFO: Success<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[4635]: 2008/02/20_16:31:55 info: foreign HA resource release completed (standby).<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:31:55 info: Local standby process completed [foreign].<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:31:56 WARN: 1 lost packet(s) for [lustre02] [58:60]<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:31:56 info: remote resource transition completed.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:31:56 info: No pkts missing from lustre02!<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:31:56 info: Other node completed standby takeover of foreign resources.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:32:26 info: lustre02 wants to go standby [foreign]<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:32:27 info: standby: acquire [foreign] resources from lustre02<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[4811]: 2008/02/20_16:32:27 info: acquire local HA resources (standby).<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4824]: 2008/02/20_16:32:27 info: Acquiring resource group: lustre01 192.168.100.1 Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4851]: 2008/02/20_16:32:27 INFO: Resource is stopped<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4824]: 2008/02/20_16:32:27 info: Running /etc/ha.d/resource.d/IPaddr 192.168.100.1 start<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4927]: 2008/02/20_16:32:27 INFO: Using calculated nic for 192.168.100.1: eth1<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4927]: 2008/02/20_16:32:27 INFO: Using calculated netmask for 192.168.100.1: 255.255.255.0<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4927]: 2008/02/20_16:32:27 INFO: eval ifconfig eth1:2 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4927]: 2008/02/20_16:32:27 ERROR: Could not add 192.168.100.1 to eth1: 255<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4910]: 2008/02/20_16:32:27 ERROR: Unknown error: 255<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4824]: 2008/02/20_16:32:27 ERROR: Return code 1 from /etc/ha.d/resource.d/IPaddr<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4824]: 2008/02/20_16:32:27 CRIT: Giving up resources due to failure of 192.168.100.1<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4824]: 2008/02/20_16:32:27 info: Releasing resource group: lustre01 192.168.100.1 Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4824]: 2008/02/20_16:32:27 info: Running /etc/ha.d/resource.d/Filesystem /dev/sdc /lustremds lustre stop<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Filesystem[5061]: 2008/02/20_16:32:27 INFO: Running stop for /dev/sdc on /lustremds<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Filesystem[5050]: 2008/02/20_16:32:27 INFO: Success<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[4824]: 2008/02/20_16:32:27 info: Running /etc/ha.d/resource.d/IPaddr 192.168.100.1 stop<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[5134]: 2008/02/20_16:32:27 INFO: Success<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[4811]: 2008/02/20_16:32:27 info: local HA resource acquisition completed (standby).<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:32:27 info: Standby resource acquisition done [foreign].<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[3728]: 2008/02/20_16:32:28 info: remote resource transition completed.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="border-top-style: none; border-right-style: none; border-left-style: none; border-width: initial; border-color: initial; border-bottom-style: solid; border-bottom-color: windowtext; border-bottom-width: 1pt; padding-top: 0in; padding-right: 0in; padding-bottom: 1pt; padding-left: 0in; "><div style="border-top-style: none; border-right-style: none; border-bottom-style: none; border-left-style: none; border-width: initial; border-color: initial; padding-top: 0in; padding-right: 0in; padding-bottom: 0in; padding-left: 0in; margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Lustre02 (secondary mdt)<o:p></o:p></span></font></div></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[4833]: 2008/02/20_16:39:24 info: lustre01 wants to go standby [foreign]<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[4833]: 2008/02/20_16:39:25 info: standby: acquire [foreign] resources from lustre01<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">heartbeat[6658]: 2008/02/20_16:39:25 info: acquire local HA resources (standby).<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">ResourceManager[6671]: 2008/02/20_16:39:25 info: Acquiring resource group: lustre02 192.168.100.2 Filesystem::/dev/sdc::/lustremds::lustre<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">However, I cannot see Lustre mounted on either device. Does anybody know what is the issue here? This statement concerns me:<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4357]: 2008/02/20_16:31:23 ERROR: Could not add 192.168.100.1 to eth1: 255<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">IPaddr[4340]: 2008/02/20_16:31:23 ERROR: Unknown error: 255<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">BTW, 192.168.100.1 is the eth1 address on lustre01 (main mdt)<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; ">Thanks<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial; "><o:p> </o:p></span></font></div></div>_______________________________________________<br>Lustre-discuss mailing list<br><a href="mailto:Lustre-discuss@lists.lustre.org" style="color: blue; text-decoration: underline; ">Lustre-discuss@lists.lustre.org</a><br><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" style="color: blue; text-decoration: underline; ">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br></div></span></blockquote></div><br><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div>Aaron Knister</div><div>Associate Systems Analyst</div><div><div><div>Center for Ocean-Land-Atmosphere Studies</div></div><div></div></div><div><br class="khtml-block-placeholder"></div><div>(301) 595-7000</div><div><a href="mailto:aaron@iges.org">aaron@iges.org</a></div><div><br class="khtml-block-placeholder"></div><br class="Apple-interchange-newline"></span></div></span></div></span><br class="Apple-interchange-newline"> </div><br></body></html>