<div dir="ltr">Hi Udai,<div><br></div><div>Can you provide /var/log/messages at the moment RA is moving the ZFS pool? </div><div><br></div><div>I see you are using pacemaker ZFS agent and Lustre agent.</div><div><br></div><div>Maybe one is to import zpool and the other for mount, but if Lustre agent is written correctly, it will import zpool and mount MDT and OSTs. </div><div><br></div><div>I believe it is safe to use only one.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">2019년 8월 26일 (월) 오전 1:53, Udai Sharma <<a href="mailto:udai.sharma5@gmail.com">udai.sharma5@gmail.com</a>>님이 작성:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><p class="MsoNormal">Hello Team,<u></u><u></u></p><p class="MsoNormal">Could you please help me out here.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">-Udai</p></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_905892297952115379gmail-m_3763579105689152847WordSection1">
<p class="MsoNormal"><b>From:</b> lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>>
<b>On Behalf Of </b>Udai Sharma<br>
<b>Sent:</b> Friday, August 23, 2019 4:40 PM<br>
<b>To:</b> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> [SPAMMY (6.924)][lustre-discuss] Lustre in HA-LVM Cluster issue<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Hi Team,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Starting with topology and configurations:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">---------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Topology:<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">[HA1]< -----[N3]----- >[N4]<u></u><u></u></p>
<p class="MsoNormal"> | | |<u></u><u></u></p>
<p class="MsoNormal"> | | |<u></u><u></u></p>
<p class="MsoNormal"> -----[Client]----------<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">[N1,N2] = HA1 --> OSTs<u></u><u></u></p>
<p class="MsoNormal"> N3 --- > MGS<u></u><u></u></p>
<p class="MsoNormal"> N4 --- > MDT<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">N1 -> 3 Logical volumes [OST1,OST2,OST3]<u></u><u></u></p>
<p class="MsoNormal">N2 -> 3 Logical volumes [OST4,OST5,OST6]<u></u><u></u></p>
<p class="MsoNormal">N3 -> 1 Logical volume [MGT1]<u></u><u></u></p>
<p class="MsoNormal">N4 -> 1 Logical volume [MDT1]<u></u><u></u></p>
<p class="MsoNormal">------------------------------------------<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">N3 [MGS]<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Created Zpool, formatted and Mounted it. <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">zpool create -f -O canmount=off -o multihost=on -o cachefile=none lustre /dev/mgs/mgs01<u></u><u></u></p>
<p class="MsoNormal">mkfs.lustre --reformat --mgs --backfstype=zfs lustre/mgs01<u></u><u></u></p>
<p class="MsoNormal">mount.lustre lustre/mgs01 /mnt/mgs/<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">------------------------------------------<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">N4 [MDT]<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Created Zpool, formatted and Mounted it. <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">zpool create -f -O canmount=off -o multihost=on -o cachefile=none lustre /dev/mdt/mdt01<u></u><u></u></p>
<p class="MsoNormal">mkfs.lustre --reformat --mdt --fsname=lustre --index=0 <a href="mailto:--mgsnode=10.2.2.202@tcp1" target="_blank">
--mgsnode=10.2.2.202@tcp1</a> --backfstype=zfs lustre/mdt01<u></u><u></u></p>
<p class="MsoNormal">mount.lustre lustre/mdt01 /mnt/mdt<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">---------------------------------------- <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">HA1 [HA-LVM system] <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><u>N1 [OST1,OST2,OST3]</u><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Created Zpool, formatted and Mounted it. <u></u><u></u></p>
<p class="MsoNormal">zpool create lustre -f -O canmount=off -o multihost=on -o cachefile=none /dev/vg_e/thinvolume1 /dev/vg_e/thinvolume2 /dev/vg_e/thinvolume3<u></u><u></u></p>
<p class="MsoNormal">mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=111
<a href="mailto:--mgsnode=10.2.2.202@tcp1" target="_blank">--mgsnode=10.2.2.202@tcp1</a> <a href="mailto:--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1" target="_blank">
--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1</a> lustre/ost01 ; mount.lustre lustre/ost01 /mnt/ost01/
<u></u><u></u></p>
<p class="MsoNormal">mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=222
<a href="mailto:--mgsnode=10.2.2.202@tcp1" target="_blank">--mgsnode=10.2.2.202@tcp1</a> <a href="mailto:--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1" target="_blank">
--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1</a> lustre/ost02 ; mount.lustre lustre/ost02 /mnt/ost02/<u></u><u></u></p>
<p class="MsoNormal">mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=333
<a href="mailto:--mgsnode=10.2.2.202@tcp1" target="_blank">--mgsnode=10.2.2.202@tcp1</a> <a href="mailto:--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1" target="_blank">
--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1</a> lustre/ost03 ; mount.lustre lustre/ost03 /mnt/ost03/<u></u><u></u></p>
<p class="MsoNormal">df -h | grep lustre<u></u><u></u></p>
<p class="MsoNormal">lustre/ost01 287G 3.0M 287G 1% /mnt/ost01<u></u><u></u></p>
<p class="MsoNormal">lustre/ost02 287G 3.0M 287G 1% /mnt/ost02<u></u><u></u></p>
<p class="MsoNormal">lustre/ost03 287G 3.0M 287G 1% /mnt/ost03<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><u>N2 [OST4,OST5,OST6]</u><u></u><u></u></p>
<p class="MsoNormal"><u> </u><u></u><u></u></p>
<p class="MsoNormal">Created Zpool, formatted and Mounted it. <u></u><u></u></p>
<p class="MsoNormal">zpool create -f -O canmount=off -o multihost=on -o cachefile=none lustre /dev/vg_p/thinvolume1 /dev/vg_p/thinvolume2 /dev/vg_p/thinvolume3
<u></u><u></u></p>
<p class="MsoNormal">mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=444
<a href="mailto:--mgsnode=10.2.2.202@tcp1" target="_blank">--mgsnode=10.2.2.202@tcp1</a> <a href="mailto:--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1" target="_blank">
--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1</a> lustre/ost04 ; mount.lustre lustre/ost04 /mnt/ost04
<u></u><u></u></p>
<p class="MsoNormal">mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=555
<a href="mailto:--mgsnode=10.2.2.202@tcp1" target="_blank">--mgsnode=10.2.2.202@tcp1</a> <a href="mailto:--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1" target="_blank">
--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1</a> lustre/ost05 ; mount.lustre lustre/ost05 /mnt/ost05<u></u><u></u></p>
<p class="MsoNormal">mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=666
<a href="mailto:--mgsnode=10.2.2.202@tcp1" target="_blank">--mgsnode=10.2.2.202@tcp1</a> <a href="mailto:--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1" target="_blank">
--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1</a> lustre/ost06 ; mount.lustre lustre/ost06 /mnt/ost06<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">df -h | grep lustre<u></u><u></u></p>
<p class="MsoNormal">lustre/ost04 287G 3.0M 287G 1% /mnt/ost04<u></u><u></u></p>
<p class="MsoNormal">lustre/ost05 287G 3.0M 287G 1% /mnt/ost05<u></u><u></u></p>
<p class="MsoNormal">lustre/ost06 287G 3.0M 287G 1% /mnt/ost06<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><u>Created PCS cluster over HA1. </u><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Resource Group: electron<u></u><u></u></p>
<p class="MsoNormal"> vg_e (ocf::heartbeat:LVM): Started gp-electron<u></u><u></u></p>
<p class="MsoNormal"> zfs-pool-electron (ocf::heartbeat:ZFS): Started electron<u></u><u></u></p>
<p class="MsoNormal"> lustre-ost1 (ocf::heartbeat:Lustre): Started electron<u></u><u></u></p>
<p class="MsoNormal"> lustre-ost2 (ocf::heartbeat:Lustre): Started electron<u></u><u></u></p>
<p class="MsoNormal"> lustre-ost3 (ocf::heartbeat:Lustre): Started electron<u></u><u></u></p>
<p class="MsoNormal"> Resource Group: proton<u></u><u></u></p>
<p class="MsoNormal"> vg_p (ocf::heartbeat:LVM): Started gp-proton<u></u><u></u></p>
<p class="MsoNormal"> zfs-pool-proton (ocf::heartbeat:ZFS): Started proton<u></u><u></u></p>
<p class="MsoNormal"> lustre-ost4 (ocf::heartbeat:Lustre): Started proton<u></u><u></u></p>
<p class="MsoNormal"> lustre-ost5 (ocf::heartbeat:Lustre): Started proton<u></u><u></u></p>
<p class="MsoNormal"> lustre-ost6 (ocf::heartbeat:Lustre): Started proton<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">---------------------------------------- <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Client: <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"># mount | grep lustre<u></u><u></u></p>
<p class="MsoNormal"><a href="mailto:10.2.2.202@tcp1:/lustre" target="_blank">10.2.2.202@tcp1:/lustre</a> on /lustre type lustre (rw,lazystatfs)<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">#lfs osts<u></u><u></u></p>
<p class="MsoNormal">OBDS:<u></u><u></u></p>
<p class="MsoNormal">1: lustre-OST0001_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">2: lustre-OST0002_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">3: lustre-OST0003_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">4: lustre-OST0004_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">5: lustre-OST0005_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">6: lustre-OST0006_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">10: lustre-OST000a_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">11: lustre-OST000b_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">20: lustre-OST0014_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">22: lustre-OST0016_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">30: lustre-OST001e_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">33: lustre-OST0021_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">40: lustre-OST0028_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">44: lustre-OST002c_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">50: lustre-OST0032_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">55: lustre-OST0037_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">60: lustre-OST003c_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">66: lustre-OST0042_UUID INACTIVE<u></u><u></u></p>
<p class="MsoNormal">111: lustre-OST006f_UUID ACTIVE<u></u><u></u></p>
<p class="MsoNormal">222: lustre-OST00de_UUID ACTIVE<u></u><u></u></p>
<p class="MsoNormal">333: lustre-OST014d_UUID ACTIVE<u></u><u></u></p>
<p class="MsoNormal">444: lustre-OST01bc_UUID ACTIVE<u></u><u></u></p>
<p class="MsoNormal">555: lustre-OST022b_UUID ACTIVE<u></u><u></u></p>
<p class="MsoNormal">666: lustre-OST029a_UUID ACTIVE<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"># lfs mdts<u></u><u></u></p>
<p class="MsoNormal">MDTS:<u></u><u></u></p>
<p class="MsoNormal">0: lustre-MDT0000_UUID ACTIVE<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">OSTs are part of active-passive HA-LVM cluster. As per configuration, in case of resource failure, RA moves the resource to other node. It’s working fine.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Issues seen:<u></u><u></u></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="gmail-m_905892297952115379gmail-m_3763579105689152847gmail-msolistparagraph" style="margin-top:0in;margin-bottom:0.0001pt">
OST00* are going to INACTIVE state if the corresponding disk in unmounted from the OST server, it never becomes active even-though if it mounted again. Had to format every time with –index to get the volumes listed again. Hence, so many inactive nodes in the
‘lctl dl’ output.<u></u><u></u></li><li class="gmail-m_905892297952115379gmail-m_3763579105689152847gmail-msolistparagraph" style="margin-top:0in;margin-bottom:0.0001pt">
Recovery_status at N4[MDT] is always inactive. How to enable recovery, in case of failure.<u></u><u></u></li><li class="gmail-m_905892297952115379gmail-m_3763579105689152847gmail-msolistparagraph" style="margin-top:0in;margin-bottom:0.0001pt">
How to re-activate the INACTIVE objects at OSTs?<u></u><u></u></li><li class="gmail-m_905892297952115379gmail-m_3763579105689152847gmail-msolistparagraph" style="margin-top:0in;margin-bottom:0.0001pt">
In case of failover of HA resource services gets moved to other peer, but IOs, iozone or dd, never resumes.
<u></u><u></u></li></ol>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Please advise.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">PS: lctl 2.12.2<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks in advance.<u></u><u></u></p>
<p class="MsoNormal">Udai<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote></div></div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><font face="'courier new', monospace">Jongwoo Han</font><div><font face="'courier new', monospace">+82-505-227-6108</font></div></div>