<div dir="ltr"><div>Hi Team,</div><div><br></div><div>Starting with topology and configurations:</div><div><br></div><div>---------------------------</div><div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Topology:</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[HA1]< -----[N3]----- >[N4]</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> |
| |</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> |
| |</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> -----[Client]----------</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[N1,N2] = HA1 --> OSTs</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> N3 --- > MGS</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> N4 --- > MDT</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">N1 -> 3 Logical volumes [OST1,OST2,OST3]</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">N2 -> 3 Logical volumes [OST4,OST5,OST6]</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">N3 -> 1 Logical volume
[MGT1]</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">N4 -> 1 Logical volume
[MDT1]</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">------------------------------------------</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">N3 [MGS]</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Created Zpool, formatted and Mounted it. </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">zpool create -f -O canmount=off -o multihost=on -o
cachefile=none lustre /dev/mgs/mgs01</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">mkfs.lustre --reformat --mgs --backfstype=zfs lustre/mgs01</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">mount.lustre lustre/mgs01 /mnt/mgs/</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">------------------------------------------</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">N4 [MDT]</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Created Zpool, formatted and Mounted it. </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">zpool create -f -O canmount=off -o multihost=on -o
cachefile=none lustre /dev/mdt/mdt01</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">mkfs.lustre
--reformat --mdt --fsname=lustre --index=0 --mgsnode=10.2.2.202@tcp1
--backfstype=zfs lustre/mdt01</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">mount.lustre lustre/mdt01 /mnt/mdt</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">---------------------------------------- </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">HA1 [HA-LVM system] </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><u>N1 [OST1,OST2,OST3]</u></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Created Zpool, formatted and Mounted it. </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">zpool create lustre -f -O canmount=off -o multihost=on -o
cachefile=none /dev/vg_e/thinvolume1 /dev/vg_e/thinvolume2
/dev/vg_e/thinvolume3</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">mkfs.lustre
--reformat --ost --backfstype=zfs --fsname=lustre --index=111
--mgsnode=10.2.2.202@tcp1 --servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1
lustre/ost01 ; mount.lustre lustre/ost01
/mnt/ost01/
</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">mkfs.lustre
--reformat --ost --backfstype=zfs --fsname=lustre --index=222 --mgsnode=10.2.2.202@tcp1
--servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1 lustre/ost02 ; mount.lustre lustre/ost02 /mnt/ost02/</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">mkfs.lustre
--reformat --ost --backfstype=zfs --fsname=lustre --index=333
--mgsnode=10.2.2.202@tcp1 --servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1 lustre/ost03 ; mount.lustre lustre/ost03 /mnt/ost03/</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">df -h | grep lustre</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">lustre/ost01
287G 3.0M 287G
1% /mnt/ost01</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">lustre/ost02
287G 3.0M 287G
1% /mnt/ost02</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">lustre/ost03
287G 3.0M 287G
1% /mnt/ost03</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><u>N2 [OST4,OST5,OST6]</u></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><u><span style="text-decoration-line:none"> </span></u></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Created Zpool, formatted and Mounted it. </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">zpool create -f -O canmount=off -o multihost=on -o cachefile=none
lustre /dev/vg_p/thinvolume1
/dev/vg_p/thinvolume2 /dev/vg_p/thinvolume3 </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">mkfs.lustre
--reformat --ost --backfstype=zfs --fsname=lustre --index=444
--mgsnode=10.2.2.202@tcp1 --servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1
lustre/ost04 ; mount.lustre lustre/ost04 /mnt/ost04
</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">mkfs.lustre
--reformat --ost --backfstype=zfs --fsname=lustre --index=555
--mgsnode=10.2.2.202@tcp1 --servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1
lustre/ost05 ; mount.lustre lustre/ost05 /mnt/ost05</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">mkfs.lustre
--reformat --ost --backfstype=zfs --fsname=lustre --index=666
--mgsnode=10.2.2.202@tcp1 --servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1
lustre/ost06 ; mount.lustre lustre/ost06 /mnt/ost06</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">df -h | grep lustre</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">lustre/ost04
287G 3.0M 287G
1% /mnt/ost04</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">lustre/ost05
287G 3.0M 287G
1% /mnt/ost05</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">lustre/ost06
287G 3.0M 287G
1% /mnt/ost06</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><u>Created PCS cluster over HA1. </u></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Resource Group: electron</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> vg_e (ocf::heartbeat:LVM): Started gp-electron</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
zfs-pool-electron
(ocf::heartbeat:ZFS): Started
electron</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> lustre-ost1 (ocf::heartbeat:Lustre): Started electron</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> lustre-ost2 (ocf::heartbeat:Lustre): Started electron</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> lustre-ost3 (ocf::heartbeat:Lustre): Started electron</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> Resource Group:
proton</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> vg_p (ocf::heartbeat:LVM): Started gp-proton</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
zfs-pool-proton
(ocf::heartbeat:ZFS): Started
proton</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> lustre-ost4 (ocf::heartbeat:Lustre): Started proton</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> lustre-ost5 (ocf::heartbeat:Lustre): Started proton</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> lustre-ost6 (ocf::heartbeat:Lustre): Started proton</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">---------------------------------------- </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Client: </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"># mount | grep lustre</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">10.2.2.202@tcp1:/lustre on /lustre type lustre
(rw,lazystatfs)</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">#lfs osts</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">OBDS:</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1: lustre-OST0001_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">2: lustre-OST0002_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">3: lustre-OST0003_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">4: lustre-OST0004_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">5: lustre-OST0005_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">6: lustre-OST0006_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">10: lustre-OST000a_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">11: lustre-OST000b_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">20: lustre-OST0014_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">22: lustre-OST0016_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">30: lustre-OST001e_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">33: lustre-OST0021_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">40: lustre-OST0028_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">44: lustre-OST002c_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">50: lustre-OST0032_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">55: lustre-OST0037_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">60: lustre-OST003c_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">66: lustre-OST0042_UUID INACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">111: lustre-OST006f_UUID ACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">222: lustre-OST00de_UUID ACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">333: lustre-OST014d_UUID ACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">444: lustre-OST01bc_UUID ACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">555: lustre-OST022b_UUID ACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">666: lustre-OST029a_UUID ACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"># lfs mdts</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">MDTS:</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">0: lustre-MDT0000_UUID ACTIVE</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><br></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-size:11pt">OSTs are part of active-passive HA-LVM cluster. As per
configuration, in case of resource failure, RA moves the resource to other
node. It’s working fine.</span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Issues seen:</p>
<ol style="margin-top:0in;margin-bottom:0in" start="1" type="1">
<li class="gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">OST00*
are going to INACTIVE state if the corresponding disk in unmounted from
the OST server, it never becomes active even-though if it mounted again.
Had to format every time with –index to get the volumes listed again. Hence,
so many inactive nodes in the ‘lctl dl’ output.</li>
<li class="gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Recovery_status
at N4[MDT] is always inactive. How to enable recovery, in case of failure.</li>
<li class="gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">How
to re-activate the INACTIVE objects at OSTs?</li>
<li class="gmail-MsoListParagraph" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">In
case of failover of HA resource services gets moved to other peer, but IOs, iozone or dd, never resumes. </li>
</ol>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Please advise.</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">PS: lctl 2.12.2</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Thanks in advance.</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Udai</p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><br></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p></div></div>