<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">Hi Andreas</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">I did take some time, to get back to this. I started to try out this configuration on bunch of VM's with the powerful underlying HW.</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><b><u>Configuration:</u></b></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">1 Physical machine hosts 2 VM ( Vm1 and Vm2 ) . Both of them have kernel 3.10.0-327.13.1.el7_lustre.x86_64 , Zfs , Iscsi</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">Vm1 - disk 1 , disk 2</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">Vm2 - disk 3, disk 4</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">After Iscsi setup</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">Vm1 - disk1 , disk 3</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">Vm2 - disk 4, disk 2</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">After zpool</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">Vm1 - disk1 || disk 3 ( zpool mirror )- for mgs</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">Vm2- disk4 || disk 2 ( zpool mirror ) - for mdt</div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><br></div><div class="gmail_default" style=""><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif">[root@lustre_mgs01_vm03 ~]# zpool status</font></div><div class="gmail_default" style=""><span style="color:rgb(153,51,153);font-family:tahoma,sans-serif"> pool: mds1_2</span><br></div><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif"> state: ONLINE</font></div><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif"> scan: none requested</font></div><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif">config:</font></div><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif"><br></font></div><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif"> NAME STATE READ WRITE CKSUM</font></div><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif"> mds1_2 ONLINE 0 0 0</font></div><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif"> mirror-0 ONLINE 0 0 0</font></div><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif"> sdb ONLINE 0 0 0</font></div><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif"> vdb2 ONLINE 0 0 0</font></div><div class="gmail_default" style="color:rgb(153,51,153);font-family:tahoma,sans-serif;font-size:small"><br></div></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)">when setting up mgs/mdt i get the following error<br></div><div class="gmail_default" style=""><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif"><br></font></div><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif">[root@lustre_mgs01_vm03 /]# mkfs.lustre --mgs --backfstype=zfs mds1_2/mgs</font></div><div class="gmail_default" style=""><span style="color:rgb(153,51,153);font-family:tahoma,sans-serif">mkfs.lustre FATAL: unhandled/unloaded fs type 5 'zfs'</span><br></div><div class="gmail_default" style=""><span style="color:rgb(153,51,153);font-family:tahoma,sans-serif">mkfs.lustre FATAL: unable to prepare backend (22)</span><br></div><div class="gmail_default" style=""><font color="#993399" face="tahoma, sans-serif">mkfs.lustre: exiting with 22 (Invalid argument)</font></div><div style="color:rgb(153,51,153);font-family:tahoma,sans-serif;font-size:small"><br></div><div style=""><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small"><font color="#993399" face="tahoma, sans-serif">when i searched for the specific error i ran into this Jira..<a href="https://jira.hpdd.intel.com/browse/LU-7601">https://jira.hpdd.intel.com/browse/LU-7601</a></font></div><div style="color:rgb(153,51,153);font-family:tahoma,sans-serif;font-size:small"><font color="#993399" face="tahoma, sans-serif">i have lustre version</font></div><div style=""><font color="#993399" face="tahoma, sans-serif" style=""><div style="">[root@lustre_mgs01_vm03 /]# cat /proc/fs/lustre/version</div><div style="">lustre: 2.8.53_11_gfd4ab6e</div><div style="">kernel: patchless_client</div><div style="">build: 2.8.53_11_gfd4ab6e</div><div style="color:rgb(153,51,153);font-family:tahoma,sans-serif;font-size:small"><br></div></font></div></div><div style="color:rgb(153,51,153);font-family:tahoma,sans-serif;font-size:small">I found an earlier discussion on similar topic. I plan to setup something similar but with Iscsi instead of common storage boxes. I dont see the output similar to this thread for mkfs.lustre command.</div><div style=""><font color="#993399" face="tahoma, sans-serif"><a href="https://lists.01.org/pipermail/hpdd-discuss/2013-December/000662.html">https://lists.01.org/pipermail/hpdd-discuss/2013-December/000662.html</a></font></div><div style=""><font color="#993399" face="tahoma, sans-serif"><br></font></div><div style=""><font color="#993399" face="tahoma, sans-serif">I understand that this might not be a regular setup, but i would like to set it up and see the performance if possible.</font></div><div style=""><font color="#993399" face="tahoma, sans-serif">Please let me if i am missing something.</font></div><div style=""><font color="#993399" face="tahoma, sans-serif"><br></font></div><div style=""><font color="#993399" face="tahoma, sans-serif">Thanks</font></div><div style=""><font color="#993399" face="tahoma, sans-serif">Divakar</font></div><div style=""><font color="#993399" face="tahoma, sans-serif"><br></font></div><div style=""><font color="#993399" face="tahoma, sans-serif"><br></font></div></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:rgb(153,51,153)"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Feb 6, 2016 at 1:57 AM, Dilger, Andreas <span dir="ltr"><<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif"><span class="">
<div>
<div>
<div>On 2016/02/05, 17:08, "lustre-discuss on behalf of sohamm" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a> on behalf of
<a href="mailto:sohamm@gmail.com" target="_blank">sohamm@gmail.com</a>> wrote:</div>
</div>
</div>
<span>
<div><br>
</div>
<blockquote style="BORDER-LEFT:#b5c4df 5 solid;PADDING:0 0 0 5;MARGIN:0 0 0 5">
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:#993399">
Hi</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:#993399">
<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small;color:#993399">
I have been reading bunch of documents on how failures are handled in Luster and almost all of them seem to indicate that i would need a shared disks/ target for MDS or OSS failover configuration. I want to know if failover configuration is possible without
the shared disks. Eg i have one physical box i want to configure as OSS/OST and another as MGS/MDS/MDT. Each physical box will have its own HDD/SDDs and are connected via ethernet. Please guide and point me to any good documentation available for such configuration.</div>
</div>
</div>
</blockquote>
</span>
<div><br>
</div>
</span><div>It is _possible_ to do this without shared disks, if there is some other mechanism to make the data available on both nodes. One option is to use iSCSI targets (SRP or iSER) and mirror the drives across the two servers using ZFS, making sure you serve
each mirrored device from only one node. Then, if the primary server fails you can mount the filesystem on the backup node. This is described in <a href="http://wiki.lustre.org/MDT_Mirroring_with_ZFS_and_SRP" target="_blank">http://wiki.lustre.org/MDT_Mirroring_with_ZFS_and_SRP</a> and <a href="http://cdn.opensfs.org/wp-content/uploads/2011/11/LUG-2012.pptx" target="_blank">http://cdn.opensfs.org/wp-content/uploads/2011/11/LUG-2012.pptx</a>
.</div>
<div><br>
</div>
<div>Note that if you only have a 2-way mirror you've lost 1/2 of your disks during failover. That might be OK for the MDT if it has been configured correctly, since there are additional copies of metadata. For the OST you could use RAID-1+5 or RAID-1+6 (e.g.
mirror of RAID-5/6 devices on each node). With a more complex configuration it would even potentially be possible to export iSCSI disks from a group of nodes and use RAID-6 of disks from different nodes so that redundancy isn't lost when a single node goes
down. That might get hairy during configuration for a large system.</div>
<div><br>
</div>
<div>Another alternative to iSCSI+ZFS would be some other form of network block device (e.g. NBD or DRBD) and then build your target on top of that. It is essentially the same but the consistency is managed by the block device instead of the filesystem. IMHO
(just a gut feeling, never tested) having a "robust" network block device would be slower than having ZFS do this because the block device doesn't know the details of what the filesystem is doing, and will add its own overhead to provide its own consistency
in addition to the consistency provided by ZFS itself.</div>
<div><br>
</div>
<div>That said, this isn't a typical Lustre configuration, but I think there would definitely be other interested parties if you tried this out and reported your results back here.</div>
<div><br>
</div>
<div>
<div>
<div>
<div>Cheers, Andreas</div><span class="HOEnZb"><font color="#888888">
<div>-- </div>
<div>Andreas Dilger</div>
</font></span></div><span class="HOEnZb"><font color="#888888">
<div>Lustre Principal Architect</div>
<div>Intel High Performance Data Division</div>
</font></span></div>
<div><br>
</div>
<span>
<div></div>
</span></div>
</div>
</blockquote></div><br></div>