<div dir="ltr">On a somewhat-related question, what are the expected trade-offs when splitting the "striping" between ZFS (striping over vdevs) and Lustre (striping over OSTs)? <div><br></div><div>Specific example: if one has an OSS with 40 disks and intends to use 10-disk raidz2 vdevs, how do these options compare?:</div><div><br></div><div>A) 4 OSTs, each on a zpool with a single raidz2 vdev,</div><div>B) 2 OSTs, each on a zpool with two vdevs, and</div><div>C) 1 OST, on a zpool with 4 vdevs?</div><div><br></div><div> I've done some simple testing with obdfilter-survey and multiple-client file operations on some actual user data, and am leaning more toward "A". However, the differences weren't overwhelming, and I am probably neglecting some important corner cases. Handling striping pattern at the Lustre level (A) also allows tuning on a per-file basis.</div><div><br></div><div>-Nate</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 3, 2017 at 1:15 AM, Dilger, Andreas <span dir="ltr"><<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">We have seen performance improvements with multiple zpools/OSTs per OSS. However, with only 5x NVMe devices per OSS you don't have many choices in terms of redundancy, unless you are not using any redundancy at all, just raw bandwidth?<br>
<br>
The other thing to consider is what the network bandwidth is vs. the NVMe bandwidth? With similar test systems using NVMe devices without redundancy we've seen multi GB/s, so if you aren't using OPA/IB network then that will likely be your bottleneck. Even if the TCP is fast enough, the CPU overhead and data copies will probably kill the performance.<br>
<br>
In the end, you can probably test with a few of configs to see which one will give the best performance - mirror, single RAID-Z, two RAID-Z pools on half-sized partitions, five no-redundancy zpools with one VDEV each, single no-redundancy zpool with five VDEVs.<br>
<br>
Cheers, Andreas<br>
<br>
PS - there is initial snapshot functionality in the 2.10 release.<br>
<div class="HOEnZb"><div class="h5"><br>
> On Jul 2, 2017, at 10:07, Brian Andrus <<a href="mailto:toomuchit@gmail.com">toomuchit@gmail.com</a>> wrote:<br>
><br>
> All,<br>
><br>
> We have been having some discussion about the best practices when creating OSTs with ZFS.<br>
><br>
> The basic question is: What is the best ration of OSTs per OSS when using ZFS?<br>
> It is easy enough to do a single OST with all disks and have reliable data protection provided by ZFS. It may be an better scenario when snapshots of lfs become a feature as well.<br>
><br>
> However, multiple OSTs can mean more stripes and faster reads/writes. I have seen some tests that were done quite some time ago which may not be so valid anymore with the updates to Lustre.<br>
><br>
> We have a system for testing that has 5 NVMes each. We can do 1 zfs file system with all or we can separate them into 5 (which would forgo some of the features of zfs).<br>
><br>
> Any prior experience/knowledge/<wbr>suggestions would be appreciated.<br>
><br>
> Brian Andrus<br>
><br>
> ______________________________<wbr>_________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><pre>Dr. Nathan Crawford <a href="mailto:nathan.crawford@uci.edu" target="_blank">nathan.crawford@uci.edu</a>
Modeling Facility Director
Department of Chemistry
1102 Natural Sciences II Office: 2101 Natural Sciences II
University of California, Irvine Phone: 949-824-4508
Irvine, CA 92697-2025, USA</pre></div></div>
</div>