<div dir="ltr">Hi Alex,<div><br></div><div class="gmail_extra"><div class="gmail_quote">On Thu, Aug 3, 2017 at 6:53 PM, Alexander I Kulyavtsev <span dir="ltr"><<a href="mailto:aik@fnal.gov" target="_blank">aik@fnal.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word">
<div>Lustre IO size is 1MB; you have zfs record 4MB.</div>
<div>Do you see IO rate change when tar record size set to 4 MB (tar -b 8192) ?</div></div></blockquote><div><div><br></div><div> I'm actually using 4M IO, which gets picked up automatically by the OSTs on 4M ZFS datasets. I tried -b 8192, but it was slightly slower than -b 2048, which is significantly faster (about 75% wall time) than the default -b 20. I do have the directory's stripe_size=1M and stripe_count=1 because that had the best overall performance for our typical workloads. I'm sure there is still LOTS of room for improvement by tweaking parameters, which I'll get to as soon as all the other fires are put out :)</div></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word">
<div><br>
</div>
<div>How many data disks do you have at raidz2?</div></div></blockquote><div><br></div><div> There are three OSTs on a single OSS. Each target is on a dataset in a pool of 30 SAS disks arranged as 3 10-disk raidz2 vdevs. We are simultaneously running a BeeGFS system on the same hardware, so each zpool has a Lustre dataset and a BeeGFS dataset. I noticed the tar file disk space discrepancy during benchmark comparisons of the two file systems.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word">
<div><br>
</div>
<div>zfs may write few extra empty blocks to improve defragmentation; IIRC this patch is on by default in zfs 0.7 to improve io rates for some disks:</div>
<div><a href="https://github.com/zfsonlinux/zfs/pull/5931" target="_blank">https://github.com/zfsonlinux/<wbr>zfs/pull/5931</a></div>
<div><br></div></div></blockquote><div> It should have that patch (using the 0.7.0 rpms from the non-testing kmod el7.3 repository), and I also have zfs_vdev_aggregation_limit=16MiB, so that should not be the issue.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div>
</div>
<div> If I understand it correctly, for very small files (untarred) there will be overhead to pad file to record size, and for extra padding to P+1 records (=P extra) and for parity records (+P). Plus metadata size for the lustre ost object. For raidz2
with P=2 it is factor 5x or more.</div>
<div><br></div></div></blockquote><div>I am absolutely UNsurprised that the unpacked files take a lot of space. From the nominal size of the uncompressed tar, they average 8.6K/file. Unarchived, but compressed at the zfs level with lz4, they average 11.4K/file. If an 8K file can be compressed to less than 4K, storing it with two parity blocks should take minimum 12K. 11.4K seems a reasonable average if there are any empty files in there.</div><div><br></div><div>We have the metadata target on SSDs with ashift=9. This works out to 0.4K/file on the MDT. We also have dnodesize=auto on the MDT, so I am waiting for the zfs 0.7.1 rpms to include <a href="https://github.com/zfsonlinux/zfs/pull/6439">https://github.com/zfsonlinux/zfs/pull/6439</a> before opening the file system up to non-scratch usage.</div><div><br></div><div>-Nate</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div>
</div>
<div>
<div>Alex.</div>
</div>
<br>
<div>
<blockquote type="cite"><div><div class="gmail-m_470226709547922736h5">
<div>On Aug 3, 2017, at 7:28 PM, Nathan R.M. Crawford <<a href="mailto:nrcrawfo@uci.edu" target="_blank">nrcrawfo@uci.edu</a>> wrote:</div>
<br class="gmail-m_470226709547922736m_7966596734538841806Apple-interchange-newline">
</div></div><div><div><div class="gmail-m_470226709547922736h5">
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Off-list, it was suggested that tar's default 10K blocking may be the cause. I increased it to 1MiB using "tar -b 2048 ...", which seems to result in the expected 9.3 GiB disk usage. It probably makes archives incompatible with very old versions of tar, but
meh.
<div><br>
</div>
<div>-Nate</div>
</div>
<div class="gmail_extra" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br>
<div class="gmail_quote">On Thu, Aug 3, 2017 at 3:07 PM, Nathan R.M. Crawford<span class="gmail-m_470226709547922736m_7966596734538841806Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:nrcrawfo@uci.edu" target="_blank">nrcrawfo@uci.edu</a>></span><span class="gmail-m_470226709547922736m_7966596734538841806Apple-converted-space"> </span>wr<wbr>ote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"> In testing how to cope with naive users generating millions of tiny files, I noticed some surprising (to me) behavior on a lustre 2.10/ZFS 0.7.0 system.
<div><br>
</div>
<div> The test directory (based on actual user data) contains about 4 million files (avg size 8.6K) in three subdirectories. Making tar files of each subdirectory gives the total nominal size of 34GB, and using "zfs list", the tar files took up 33GB
on disk.</div>
<div><br>
</div>
<div> The initially surprising part is that making copies of the tar files only adds 9GB to the disk usage. I suspect that the creation of the tar files is as a bunch of tiny appendings, and with a raidz2 on ashift=12 disks (4MB max recordsize), there
is some overhead/wasted space on each mini-write. The copies of the tar files, however, could be made as a single write that avoided the overhead and probably allowed the lz4 compression to be more efficient. <br clear="all">
<div><br>
</div>
<div> Are there any tricks or obscure tar options that make archiving millions of tiny files on a Lustre system avoid this? It is not a critical issue, as taking a minute to copy the tar files is simple enough.</div>
<div><br>
</div>
<div>-Nate</div>
<span class="gmail-m_470226709547922736m_7966596734538841806HOEnZb"><font color="#888888">
<div><br>
</div>
--<span class="gmail-m_470226709547922736m_7966596734538841806Apple-converted-space"> </span><br>
<div class="gmail-m_470226709547922736m_7966596734538841806m_7323882048108971997gmail_signature">
<div dir="ltr">
<pre>Dr. Nathan Crawford <a href="mailto:nathan.crawford@uci.edu" target="_blank">nathan.crawford@uci.edu</a>
Modeling Facility Director
Department of Chemistry
1102 Natural Sciences II Office: 2101 Natural Sciences II
University of California, Irvine Phone: <a href="tel:(949)%20824-4508" value="+19498244508" target="_blank">949-824-4508</a>
Irvine, CA 92697-2025, USA</pre>
</div>
</div>
</font></span></div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
--<span class="gmail-m_470226709547922736m_7966596734538841806Apple-converted-space"> </span><br>
<div class="gmail-m_470226709547922736m_7966596734538841806gmail_signature">
<div dir="ltr">
<pre>Dr. Nathan Crawford <a href="mailto:nathan.crawford@uci.edu" target="_blank">nathan.crawford@uci.edu</a>
Modeling Facility Director
Department of Chemistry
1102 Natural Sciences II Office: 2101 Natural Sciences II
University of California, Irvine Phone: <a href="tel:(949)%20824-4508" value="+19498244508" target="_blank">949-824-4508</a>
Irvine, CA 92697-2025, USA</pre>
</div>
</div>
</div>
</div></div><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline">______________________________<wbr>_________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline">lustre-discuss
mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<a href="mailto:lustre-discuss@lists.lustre.org" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">lustre-discuss@lists.lustre.or<wbr>g</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">http://lists.lustre.org/listin<wbr>fo.cgi/lustre-discuss-lustre.<wbr>org</a></div>
</blockquote>
</div>
<br>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail-m_470226709547922736gmail_signature"><div dir="ltr"><pre>Dr. Nathan Crawford <a href="mailto:nathan.crawford@uci.edu" target="_blank">nathan.crawford@uci.edu</a>
Modeling Facility Director
Department of Chemistry
1102 Natural Sciences II Office: 2101 Natural Sciences II
University of California, Irvine Phone: <a href="tel:(949)%20824-4508" value="+19498244508" target="_blank">949-824-4508</a>
Irvine, CA 92697-2025, USA</pre></div></div>
</div></div>