<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">Interesting, so my fears were well founded. Basically, once you choose ZFS, you are stuck with it. The only way to migrate off ZFS is to create a new Lustre file system and copy all the contents out.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">I'll get to the bottom of the slow send/receive. I am using ashift=12 for both file systems. I'll switch to using ashift=9 for the SSD's. Though, I assume this only reduces the disk usage of the MDT and doesn't help latency at all. Thanks.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jul 23, 2017 at 7:14 AM, Dilger, Andreas <span dir="ltr"><<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>Using rsync or tar to backup/restore a ZFS MDT is not supported, because this changes the dnode numbering, but ZFS OI Scrub is not yet implemented (there is a Jira ticket for this, and some work is underway there).</div>
<div id="m_-1434197144081090095AppleMailSignature"><br>
</div>
<div id="m_-1434197144081090095AppleMailSignature">Options include using zfs send/recv, as you were using, or just incrementally replacing the disks in the pool one at a time and letting them resilver to the SSDs (assuming they are larger than the HDDs they are replacing). </div>
<div id="m_-1434197144081090095AppleMailSignature"><br>
</div>
<div id="m_-1434197144081090095AppleMailSignature">I'm not sure why send/recv is so slow and exploding the metadata size, but it might relate to the ashift=12 on the target and ashift=9 on the source? This can be particularly bad with RAIDz compared to mirrors, since small blocks
(as typically used on the MDT) will always need to write 16KB instead of 8 or 12KB (with 2 or 3 mirrors). </div>
<div id="m_-1434197144081090095AppleMailSignature"><br>
Cheers, Andreas</div><div><div class="h5">
<div><br>
On Jul 22, 2017, at 07:48, Raj <<a href="mailto:rajgautam@gmail.com" target="_blank">rajgautam@gmail.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">Stu,
<div>Is there a reason why you picked Raidz 3 rather than 4 way mirror across 4 disks?</div>
<div>Raidz 3 parity calculation might take more cpu resources rather than mirroring across disks but also the latency may be higher in mirroring to sync across all the disks. Wondering if you did some testing before deciding it.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Fri, Jul 21, 2017 at 12:27 AM Stu Midgley <<a href="mailto:sdm900@gmail.com" target="_blank">sdm900@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_default" style="font-family:monospace,monospace">we have been happily using 2.9.52+0.7.0-rc3 for a while now.</div>
<div class="gmail_default" style="font-family:monospace,monospace"><br>
</div>
<div class="gmail_default" style="font-family:monospace,monospace">The MDT is a raidz3 across 4 disks.</div>
</div>
<div class="gmail_extra"></div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jul 21, 2017 at 1:19 PM, Isaac Huang <span dir="ltr">
<<a href="mailto:he.huang@intel.com" target="_blank">he.huang@intel.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span>On Fri, Jul 21, 2017 at 12:54:15PM +0800, Stu Midgley wrote:<br>
> Afternoon<br>
><br>
> I have an MDS running on spinning media and wish to migrate it to SSD's.<br>
><br>
> Lustre 2.9.52<br>
> ZFS 0.7.0-rc3<br>
<br>
</span>This may not be a stable combination - I don't think Lustre officially<br>
supports 0.7.0-rc yet. Plus, there's a recent Lustre osd-zfs bug and<br>
its fix hasn't been back ported to 2.9 yet (to the best of my knowledge):<br>
<a href="https://jira.hpdd.intel.com/browse/LU-9305" rel="noreferrer" target="_blank">https://jira.hpdd.intel.com/<wbr>browse/LU-9305</a><br>
<span><br>
> How do I do it?<br>
<br>
</span>Depends on how you've configured the MDT pool. If the disks are<br>
mirrored or just plan disks without any redundancy (i.e. not RAIDz),<br>
you can simply attach the SSDs to the hard drives to form or extend<br>
mirrors and then detach the hard drives - see zpool attach/detach.<br>
<span class="m_-1434197144081090095m_-7255051117978200532HOEnZb"><font color="#888888"><br>
-Isaac<br>
</font></span></blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
</div>
<div class="gmail_extra">-- <br>
<div class="m_-1434197144081090095m_-7255051117978200532gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div class="m_-1434197144081090095m_-7255051117978200532gmail_signature" data-smartmail="gmail_signature">
<font face="monospace, monospace">Dr Stuart Midgley<br>
<a href="mailto:sdm900@gmail.com" target="_blank">sdm900@gmail.com</a></font></div>
</div>
</div>
</div>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
</blockquote>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>______________________________<wbr>_________________</span><br>
<span>lustre-discuss mailing list</span><br>
<span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.<wbr>org</a></span><br>
<span><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a></span><br>
</div>
</blockquote>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><font face="monospace, monospace">Dr Stuart Midgley<br><a href="mailto:sdm900@gmail.com" target="_blank">sdm900@gmail.com</a></font></div></div></div>
</div>