[lustre-discuss] backup zfs MDT or migrate from ZFS back to ldiskfs

[Dilger, Andreas]

Using rsync or tar to backup/restore a ZFS MDT is not supported, because this changes the dnode numbering, but ZFS OI Scrub is not yet implemented (there is a Jira ticket for this, and some work is underway there).

Options include using zfs send/recv, as you were using, or just incrementally replacing the disks in the pool one at a time and letting them resilver to the SSDs (assuming they are larger than the HDDs they are replacing).

I'm not sure why send/recv is so slow and exploding the metadata size, but it might relate to the ashift=12 on the target and ashift=9 on the source?  This can be particularly bad with RAIDz compared to mirrors, since small blocks (as typically used on the MDT) will always need to write 16KB instead of 8 or 12KB (with 2 or 3 mirrors).

Cheers, Andreas

On Jul 22, 2017, at 07:48, Raj <rajgautam at gmail.com<mailto:rajgautam at gmail.com>> wrote:

Stu,
Is there a reason why you picked Raidz 3 rather than 4 way mirror across 4 disks?
Raidz 3 parity calculation might take more cpu resources rather than mirroring across disks but also the latency may be higher in mirroring to sync across all the disks. Wondering if you did some testing before deciding it.

On Fri, Jul 21, 2017 at 12:27 AM Stu Midgley <sdm900 at gmail.com<mailto:sdm900 at gmail.com>> wrote:
we have been happily using 2.9.52+0.7.0-rc3 for a while now.

The MDT is a raidz3 across 4 disks.

On Fri, Jul 21, 2017 at 1:19 PM, Isaac Huang <he.huang at intel.com<mailto:he.huang at intel.com>> wrote:
On Fri, Jul 21, 2017 at 12:54:15PM +0800, Stu Midgley wrote:
> Afternoon
>
> I have an MDS running on spinning media and wish to migrate it to SSD's.
>
>     Lustre 2.9.52
>     ZFS 0.7.0-rc3

This may not be a stable combination - I don't think Lustre officially
supports 0.7.0-rc yet. Plus, there's a recent Lustre osd-zfs bug and
its fix hasn't been back ported to 2.9 yet (to the best of my knowledge):
https://jira.hpdd.intel.com/browse/LU-9305

> How do I do it?

Depends on how you've configured the MDT pool. If the disks are
mirrored or just plan disks without any redundancy (i.e. not RAIDz),
you can simply attach the SSDs to the hard drives to form or extend
mirrors and then detach the hard drives - see zpool attach/detach.

-Isaac



--
Dr Stuart Midgley
sdm900 at gmail.com<mailto:sdm900 at gmail.com>
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170722/b822fb29/attachment.htm>