[lustre-discuss] zpool features, recordsize, post-upgrade?

[Andreas Dilger]

On Oct 9, 2018, at 23:16, Jongwoo Han <jongwoohan at gmail.com> wrote:
> 
> Hi Marion,
> 
> Enabling PFL and relocating file will double space consumption, so safe way is not to enable new features. Among other things, compression may be a good idea to try but remember it will increase OSS CPU utilization.

Just to clarify, PFL (progressive file layout) does not increase space consumption.  You are probably mixing this up with FLR (file level redundancy)?  

In any case, migration will not double space consumption either, only for a short term while the file is being migrated to new OSTs will there be an increase in space usage.

However, if you have snapshots of the filesystem then the old snapshots will consume space from the old file copies that will not be released until the snapshots are removed.  In that case, the migration would probably not be a good idea.

On Tue, Oct 9, 2018 at 9:00 AM Marion Hakanson <hakansom at ohsu.edu> wrote:
> Greetings,
> 
> We're upgrading our Lustre servers from CentOS-6 with IEEL-2.3, to
> CentOS-7.5 with Lustre community release 2.10.5.  The OSTs are all
> on the ZFS backend, originally built on ZoL-0.6.5.
> 
> The new 2.10.5 servers come with ZoL-0.7.9, which of course has some
> new zpool features available.
> 
> Once we've upgraded, should we enable one or more of the new features
> in our existing OST zpools?  E.g. large_blocks?  And then update the
> ZFS recordsize to whatever a newly-created Lustre 2.10.5 OST would use?

Like all new features for Lustre and ZFS, it is recommended not to enable them after the upgrade until you are sure that the new Lustre/ZFS version is working well.  Otherwise, if you have problems after enabling the new feature you may not be able to downgrade to the old version.

The ZFS recordsize defaults to 1MB on new OST filesystems, and can be enabled on existing OSTs as well.  I would recommend against changing the recordsize on MDT filesystems.  This will improve performance for large files, if your filesystem typically has large and sequentially read/written files.  If you do a lot of small and random IO, it may not be the best choice.

> I do realize the new recordize setting wouldn't affect existing files,
> but we're also looking at enabling PFL and doing a global "lfs migrate"
> operation to re-layout everything for better performance.

The benefits of doing a full-filesystem migration depend on your configuration.  If the files in the filesystem change fairly regularly, then there is unlikely to be a large benefit from doing this since the new files will get the large record size and the old files will eventually be removed.

If the files are kept for a long time, and you do not keep long-term snapshots around, then doing the migration might make sense.  It could be run as a background task on old files in the filesystem (which are unlikely to be used/modified).  I agree that enabling "light" compression can also be useful (e.g. lz4), if you have enough spare CPU cycles, files are not modified very often, and are compressible (otherwise it just adds overhead).  In many systems compression can reduce space usage by 40% or more, and in some cases even improve read performance if the system is limited by disk bandwidth.

Cheers, Andreas
---
Andreas Dilger
Principal Lustre Architect
Whamcloud