[lustre-discuss] No ZFS compression on OST's when using PFL
Nehring, Shane R [ITS]
snehring at iastate.edu
Fri Aug 1 08:48:06 PDT 2025
I'm running Lustre 2.15.6 on RHEL 8.10 with ZFS 2.2.7 (lz4 compression
on the ost datasets) on my servers (clients are RHEL 9.4 with lustre
2.15.6) with a comparatively simple PFL applied. I'm in the middle of
migrating data with lfs_migrate from older osts onto these newer osts
and zfs compression seems to still be working. My PFL doesn't use any
ext-size stuff it's just -E 100G -c 1 -S 1M -E -1 -c 2 -S 1M, that and
zstd vs lz4 are the only meaningful differences I see between our
setups. Perhaps either of those are causing the issue?
Shane
On Fri, 2025-08-01 at 12:33 +0000, BALVERS Martin via lustre-discuss
wrote:
>
> I ended up installing lustre 2.15.1 on the new servers. Now
> compression works as expected when PFL is enabled.
>
>
> What I have tested so far is:
> AlmaLinux 9.4 - Lustre 2.16.1 - ZFS 2.1.16 - No compression with PFL
> AlmaLinux 8.10 - Lustre 2.15.7 - ZFS 2.2.8 - No compression with PFL
> AlmaLinux 8.10 - Lustre 2.15.5 - ZFS 2.1.15 - No compression with PFL
> AlmaLinux 8.6 - Lustre 2.15.1 - ZFS 2.1.5 - PFL + compression works
> as expected
>
>
> I tried installing 2.15.2, 2.15.3 and 2.15.4 but that failed with
> ksym errors.
>
>
> It seems that somewhere between lustre 2.15.1 and 2.15.5 the PFL +
> compression breaks.
>
>
> gr,
> Martin Balvers
>
>
> AlmaLinux 8.6 - Lustre 2.15.1 - ZFS 2.1.5
>
>
> [root at microbio pfl]# ls -lh *
> -rw-r--r-- 1 root root 6.3G Jul 1 13:28 test1.txt
> -rw-r--r-- 1 root root 6.3G Jul 23 09:34 test2.txt
> -rw-r--r-- 1 root root 6.3G Jul 23 09:35 test3.txt
>
>
> LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED
> LUSED
> lustre-MDT0000 zstd-fast mdt/mdt 1.51x 1.51x 8.23M
> 6.80M
> lustre-OST0000 zstd lustre/ost 4.60x 4.60x 1.37G
> 6.29G
> lustre-OST0001 zstd lustre/ost 4.60x 4.60x 1.37G
> 6.29G
> lustre-OST0002 zstd lustre/ost 4.60x 4.60x 1.37G
> 6.29G
>
>
> [root at microbio pfl]# lfs getstripe test1.txt
> test1.txt
> lcm_layout_gen: 40
> lcm_mirror_count: 1
> lcm_entry_count: 5
> lcme_id: 1
> lcme_mirror_id: 0
> lcme_flags: init
> lcme_extent.e_start: 0
> lcme_extent.e_end: 1048576
> lmm_stripe_count: 0
> lmm_stripe_size: 1048576
> lmm_pattern: mdt
> lmm_layout_gen: 0
> lmm_stripe_offset: 0
>
>
> lcme_id: 2
> lcme_mirror_id: 0
> lcme_flags: init
> lcme_extent.e_start: 1048576
> lcme_extent.e_end: 134217728
> lmm_stripe_count: 1
> lmm_stripe_size: 4194304
> lmm_pattern: raid0
> lmm_layout_gen: 0
> lmm_stripe_offset: 0
> lmm_objects:
> - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6e:0x0] }
>
>
> lcme_id: 3
> lcme_mirror_id: 0
> lcme_flags: init
> lcme_extent.e_start: 134217728
> lcme_extent.e_end: 2147483648
> lmm_stripe_count: 2
> lmm_stripe_size: 4194304
> lmm_pattern: raid0
> lmm_layout_gen: 0
> lmm_stripe_offset: 2
> lmm_objects:
> - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x6e:0x0] }
> - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x6f:0x0] }
>
>
> lcme_id: 5
> lcme_mirror_id: 0
> lcme_flags: init
> lcme_extent.e_start: 2147483648
> lcme_extent.e_end: 6979321856
> lmm_stripe_count: 3
> lmm_stripe_size: 4194304
> lmm_pattern: raid0
> lmm_layout_gen: 0
> lmm_stripe_offset: 2
> lmm_objects:
> - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x6f:0x0] }
> - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x70:0x0] }
> - 2: { l_ost_idx: 0, l_fid: [0x100000000:0x6f:0x0] }
>
>
> lcme_id: 6
> lcme_mirror_id: 0
> lcme_flags: extension
> lcme_extent.e_start: 6979321856
> lcme_extent.e_end: EOF
> lmm_stripe_count: 0
> lmm_extension_size: 268435456
> lmm_pattern: raid0
> lmm_layout_gen: 0
> lmm_stripe_offset: -1
>
>
>
>
> From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on
> behalf of BALVERS Martin via lustre-discuss
> <lustre-discuss at lists.lustre.org>
> Sent: Tuesday, July 29, 2025 09:19
> To: Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]
> <darby.vicker-1 at nasa.gov>; lustre-discuss
> <lustre-discuss at lists.lustre.org>
> Subject: Re: [lustre-discuss] [EXTERNAL] [BULK] No ZFS compression on
> OST's when using PFL
>
>
> WARNING - EXTERNAL SENDER - BE CYBERSAFE
>
>
> Hi,
>
>
> My initial copy from old lustre to new lustre was with rsync, using
> this syntax:
> SOURCE="/lustre/workgroups" DEST="${SOURCE#/lustre/}" && find
> "$SOURCE" -maxdepth 1 -mindepth 1 -type d | xargs -t -n1 -P6 -I%
> rsync -aPA --delete-during % "/mnt/lustre_new/$DEST"
>
>
> I do indeed need to copy the linux ACL's
>
>
> But when debugging the issue, I used files without any extended
> attributes, from other filesystems than lustre.
> Just enabling DoM with "lfs setstripe -E 1M -L mdt -E -1 -c 1
> /mnt/lustre_new/dom" will disable compression on the OST's.
> (dom_stripesize is set to 1048576)
>
>
> I'm thinking about trying 2.15.1 on the new servers...
>
>
> Regards,
> Martin Balvers
>
>
>
>
>
> From: Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]
> <darby.vicker-1 at nasa.gov>
> Sent: Monday, July 28, 2025 16:57
> To: BALVERS Martin <Martin.BALVERS at danone.com>; lustre-discuss
> <lustre-discuss at lists.lustre.org>
> Subject: Re: [EXTERNAL] [BULK] [lustre-discuss] No ZFS compression on
> OST's when using PFL
>
>
> WARNING - EXTERNAL SENDER - BE CYBERSAFE
>
>
>
> I’m not sure if this is related to your problem but you need to be
> careful when copying data between two lustre filesystems. If you use
> something that copies the extended attributes (e.g. rsync -X), you
> can copy the lustre xattrs, which is probably not what you want.
> This will override the PFL you have on the new lustre FS. You can
> either not copy the extended attributes (which isn’t a great option
> if you are using ACL’s or other xattrs) or use a lustre-aware tool
> like mpifileutils.
>
>
>
>
>
>
>
> From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on
> behalf of BALVERS Martin via lustre-discuss
> <lustre-discuss at lists.lustre.org>
> Date: Friday, July 25, 2025 at 7:07 AM
> To: lustre-discuss <lustre-discuss at lists.lustre.org>
> Subject: [EXTERNAL] [BULK] [lustre-discuss] No ZFS compression on
> OST's when using PFL
> CAUTION: This email originated from outside of NASA. Please take
> care when clicking links or opening attachments. Use the "Report
> Message" button to report suspicious messages to the NASA SOC.
>
>
>
>
>
>
>
> Hi,
>
>
>
> I am using a lustre 2.15.1 cluster with PFL. I'm using the following
> layout:
>
> lfs setstripe -E 1M -L mdt -E 128M -c 1 -S 4M -E 2G -c 2 -z 64M -E -1
> -c -1 -z 256M /lustre
>
>
>
> OST's have zpools with zstd compression enabled.
>
> This has worked fine for me for many years.
>
>
>
> I now have new servers and installed 2.16.1 on them. I used the same
> PFL layout, and after some benchmarking I proceeded to copy data from
> the old lustre to the new.
>
> After a while I noticed that it was filling up faster that expected,
> and say that the compression ratio on the zpools was 1.
>
>
>
> Compression on the zpools is enabled, it should have worked as far as
> I can tell.
>
> When I disable PFL, compression works as expected.
>
>
>
> I have attached the some results in a txt file.
>
>
>
> I have also tried 2.15.7 with the same results. Enabling PFL, or just
> DoM disables compression on the OST's.
>
>
>
> Am I missing something obvious here?
>
>
>
> Regards,
>
> Martin Balvers
>
> Ce message électronique et tous les fichiers attachés qu'il contient
> sont confidentiels et destinés exclusivement à l'usage de la personne
> à laquelle ils sont adressés. Si vous avez reçu ce message par
> erreur, merci de le retourner à son émetteur. Les idées et opinions
> présentées dans ce message sont celles de son auteur, et ne
> représentent pas nécessairement celles de DANONE ou d'une quelconque
> de ses filiales. La publication, l'usage, la distribution,
> l'impression ou la copie non autorisée de ce message et des
> attachements qu'il contient sont strictement interdits.
>
> This e-mail and any files transmitted with it are confidential and
> intended solely for the use of the individual to whom it is
> addressed. If you have received this email in error please send it
> back to the person that sent it to you. Any views or opinions
> presented are solely those of its author and do not necessarily
> represent those of DANONE or any of its subsidiary companies.
> Unauthorized publication, use, dissemination, forwarding, printing or
> copying of this email and its associated attachments is strictly
> prohibited.
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6084 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20250801/c442b0e6/attachment-0001.p7s>
More information about the lustre-discuss
mailing list