[lustre-discuss] No ZFS compression on OST's when using PFL
BALVERS Martin
Martin.BALVERS at danone.com
Tue Aug 5 00:21:36 PDT 2025
Hi Shane, thanks for your input.
I guess that with your PFL, for any file <100G it is essentially as if PLF is not enabled, so the whole file is written to a single OST.
In my testing, compression does work in that scenario.
I have DoM enabled, so the first 1M of every file is written to the MDT. In that case, compression is no longer applied on the OST's.
I have tested with just PFL, no DoM or SEL. My files were always larger that the first PFL layout segment, so OST striping was always triggered. In those cases OST compression did not work.
Maybe for files that fit the first PFL segment (write to a single OST) compression would have worked...
Gr,
Martin Balvers
________________________________________
From: Nehring, Shane R [ITS]
Sent: Friday, August 01, 2025 17:48
To: lustre-discuss at lists.lustre.org; BALVERS Martin
Subject: Re: [lustre-discuss] No ZFS compression on OST's when using PFL
I'm running Lustre 2.15.6 on RHEL 8.10 with ZFS 2.2.7 (lz4 compression
on the ost datasets) on my servers (clients are RHEL 9.4 with lustre
2.15.6) with a comparatively simple PFL applied. I'm in the middle of
migrating data with lfs_migrate from older osts onto these newer osts
and zfs compression seems to still be working. My PFL doesn't use any
ext-size stuff it's just -E 100G -c 1 -S 1M -E -1 -c 2 -S 1M, that and
zstd vs lz4 are the only meaningful differences I see between our
setups. Perhaps either of those are causing the issue?
Shane
On Fri, 2025-08-01 at 12:33 +0000, BALVERS Martin via lustre-discuss
wrote:
>
> I ended up installing lustre 2.15.1 on the new servers. Now
> compression works as expected when PFL is enabled.
>
>
> What I have tested so far is:
> AlmaLinux 9.4 - Lustre 2.16.1 - ZFS 2.1.16 - No compression with PFL
> AlmaLinux 8.10 - Lustre 2.15.7 - ZFS 2.2.8 - No compression with PFL
> AlmaLinux 8.10 - Lustre 2.15.5 - ZFS 2.1.15 - No compression with PFL
> AlmaLinux 8.6 - Lustre 2.15.1 - ZFS 2.1.5 - PFL + compression works
> as expected
>
>
> I tried installing 2.15.2, 2.15.3 and 2.15.4 but that failed with
> ksym errors.
>
>
> It seems that somewhere between lustre 2.15.1 and 2.15.5 the PFL +
> compression breaks.
>
>
> gr,
> Martin Balvers
>
>
> AlmaLinux 8.6 - Lustre 2.15.1 - ZFS 2.1.5
>
>
> [root at microbio pfl]# ls -lh *
> -rw-r--r-- 1 root root 6.3G Jul 1 13:28 test1.txt
> -rw-r--r-- 1 root root 6.3G Jul 23 09:34 test2.txt
> -rw-r--r-- 1 root root 6.3G Jul 23 09:35 test3.txt
>
>
> LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED
> LUSED
> lustre-MDT0000 zstd-fast mdt/mdt 1.51x 1.51x 8.23M
> 6.80M
> lustre-OST0000 zstd lustre/ost 4.60x 4.60x 1.37G
> 6.29G
> lustre-OST0001 zstd lustre/ost 4.60x 4.60x 1.37G
> 6.29G
> lustre-OST0002 zstd lustre/ost 4.60x 4.60x 1.37G
> 6.29G
>
>
> [root at microbio pfl]# lfs getstripe test1.txt
> test1.txt
> lcm_layout_gen: 40
> lcm_mirror_count: 1
> lcm_entry_count: 5
> lcme_id: 1
> lcme_mirror_id: 0
> lcme_flags: init
> lcme_extent.e_start: 0
> lcme_extent.e_end: 1048576
> lmm_stripe_count: 0
> lmm_stripe_size: 1048576
> lmm_pattern: mdt
> lmm_layout_gen: 0
> lmm_stripe_offset: 0
>
>
> lcme_id: 2
> lcme_mirror_id: 0
> lcme_flags: init
> lcme_extent.e_start: 1048576
> lcme_extent.e_end: 134217728
> lmm_stripe_count: 1
> lmm_stripe_size: 4194304
> lmm_pattern: raid0
> lmm_layout_gen: 0
> lmm_stripe_offset: 0
> lmm_objects:
> - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6e:0x0] }
>
>
> lcme_id: 3
> lcme_mirror_id: 0
> lcme_flags: init
> lcme_extent.e_start: 134217728
> lcme_extent.e_end: 2147483648
> lmm_stripe_count: 2
> lmm_stripe_size: 4194304
> lmm_pattern: raid0
> lmm_layout_gen: 0
> lmm_stripe_offset: 2
> lmm_objects:
> - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x6e:0x0] }
> - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x6f:0x0] }
>
>
> lcme_id: 5
> lcme_mirror_id: 0
> lcme_flags: init
> lcme_extent.e_start: 2147483648
> lcme_extent.e_end: 6979321856
> lmm_stripe_count: 3
> lmm_stripe_size: 4194304
> lmm_pattern: raid0
> lmm_layout_gen: 0
> lmm_stripe_offset: 2
> lmm_objects:
> - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x6f:0x0] }
> - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x70:0x0] }
> - 2: { l_ost_idx: 0, l_fid: [0x100000000:0x6f:0x0] }
>
>
> lcme_id: 6
> lcme_mirror_id: 0
> lcme_flags: extension
> lcme_extent.e_start: 6979321856
> lcme_extent.e_end: EOF
> lmm_stripe_count: 0
> lmm_extension_size: 268435456
> lmm_pattern: raid0
> lmm_layout_gen: 0
> lmm_stripe_offset: -1
>
>
>
>
> From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on
> behalf of BALVERS Martin via lustre-discuss
> <lustre-discuss at lists.lustre.org>
> Sent: Tuesday, July 29, 2025 09:19
> To: Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]
> <darby.vicker-1 at nasa.gov>; lustre-discuss
> <lustre-discuss at lists.lustre.org>
> Subject: Re: [lustre-discuss] [EXTERNAL] [BULK] No ZFS compression on
> OST's when using PFL
>
>
> WARNING - EXTERNAL SENDER - BE CYBERSAFE
>
>
> Hi,
>
>
> My initial copy from old lustre to new lustre was with rsync, using
> this syntax:
> SOURCE="/lustre/workgroups" DEST="${SOURCE#/lustre/}" && find
> "$SOURCE" -maxdepth 1 -mindepth 1 -type d | xargs -t -n1 -P6 -I%
> rsync -aPA --delete-during % "/mnt/lustre_new/$DEST"
>
>
> I do indeed need to copy the linux ACL's
>
>
> But when debugging the issue, I used files without any extended
> attributes, from other filesystems than lustre.
> Just enabling DoM with "lfs setstripe -E 1M -L mdt -E -1 -c 1
> /mnt/lustre_new/dom" will disable compression on the OST's.
> (dom_stripesize is set to 1048576)
>
>
> I'm thinking about trying 2.15.1 on the new servers...
>
>
> Regards,
> Martin Balvers
>
>
>
>
>
> From: Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]
> <darby.vicker-1 at nasa.gov>
> Sent: Monday, July 28, 2025 16:57
> To: BALVERS Martin <Martin.BALVERS at danone.com>; lustre-discuss
> <lustre-discuss at lists.lustre.org>
> Subject: Re: [EXTERNAL] [BULK] [lustre-discuss] No ZFS compression on
> OST's when using PFL
>
>
> WARNING - EXTERNAL SENDER - BE CYBERSAFE
>
>
>
> I’m not sure if this is related to your problem but you need to be
> careful when copying data between two lustre filesystems. If you use
> something that copies the extended attributes (e.g. rsync -X), you
> can copy the lustre xattrs, which is probably not what you want.
> This will override the PFL you have on the new lustre FS. You can
> either not copy the extended attributes (which isn’t a great option
> if you are using ACL’s or other xattrs) or use a lustre-aware tool
> like mpifileutils.
>
>
>
>
>
>
>
> From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on
> behalf of BALVERS Martin via lustre-discuss
> <lustre-discuss at lists.lustre.org>
> Date: Friday, July 25, 2025 at 7:07 AM
> To: lustre-discuss <lustre-discuss at lists.lustre.org>
> Subject: [EXTERNAL] [BULK] [lustre-discuss] No ZFS compression on
> OST's when using PFL
> CAUTION: This email originated from outside of NASA. Please take
> care when clicking links or opening attachments. Use the "Report
> Message" button to report suspicious messages to the NASA SOC.
>
>
>
>
>
>
>
> Hi,
>
>
>
> I am using a lustre 2.15.1 cluster with PFL. I'm using the following
> layout:
>
> lfs setstripe -E 1M -L mdt -E 128M -c 1 -S 4M -E 2G -c 2 -z 64M -E -1
> -c -1 -z 256M /lustre
>
>
>
> OST's have zpools with zstd compression enabled.
>
> This has worked fine for me for many years.
>
>
>
> I now have new servers and installed 2.16.1 on them. I used the same
> PFL layout, and after some benchmarking I proceeded to copy data from
> the old lustre to the new.
>
> After a while I noticed that it was filling up faster that expected,
> and say that the compression ratio on the zpools was 1.
>
>
>
> Compression on the zpools is enabled, it should have worked as far as
> I can tell.
>
> When I disable PFL, compression works as expected.
>
>
>
> I have attached the some results in a txt file.
>
>
>
> I have also tried 2.15.7 with the same results. Enabling PFL, or just
> DoM disables compression on the OST's.
>
>
>
> Am I missing something obvious here?
>
>
>
> Regards,
>
> Martin Balvers
>
> Ce message électronique et tous les fichiers attachés qu'il contient
> sont confidentiels et destinés exclusivement à l'usage de la personne
> à laquelle ils sont adressés. Si vous avez reçu ce message par
> erreur, merci de le retourner à son émetteur. Les idées et opinions
> présentées dans ce message sont celles de son auteur, et ne
> représentent pas nécessairement celles de DANONE ou d'une quelconque
> de ses filiales. La publication, l'usage, la distribution,
> l'impression ou la copie non autorisée de ce message et des
> attachements qu'il contient sont strictement interdits.
>
> This e-mail and any files transmitted with it are confidential and
> intended solely for the use of the individual to whom it is
> addressed. If you have received this email in error please send it
> back to the person that sent it to you. Any views or opinions
> presented are solely those of its author and do not necessarily
> represent those of DANONE or any of its subsidiary companies.
> Unauthorized publication, use, dissemination, forwarding, printing or
> copying of this email and its associated attachments is strictly
> prohibited.
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!OUGTln_Lrg!W-Co5GVczTbSk5gOvGpgHdUk92SAUloyfkCmGpwtjDEU92glalEiZgA_J3FkiBQIfevF30vATNCF6yUQ0wEbY4o$
Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et des attachements qu'il contient sont strictement interdits.
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.
More information about the lustre-discuss
mailing list