[lustre-discuss] No ZFS compression on OST's when using PFL

Fri Sep 12 04:53:32 PDT 2025

Hi Andreas,

I have done some more testing in a virtualized environment. I took AlmaLinux 8.10, fully updated (kernel 4.18.0-553.74.1.el8_10.x86_64) and build zfs 2.1.16 and lustre 2.15.6 + 2.15.7
For lustre 2.15.6 compression works as expected. (see attached file)
For lustre 2.15.7, depending on the PFL layout, there is some compression, but less than with 2.15.6. Or there is no compression on the OST's at all.
In the test where there is some compression, the PFL layout is simple, few components and no SEL.

I have also tested lustre 2.15.5 on AlmaLinux 8.6, self compiled, and that also worked as it should.
In my previous test with AlmaLinux 8.10 - Lustre 2.15.5 - ZFS 2.1.15, compression did not work. That was on bare metal, where zfs was installed from the zfsonlinux repo. Lustre was installed from the whamcloud repo.

No idea what can cause this, but there seems to be something not ok with lustre 2.15.7 in combination with zfs compression.

Best regards,
Martin Balvers

Danone Global Research & Innovation Center B.V.

Uppsalalaan 12, 3584CT Utrecht

________________________________
From: Andreas Dilger <adilger at ddn.com>
Sent: Friday, August 01, 2025 22:58
To: BALVERS Martin <Martin.BALVERS at danone.com>
Cc: lustre-discuss <lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] No ZFS compression on OST's when using PFL

WARNING - EXTERNAL SENDER - BE CYBERSAFE

You are changing a number of different things with each of your tests, not just Lustre.  It could be ZFS 2.1.5->2.1.15 or Alma 8.6->8.10 is also causing issues...  Hard to say for sure without apples-to-apples comparisons.

It should be possible to build Lustre 2.15.5 on Alma 8.6 + ZFS 2.1.5 to help isolate the issue.

There was LU-19193<https://urldefense.com/v3/__https://jira.whamcloud.com/browse/LU-19193__;!!OUGTln_Lrg!Ugs-QNv1jVaQ0FZzMnQ9mLWiu0QW-B3iSDiiXoIU2XwGrg2ab0Uh-J-krG6w9pYI3ffwwwLgjxtjhLeNfA$> filed recently about ZFS and blocksize detection, which might hurt ZFS compression, but that was never landed on 2.15.

Cheers, Andreas

On Aug 1, 2025, at 06:33, BALVERS Martin via lustre-discuss <lustre-discuss at lists.lustre.org> wrote:

I ended up installing lustre 2.15.1 on the new servers. Now compression works as expected when PFL is enabled.

What I have tested so far is:
AlmaLinux 9.4 - Lustre 2.16.1 - ZFS 2.1.16 - No compression with PFL
AlmaLinux 8.10 - Lustre 2.15.7 - ZFS 2.2.8 - No compression with PFL
AlmaLinux 8.10 - Lustre 2.15.5 - ZFS 2.1.15 - No compression with PFL
AlmaLinux 8.6 - Lustre 2.15.1 - ZFS 2.1.5 - PFL + compression works as expected

I tried installing 2.15.2, 2.15.3 and 2.15.4 but that failed with ksym errors.

It seems that somewhere between lustre 2.15.1 and 2.15.5 the PFL + compression breaks.

gr,
Martin Balvers

Cheers, Andreas
—
Andreas Dilger
Lustre Principal Architect
Whamcloud/DDN

Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et des attachements qu'il contient sont strictement interdits. 

This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20250912/1e60e7d1/attachment-0001.htm>
-------------- next part --------------
## Test data
# ls -gGh
total 9.4G
-rw-rwxr-- 1 143M Oct 24  2024 file1.txt
-rw-rwxr-- 1 162M Oct 25  2024 file2.txt
-rw-rwxr-- 1 166M Oct 25  2024 file3.txt
-rw-rwxr-- 1 122M Oct 25  2024 file4.txt
-rwxr-xr-x 1 8.3G Mar 20  2024 file5.fastq
-rw-r--r-- 1 197M Oct 24  2024 file6.fastq
-rw-r--r-- 1 304M Sep 11 15:28 file7.txt

# Lustre 2.15.6, zfs 2.1.16, AlamLinux 8.10, 4.18.0-553.74.1.el8_10.x86_64

lfs setstripe -E 1M -L mdt -E -1 -c -1 -S 4M /mnt/test
LUSTRE:SVNAME   COMPRESS        NAME         RATIO  REFRATIO   USED  LUSED
lustre-MDT0000  zstd-fast       mdt/mdt      1.84x     1.84x  9.59M  10.8M
lustre-OST0000  zstd            lustre/ost1  5.61x     5.61x   572M  3.11G
lustre-OST0001  zstd            lustre/ost1  5.62x     5.62x   571M  3.11G
lustre-OST0002  zstd            lustre/ost1  5.61x     5.61x   571M  3.11G

lfs setstripe -E 1M -L mdt -E 128M -c 1 -S 4M -E 2G -c 2 -z 64M -E -1 -c -1 -z 256M /mnt/test
LUSTRE:SVNAME   COMPRESS        NAME         RATIO  REFRATIO   USED  LUSED
lustre-MDT0000  zstd-fast       mdt/mdt      1.84x     1.84x  9.59M  10.8M
lustre-OST0000  zstd            lustre/ost1  5.72x     5.72x   574M  3.19G
lustre-OST0001  zstd            lustre/ost1  5.72x     5.72x   654M  3.64G
lustre-OST0002  zstd            lustre/ost1  5.34x     5.34x   485M  2.51G

# Lustre 2.15.7, zfs 2.1.16, AlamLinux 8.10, 4.18.0-553.74.1.el8_10.x86_64

lfs setstripe -E 1M -L mdt -E -1 -c -1 -S 4M /mnt/test
LUSTRE:SVNAME   COMPRESS        NAME         RATIO  REFRATIO   USED  LUSED
lustre-MDT0000  zstd-fast       mdt/mdt      1.84x     1.84x  9.61M  10.8M
lustre-OST0000  zstd            lustre/ost1  4.56x     4.56x   704M  3.12G
lustre-OST0001  zstd            lustre/ost1  1.06x     1.06x  2.97G  3.14G
lustre-OST0002  zstd            lustre/ost1  4.91x     4.91x   653M  3.11G

lfs setstripe -E 1M -L mdt -E 128M -c 1 -S 4M -E 2G -c 2 -z 64M -E -1 -c -1 -z 256M /mnt/test
LUSTRE:SVNAME   COMPRESS        NAME         RATIO  REFRATIO   USED  LUSED
lustre-MDT0000  zstd-fast       mdt/mdt      1.84x     1.84x  9.62M  10.8M
lustre-OST0000  zstd            lustre/ost1  1.00x     1.00x  3.26G  3.22G
lustre-OST0001  zstd            lustre/ost1  1.00x     1.00x  3.71G  3.67G
lustre-OST0002  zstd            lustre/ost1  1.00x     1.00x  2.56G  2.53G