[lustre-discuss] No ZFS compression on OST's when using PFL

[BALVERS Martin]

Unfortunately, I don't have any more time to figure this out. I opted to just install lustre 2.15.1, because that is the one that seems to work for me.

I have attached my install procedure for lustre 2.15.7, is there something I should be doing differently?

Gr,
Martin Balvers

________________________________
From: Andreas Dilger <adilger at ddn.com>
Sent: Friday, August 01, 2025 22:58
To: BALVERS Martin <Martin.BALVERS at danone.com>
Cc: lustre-discuss <lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] No ZFS compression on OST's when using PFL

WARNING - EXTERNAL SENDER - BE CYBERSAFE

You are changing a number of different things with each of your tests, not just Lustre.  It could be ZFS 2.1.5->2.1.15 or Alma 8.6->8.10 is also causing issues...  Hard to say for sure without apples-to-apples comparisons.

It should be possible to build Lustre 2.15.5 on Alma 8.6 + ZFS 2.1.5 to help isolate the issue.

There was LU-19193<https://urldefense.com/v3/__https://jira.whamcloud.com/browse/LU-19193__;!!OUGTln_Lrg!Ugs-QNv1jVaQ0FZzMnQ9mLWiu0QW-B3iSDiiXoIU2XwGrg2ab0Uh-J-krG6w9pYI3ffwwwLgjxtjhLeNfA$> filed recently about ZFS and blocksize detection, which might hurt ZFS compression, but that was never landed on 2.15.

Cheers, Andreas

On Aug 1, 2025, at 06:33, BALVERS Martin via lustre-discuss <lustre-discuss at lists.lustre.org> wrote:

I ended up installing lustre 2.15.1 on the new servers. Now compression works as expected when PFL is enabled.

What I have tested so far is:
AlmaLinux 9.4 - Lustre 2.16.1 - ZFS 2.1.16 - No compression with PFL
AlmaLinux 8.10 - Lustre 2.15.7 - ZFS 2.2.8 - No compression with PFL
AlmaLinux 8.10 - Lustre 2.15.5 - ZFS 2.1.15 - No compression with PFL
AlmaLinux 8.6 - Lustre 2.15.1 - ZFS 2.1.5 - PFL + compression works as expected

I tried installing 2.15.2, 2.15.3 and 2.15.4 but that failed with ksym errors.

It seems that somewhere between lustre 2.15.1 and 2.15.5 the PFL + compression breaks.

gr,
Martin Balvers

Cheers, Andreas
—
Andreas Dilger
Lustre Principal Architect
Whamcloud/DDN




Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et des attachements qu'il contient sont strictement interdits. 

This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20250805/198dc7de/attachment.htm>
-------------- next part --------------
### SETUP NODES
# Install AlmaLinux 8.10 from minimal iso and configure networking etc.

# For any server, disable selinux!
nano /etc/selinux/config
SELINUX=disabled

dnf install epel-release
dnf install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm

# Disable [zfs] and enable [zfs-kmod] repos
nano /etc/yum.repos.d/zfs.repo

dnf install zfs

nano /etc/yum.repos.d/lustre.repo

[lustre-server]
name=lustre-server
baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.15.7/el8.10/server/
enabled=1
exclude=*debuginfo*
gpgcheck=0

# to allow port 988
firewall-cmd --zone=public --add-port 988/tcp --permanent && firewall-cmd --reload

# create zfs.conf file
# by default zfs takes max 50% memory, set zfs_arc_max to 80-90% of total memory (in bytes) Use 'free -b' tot find the total memory in bytes
nano /etc/modprobe.d/zfs.conf

### OSS
# Optimized ZFS settings for Lustre OST with HDD mirrors and 265GB RAM
options zfs zfs_arc_max=227633266688
options zfs zfs_arc_min=8589934592
options zfs zfs_prefetch_disable=1
options zfs metaslab_debug_unload=0
options zfs zfs_vdev_cache_size=0
options zfs zfs_read_chunk_size=1048576
options zfs zfs_vdev_async_write_min_active=10
options zfs zfs_vdev_async_write_max_active=30
options zfs zfs_vdev_async_write_active_min_dirty_percent=15
options zfs zfs_vdev_scheduler=deadline

reboot

dnf install lustre kmod-lustre kmod-lustre-osd-zfs

### setup OSS
# create the zfs pool
zpool create -o ashift=12 -f lustre -o cachefile=none -O acltype=posixacl -O compression=zstd -O xattr=sa -O canmount=off mirror /dev/sda /dev/sdb mirror /dev/sdc /dev/sdd mirror /dev/sde /dev/sdf mirror /dev/sdg /dev/sdh mirror /dev/sdi /dev/sdj mirror /dev/sdk /dev/sdl

# Create the lustre fs. Index needs to be unique for each OST
mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=<i> --mgsnode=192.168.6.1 at tcp lustre/ost

mkdir /mnt/ost

# mounting the pool starts lustre
mount -t lustre_tgt -v lustre/ost /mnt/ost

# Make sure compression is enabled:
zfs set compression=zstd lustre/ost

## From a client
lfs setstripe -E 1M -L mdt -E 128M -c 1 -S 4M -E 2G -c 2 -z 128M -E -1 -c -1 -z 256M /mnt/lustre_new