[lustre-discuss] No ZFS compression on OST's when using PFL
BALVERS Martin
Martin.BALVERS at danone.com
Fri Jul 25 06:00:00 PDT 2025
Hi,
I am using a lustre 2.15.1 cluster with PFL. I'm using the following layout:
lfs setstripe -E 1M -L mdt -E 128M -c 1 -S 4M -E 2G -c 2 -z 64M -E -1 -c -1 -z 256M /lustre
OST's have zpools with zstd compression enabled.
This has worked fine for me for many years.
I now have new servers and installed 2.16.1 on them. I used the same PFL layout, and after some benchmarking I proceeded to copy data from the old lustre to the new.
After a while I noticed that it was filling up faster that expected, and say that the compression ratio on the zpools was 1.
Compression on the zpools is enabled, it should have worked as far as I can tell.
When I disable PFL, compression works as expected.
I have attached the some results in a txt file.
I have also tried 2.15.7 with the same results. Enabling PFL, or just DoM disables compression on the OST's.
Am I missing something obvious here?
Regards,
Martin Balvers
Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et des attachements qu'il contient sont strictement interdits.
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20250725/580c49b9/attachment-0001.htm>
-------------- next part --------------
### No PFL
lfs setstripe -c 1 /mnt/lustre_new
## 1 file
# lfs getstripe test1.txt
test1.txt
lmm_stripe_count: 1
lmm_stripe_size: 4194304
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 1
obdidx objid objid group
1 24 0x18 0x280000400
# ssh 192.168.6.2 zfs list -o lustre:svname,compression,name,compressratio,refcompressratio,used,logicalused lustre/ost && ssh 192.168.6.3 zfs list -o lustre:svname,compression,name,compressratio,refcompressratio,used,logicalused lustre/ost && ssh 192.168.6.4 zfs list -o lustre:svname,compression,name,compressratio,refcompressratio,used,logicalused lustre/ost
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0000 zstd lustre/ost 1.00x 1.00x 8.32M 4.15M
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0001 zstd lustre/ost 4.60x 4.60x 1.37G 6.29G
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0002 zstd lustre/ost 1.00x 1.00x 8.32M 4.15M
## 2 files
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0000 zstd lustre/ost 1.00x 1.00x 8.32M 4.15M
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0001 zstd lustre/ost 4.60x 4.60x 1.37G 6.29G
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0002 zstd lustre/ost 4.60x 4.60x 1.37G 6.29G
## 3 files
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0000 zstd lustre/ost 4.60x 4.60x 1.37G 6.29G
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0001 zstd lustre/ost 4.60x 4.60x 1.37G 6.29G
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0002 zstd lustre/ost 4.60x 4.60x 1.37G 6.29G
# du -h *
1.4G test1.txt
1.4G test2.txt
1.4G test3.txt
# ls -lh *
-rw-rw-r-- 1 lohrjl users 6.3G Jul 1 13:28 test1.txt
-rw-r--r-- 1 root root 6.3G Jul 23 09:34 test2.txt
-rw-r--r-- 1 root root 6.3G Jul 23 09:35 test3.txt
### PFL
lfs setstripe -E 1M -L mdt -E 128M -c 1 -S 4M -E 2G -c 2 -z 64M -E -1 -c -1 -z 256M /mnt/lustre_new
## 1 file
# lfs getstripe test1.txt
test1.txt
lcm_layout_gen: 55
lcm_mirror_count: 1
lcm_entry_count: 5
lcme_id: 1
lcme_mirror_id: 0
lcme_flags: init
lcme_extent.e_start: 0
lcme_extent.e_end: 1048576
lmm_stripe_count: 0
lmm_stripe_size: 1048576
lmm_pattern: mdt
lmm_layout_gen: 0
lmm_stripe_offset: 0
lcme_id: 2
lcme_mirror_id: 0
lcme_flags: init
lcme_extent.e_start: 1048576
lcme_extent.e_end: 134217728
lmm_stripe_count: 1
lmm_stripe_size: 4194304
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 1
lmm_objects:
- 0: { l_ost_idx: 1, l_fid: [0x280000400:0x19:0x0] }
lcme_id: 3
lcme_mirror_id: 0
lcme_flags: init
lcme_extent.e_start: 134217728
lcme_extent.e_end: 2147483648
lmm_stripe_count: 2
lmm_stripe_size: 4194304
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 2
lmm_objects:
- 0: { l_ost_idx: 2, l_fid: [0x2c0000400:0x18:0x0] }
- 1: { l_ost_idx: 0, l_fid: [0x240000400:0x1c:0x0] }
lcme_id: 5
lcme_mirror_id: 0
lcme_flags: init
lcme_extent.e_start: 2147483648
lcme_extent.e_end: 6979321856
lmm_stripe_count: 3
lmm_stripe_size: 4194304
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 1
lmm_objects:
- 0: { l_ost_idx: 1, l_fid: [0x280000400:0x1a:0x0] }
- 1: { l_ost_idx: 2, l_fid: [0x2c0000400:0x19:0x0] }
- 2: { l_ost_idx: 0, l_fid: [0x240000400:0x1d:0x0] }
lcme_id: 6
lcme_mirror_id: 0
lcme_flags: extension
lcme_extent.e_start: 6979321856
lcme_extent.e_end: EOF
lmm_stripe_count: 0
lmm_extension_size: 268435456
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: -1
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0000 zstd lustre/ost 1.00x 1.00x 2.43G 2.40G
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0001 zstd lustre/ost 1.00x 1.00x 1.59G 1.57G
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0002 zstd lustre/ost 1.00x 1.00x 2.42G 2.39G
# du -h *
6.5G test1.txt
# ls -lh *
-rw-rw-r-- 1 lohrjl users 6.3G Jul 1 13:28 test1.txt
## 3 files
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0000 zstd lustre/ost 1.00x 1.00x 7.26G 7.18G
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0001 zstd lustre/ost 1.00x 1.00x 5.59G 5.53G
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-OST0002 zstd lustre/ost 1.00x 1.00x 6.43G 6.35G
# du -h *
6.5G test1.txt
6.5G test2.txt
6.5G test3.txt
# ls -lh *
-rw-rw-r-- 1 lohrjl users 6.3G Jul 1 13:28 test1.txt
-rw-r--r-- 1 root root 6.3G Jul 23 09:34 test2.txt
-rw-r--r-- 1 root root 6.3G Jul 23 09:35 test3.txt
More information about the lustre-discuss
mailing list