[lustre-discuss] The pre-allocation and parallel offset writing lead to massive ZFS space waste
Homer Li
01jay.ly at gmail.com
Tue Jun 23 07:40:38 UTC 2026
Hi David
Thanks.
My test environment is based on Lustre 2.15.8 and OpenZFS 2.3.2-1. In
LU-19193, Alex mentioned, "we noticed that disabling PFL also
circumvented the problem."
After reviewing the rclone code, I easily reproduced the issue without
any parallel operations or random writes, using simply:
dd if=/dev/urandom of=test_1 bs=1M seek=1 count=555
This issue is triggered by any seek operation, regardless of whether
the seek offset aligns with the ZFS recordsize.
Therefore, I believe this appears to be unrelated to LU-19193,
although both are issues stemming from osd_zfs writes. That said, I am
not entirely clear yet whether they are linked at a micro-level.
However, if a massive waste of ZFS space and cache can be triggered by
any arbitrary seek—even an aligned one—it strongly suggests a flaw in
the osd_zfs implementation. A correlation in their underlying
behaviors might still exist.
OSS$ ls -lhsi ./O/300000400/d2/610
771 818M -rw-rw-rw- 1 root root 556M Jun 23 15:04 ./O/300000400/d2/610
OSS$ zdb -ddddd test_ost1/test_ost1 771
Object lvl iblk dblk dsize dnsize lsize %full type
771 3 128K 4K 818M 512 556M 99.82 ZFS plain file
dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED SPILL_BLKPTR
dnode maxblkid: 142335
path /O/300000400/d2/610
Spill block: 0:55d10b000:3000 200L/200P F=1 B=18365/18365
cksum=00000004e33f9590:000001f55e95bc2a:000067415ce5f733:000e8bf71249da1b
Indirect blocks:
0 L2 0:55d129000:6000 20000L/2000P F=142080
B=18365/18365 cksum=000002e732cdb1dd:000c6d79cf6b7e09:20c64bd91e328904:dccb8db21d625648
0 L1 0:408877000:12000 20000L/8000P F=768
B=18364/18364 cksum=00000ba8006726b4:00c6dfeef5929795:5a08d96ddbf7d946:11860a2720229dc7
100000 L0 0:45be3c000:3000 1000L/1000P F=1 B=18364/18364
cksum=000001fd2b012e1d:0003fb90edce4119:0554dfaadb8e004c:5dc10c44c4295a16
101000 L0 0:45be3f000:3000 1000L/1000P F=1 B=18364/18364
cksum=00000200d5999b15:0003fff5e3eb0af4:05504ba2292e3ffa:51549dfa08faf869
102000 L0 0:45be45000:3000 1000L/1000P F=1 B=18364/18364
cksum=0000020f1f369d30:00042c64d9d74c88:05a55f5d8b6c73ae:b6784f0e99facfb2
103000 L0 0:45be42000:3000 1000L/1000P F=1 B=18364/18364
cksum=000001fe8650df2c:0003fa93d62eba6f:0553b01c9404084b:5c3180d7716ae57e
104000 L0 0:45be48000:3000 1000L/1000P F=1 B=18364/18364
cksum=0000020ff8d32130:0004165fcd65bf45:056ae4c5dc462c0d:6685186e3e73be3a
.......
22bfc000 L0 0:55d0ff000:3000 1000L/1000P F=1 B=18365/18365
cksum=000001fa32329f80:0003ee608baf274b:053db2fc3f83dd58:3d9703d9ca7040f2
22bfd000 L0 0:55d102000:3000 1000L/1000P F=1 B=18365/18365
cksum=000001ede321b4ce:0003e1cb966ee514:05399c6d901e2a82:4747bec737a9960a
22bfe000 L0 0:55d105000:3000 1000L/1000P F=1 B=18365/18365
cksum=000001feb221eba1:0003fa18810584d9:05520f713c6d460f:57db8d048aaea8ec
22bff000 L0 0:55d108000:3000 1000L/1000P F=1 B=18365/18365
cksum=0000020043cd43a9:0003fcda8fa1b211:055aac69168825c2:65d26516ef5bdcf0
segment [0000000000100000, 0000000022c00000) size 555M
OSS$ zfs get recordsize
NAME PROPERTY VALUE SOURCE
test_ost1 recordsize 128K default
test_ost1/test_ost1 recordsize 1M local
client$ lfs getstripe -v file1/test_file file2/test_file test_1
file1/test_file
lmm_magic: 0x0BD10BD0
lmm_seq: 0x20000040f
lmm_object_id: 0x2
lmm_fid: [0x20000040f:0x2:0x0]
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 1
obdidx objid objid group
1 229 0xe5 0
file2/test_file
lmm_magic: 0x0BD10BD0
lmm_seq: 0x24000040c
lmm_object_id: 0x2
lmm_fid: [0x24000040c:0x2:0x0]
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 1
obdidx objid objid group
1 227 0xe3 0x300000401
test_1
lmm_magic: 0x0BD10BD0
lmm_seq: 0x280000413
lmm_object_id: 0x1
lmm_fid: [0x280000413:0x1:0x0]
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 1
obdidx objid objid group
1 610 0x262 0x300000400
On Tue, Jun 23, 2026 at 4:30 AM David Schanzenbach <davidls at hawaii.edu> wrote:
>
> Hi Homer,
>
> Is it possible you are hitting this know bug with Lustre 2.15.7 + zfs?
> https://jira.whamcloud.com/browse/LU-19193
>
>
> David
>
> On 6/22/2026 6:20 AM, Homer Li via lustre-discuss wrote:
>
> Hi All
> Using rclone to read from object storage and write in parallel to
> multiple versions of Lustre 2.15 + ZFS causes some space waste. This
> issue does not occur with ZFS and any version of Lustre 2.12.
>
> In multi-threaded mode, multiple pwrite64 calls with different offsets
> are invoked, while single-threaded sequential write operations
> (detailed below) do not result in space amplification. The Lustre
> implementation causes issues with ZFS space allocation.
>
> Below are the commands to reproduce this issue:
> rclone copy /tmp/test_file /mnt/lustre/zfs_index1/file1 --transfers 1
> --multi-thread-streams 32
> zfs dnode=648
>
> rclone copy /tmp/test_file /mnt/lustre/zfs_index1/file2 --transfers 1
> --multi-thread-streams 1
> zfs dnode=839
>
> client$ ls -lhs /mnt/lustre/zfs_index1/file1 /mnt/lustre/zfs_index1/file2
> /mnt/lustre/zfs_index1/file1:
> total 1.5G
> 1.5G -rw-r--r-- 1 root root 1.0G Jun 22 20:57 test_file
>
> /mnt/lustre/zfs_index1/file2:
> total 994M
> 994M -rw-r--r-- 1 root root 1.0G Jun 22 20:57 test_file
>
> client$ xxh128sum /mnt/lustre/zfs_index1/file1/test_file
> /mnt/lustre/zfs_index1/file2/test_file
> 16024760318c6298efd1151033ad2e9f /mnt/lustre/zfs_index1/file1/test_file
> 16024760318c6298efd1151033ad2e9f /mnt/lustre/zfs_index1/file2/test_file
>
> oss$ umount -t lustre
> oss$ zfs mount -a
>
> oss$ ls -lihs ./0/d5/229 ./300000401/d3/227
> 648 1.5G -rw-rw-rw- 1 root root 1.0G Jun 22 20:57 ./0/d5/229
> 839 994M -rw-rw-rw- 1 root root 1.0G Jun 22 20:57 ./300000401/d3/227
>
>
> oss$ xxh128sum ./0/d5/229 ./300000401/d3/227
> 16024760318c6298efd1151033ad2e9f ./0/d5/229
> 16024760318c6298efd1151033ad2e9f ./300000401/d3/227
>
> What should have been mapped by a single L1 indirect block fractured
> into 256 L1 blocks and an additional L2 indirect block, triggering
> performance degradations and an ARC footprint explosion.
>
> A 4KB dblk completely fragments orderly sequential writes into a
> massive amount of tiny random I/Os. This not only leads to
> catastrophic space alignment and parity amplification on RAID-Z
> arrays, but also severely wastes ARC memory cache due to the metadata
> tree structure.
>
> oss$ zdb -ddddd test_ost1/test_ost1 648
> Object lvl iblk dblk dsize dnsize lsize %full type
> 648 3 128K 4K 1.46G 512 1G 100.00 ZFS plain file
> 192 bonus System attributes
> dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED SPILL_BLKPTR
> dnode maxblkid: 262143
> path /O/0/d5/229
> ...
> Spill block: 0:3b01ca7000:3000 200L/200P F=1 B=7452/7452
> cksum=000000046340a6b6:000001c2df0b4eca:00005d31f69d6ee9:000d32915fcfae51
> Indirect blocks:
> 0 L2 0:3b01d04000:9000 20000L/3000P F=262144
> B=7452/7452 cksum=0000049e9f732fed:001a617da1cd2146:68ba901169b7e8d6:1203fa42b8e153ca
> 0 L1 0:3b01caa000:6000 20000L/2000P F=1024
> B=7452/7452 cksum=0000019630118006:0008913b752091c4:18e62b360f606bea:31823a30bb201c6b
> 0 L0 0:fc08b2000:3000 1000L/1000P F=1 B=7447/7447
> cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> 1000 L0 0:fc08b5000:3000 1000L/1000P F=1 B=7447/7447
> cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> 2000 L0 0:fc08b8000:3000 1000L/1000P F=1 B=7447/7447
> cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> 3000 L0 0:fc08bb000:3000 1000L/1000P F=1 B=7447/7447
> cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> 4000 L0 0:fc08be000:3000 1000L/1000P F=1 B=7447/7447
> cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
>
>
>
> oss$ zdb -ddddd test_ost1/test_ost1 839
> Object lvl iblk dblk dsize dnsize lsize %full type
> 839 2 128K 1M 994M 512 1G 100.00 ZFS plain file
> 192 bonus System attributes
> dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED SPILL_BLKPTR
> dnode maxblkid: 1023
> path /O/300000401/d3/227
>
> ....
> Indirect blocks:
> 0 L1 0:3b000e1000:6000 20000L/2000P F=1024 B=7452/7452
> cksum=00000229556c8638:000a9a2f615cde18:1d4cdd6e53eba13e:bbc5d925c17605f6
> 0 L0 0:397f896000:201000 100000L/100000P F=1
> B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> 100000 L0 0:397fa97000:201000 100000L/100000P F=1
> B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> 200000 L0 0:397fc98000:201000 100000L/100000P F=1
> B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> 300000 L0 0:398009a000:201000 100000L/100000P F=1
> B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> 400000 L0 0:397fe99000:201000 100000L/100000P F=1
> B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
>
>
>
> --
> May the wind guide your path; May the stars light your way
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!PvDODwlR4mBZyAb0!TYRsnPPco3pZxVAJ_z0a9A3CCP3IIEmwfpDcJZwatwa929qir0JLvzxpHQQpFLmgbHriuLevycbJC-Q0DivC_EVuI8CF1E0$
>
>
--
May the wind guide your path; May the stars light your way
More information about the lustre-discuss
mailing list