[lustre-discuss] The pre-allocation and parallel offset writing lead to massive ZFS space waste
Homer Li
01jay.ly at gmail.com
Fri Jun 26 01:46:30 UTC 2026
The issue is fixed in the dev branch, but not in any releases yet. I
will test it next time.
osd_choose_next_blocksize 2.17.54:
if (off == obj->oo_attr.la_size) blksz = (uint32_t)max_t(uint64_t,
osd->od_min_blksz, off + len);
else blksz = (uint32_t)max_t(uint64_t, osd->od_min_blksz, len);
osd_choose_next_blocksize 2.17.0:
/*
* client sends data from own writeback cache after local
* aggregation. there is a chance this is a "unit of write"
* so blocksize.
*/
if (off != 0)
return;
On Tue, Jun 23, 2026 at 3:40 PM Homer Li <01jay.ly at gmail.com> wrote:
>
> Hi David
> Thanks.
> My test environment is based on Lustre 2.15.8 and OpenZFS 2.3.2-1. In
> LU-19193, Alex mentioned, "we noticed that disabling PFL also
> circumvented the problem."
> After reviewing the rclone code, I easily reproduced the issue without
> any parallel operations or random writes, using simply:
>
> dd if=/dev/urandom of=test_1 bs=1M seek=1 count=555
>
> This issue is triggered by any seek operation, regardless of whether
> the seek offset aligns with the ZFS recordsize.
> Therefore, I believe this appears to be unrelated to LU-19193,
> although both are issues stemming from osd_zfs writes. That said, I am
> not entirely clear yet whether they are linked at a micro-level.
> However, if a massive waste of ZFS space and cache can be triggered by
> any arbitrary seek—even an aligned one—it strongly suggests a flaw in
> the osd_zfs implementation. A correlation in their underlying
> behaviors might still exist.
>
> OSS$ ls -lhsi ./O/300000400/d2/610
> 771 818M -rw-rw-rw- 1 root root 556M Jun 23 15:04 ./O/300000400/d2/610
>
> OSS$ zdb -ddddd test_ost1/test_ost1 771
> Object lvl iblk dblk dsize dnsize lsize %full type
> 771 3 128K 4K 818M 512 556M 99.82 ZFS plain file
> dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED SPILL_BLKPTR
> dnode maxblkid: 142335
> path /O/300000400/d2/610
>
> Spill block: 0:55d10b000:3000 200L/200P F=1 B=18365/18365
> cksum=00000004e33f9590:000001f55e95bc2a:000067415ce5f733:000e8bf71249da1b
> Indirect blocks:
> 0 L2 0:55d129000:6000 20000L/2000P F=142080
> B=18365/18365 cksum=000002e732cdb1dd:000c6d79cf6b7e09:20c64bd91e328904:dccb8db21d625648
> 0 L1 0:408877000:12000 20000L/8000P F=768
> B=18364/18364 cksum=00000ba8006726b4:00c6dfeef5929795:5a08d96ddbf7d946:11860a2720229dc7
> 100000 L0 0:45be3c000:3000 1000L/1000P F=1 B=18364/18364
> cksum=000001fd2b012e1d:0003fb90edce4119:0554dfaadb8e004c:5dc10c44c4295a16
> 101000 L0 0:45be3f000:3000 1000L/1000P F=1 B=18364/18364
> cksum=00000200d5999b15:0003fff5e3eb0af4:05504ba2292e3ffa:51549dfa08faf869
> 102000 L0 0:45be45000:3000 1000L/1000P F=1 B=18364/18364
> cksum=0000020f1f369d30:00042c64d9d74c88:05a55f5d8b6c73ae:b6784f0e99facfb2
> 103000 L0 0:45be42000:3000 1000L/1000P F=1 B=18364/18364
> cksum=000001fe8650df2c:0003fa93d62eba6f:0553b01c9404084b:5c3180d7716ae57e
> 104000 L0 0:45be48000:3000 1000L/1000P F=1 B=18364/18364
> cksum=0000020ff8d32130:0004165fcd65bf45:056ae4c5dc462c0d:6685186e3e73be3a
> .......
> 22bfc000 L0 0:55d0ff000:3000 1000L/1000P F=1 B=18365/18365
> cksum=000001fa32329f80:0003ee608baf274b:053db2fc3f83dd58:3d9703d9ca7040f2
> 22bfd000 L0 0:55d102000:3000 1000L/1000P F=1 B=18365/18365
> cksum=000001ede321b4ce:0003e1cb966ee514:05399c6d901e2a82:4747bec737a9960a
> 22bfe000 L0 0:55d105000:3000 1000L/1000P F=1 B=18365/18365
> cksum=000001feb221eba1:0003fa18810584d9:05520f713c6d460f:57db8d048aaea8ec
> 22bff000 L0 0:55d108000:3000 1000L/1000P F=1 B=18365/18365
> cksum=0000020043cd43a9:0003fcda8fa1b211:055aac69168825c2:65d26516ef5bdcf0
>
> segment [0000000000100000, 0000000022c00000) size 555M
>
>
> OSS$ zfs get recordsize
> NAME PROPERTY VALUE SOURCE
> test_ost1 recordsize 128K default
> test_ost1/test_ost1 recordsize 1M local
>
> client$ lfs getstripe -v file1/test_file file2/test_file test_1
> file1/test_file
> lmm_magic: 0x0BD10BD0
> lmm_seq: 0x20000040f
> lmm_object_id: 0x2
> lmm_fid: [0x20000040f:0x2:0x0]
> lmm_stripe_count: 1
> lmm_stripe_size: 1048576
> lmm_pattern: raid0
> lmm_layout_gen: 0
> lmm_stripe_offset: 1
> obdidx objid objid group
> 1 229 0xe5 0
>
> file2/test_file
> lmm_magic: 0x0BD10BD0
> lmm_seq: 0x24000040c
> lmm_object_id: 0x2
> lmm_fid: [0x24000040c:0x2:0x0]
> lmm_stripe_count: 1
> lmm_stripe_size: 1048576
> lmm_pattern: raid0
> lmm_layout_gen: 0
> lmm_stripe_offset: 1
> obdidx objid objid group
> 1 227 0xe3 0x300000401
>
> test_1
> lmm_magic: 0x0BD10BD0
> lmm_seq: 0x280000413
> lmm_object_id: 0x1
> lmm_fid: [0x280000413:0x1:0x0]
> lmm_stripe_count: 1
> lmm_stripe_size: 1048576
> lmm_pattern: raid0
> lmm_layout_gen: 0
> lmm_stripe_offset: 1
> obdidx objid objid group
> 1 610 0x262 0x300000400
>
> On Tue, Jun 23, 2026 at 4:30 AM David Schanzenbach <davidls at hawaii.edu> wrote:
> >
> > Hi Homer,
> >
> > Is it possible you are hitting this know bug with Lustre 2.15.7 + zfs?
> > https://jira.whamcloud.com/browse/LU-19193
> >
> >
> > David
> >
> > On 6/22/2026 6:20 AM, Homer Li via lustre-discuss wrote:
> >
> > Hi All
> > Using rclone to read from object storage and write in parallel to
> > multiple versions of Lustre 2.15 + ZFS causes some space waste. This
> > issue does not occur with ZFS and any version of Lustre 2.12.
> >
> > In multi-threaded mode, multiple pwrite64 calls with different offsets
> > are invoked, while single-threaded sequential write operations
> > (detailed below) do not result in space amplification. The Lustre
> > implementation causes issues with ZFS space allocation.
> >
> > Below are the commands to reproduce this issue:
> > rclone copy /tmp/test_file /mnt/lustre/zfs_index1/file1 --transfers 1
> > --multi-thread-streams 32
> > zfs dnode=648
> >
> > rclone copy /tmp/test_file /mnt/lustre/zfs_index1/file2 --transfers 1
> > --multi-thread-streams 1
> > zfs dnode=839
> >
> > client$ ls -lhs /mnt/lustre/zfs_index1/file1 /mnt/lustre/zfs_index1/file2
> > /mnt/lustre/zfs_index1/file1:
> > total 1.5G
> > 1.5G -rw-r--r-- 1 root root 1.0G Jun 22 20:57 test_file
> >
> > /mnt/lustre/zfs_index1/file2:
> > total 994M
> > 994M -rw-r--r-- 1 root root 1.0G Jun 22 20:57 test_file
> >
> > client$ xxh128sum /mnt/lustre/zfs_index1/file1/test_file
> > /mnt/lustre/zfs_index1/file2/test_file
> > 16024760318c6298efd1151033ad2e9f /mnt/lustre/zfs_index1/file1/test_file
> > 16024760318c6298efd1151033ad2e9f /mnt/lustre/zfs_index1/file2/test_file
> >
> > oss$ umount -t lustre
> > oss$ zfs mount -a
> >
> > oss$ ls -lihs ./0/d5/229 ./300000401/d3/227
> > 648 1.5G -rw-rw-rw- 1 root root 1.0G Jun 22 20:57 ./0/d5/229
> > 839 994M -rw-rw-rw- 1 root root 1.0G Jun 22 20:57 ./300000401/d3/227
> >
> >
> > oss$ xxh128sum ./0/d5/229 ./300000401/d3/227
> > 16024760318c6298efd1151033ad2e9f ./0/d5/229
> > 16024760318c6298efd1151033ad2e9f ./300000401/d3/227
> >
> > What should have been mapped by a single L1 indirect block fractured
> > into 256 L1 blocks and an additional L2 indirect block, triggering
> > performance degradations and an ARC footprint explosion.
> >
> > A 4KB dblk completely fragments orderly sequential writes into a
> > massive amount of tiny random I/Os. This not only leads to
> > catastrophic space alignment and parity amplification on RAID-Z
> > arrays, but also severely wastes ARC memory cache due to the metadata
> > tree structure.
> >
> > oss$ zdb -ddddd test_ost1/test_ost1 648
> > Object lvl iblk dblk dsize dnsize lsize %full type
> > 648 3 128K 4K 1.46G 512 1G 100.00 ZFS plain file
> > 192 bonus System attributes
> > dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED SPILL_BLKPTR
> > dnode maxblkid: 262143
> > path /O/0/d5/229
> > ...
> > Spill block: 0:3b01ca7000:3000 200L/200P F=1 B=7452/7452
> > cksum=000000046340a6b6:000001c2df0b4eca:00005d31f69d6ee9:000d32915fcfae51
> > Indirect blocks:
> > 0 L2 0:3b01d04000:9000 20000L/3000P F=262144
> > B=7452/7452 cksum=0000049e9f732fed:001a617da1cd2146:68ba901169b7e8d6:1203fa42b8e153ca
> > 0 L1 0:3b01caa000:6000 20000L/2000P F=1024
> > B=7452/7452 cksum=0000019630118006:0008913b752091c4:18e62b360f606bea:31823a30bb201c6b
> > 0 L0 0:fc08b2000:3000 1000L/1000P F=1 B=7447/7447
> > cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> > 1000 L0 0:fc08b5000:3000 1000L/1000P F=1 B=7447/7447
> > cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> > 2000 L0 0:fc08b8000:3000 1000L/1000P F=1 B=7447/7447
> > cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> > 3000 L0 0:fc08bb000:3000 1000L/1000P F=1 B=7447/7447
> > cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> > 4000 L0 0:fc08be000:3000 1000L/1000P F=1 B=7447/7447
> > cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> >
> >
> >
> > oss$ zdb -ddddd test_ost1/test_ost1 839
> > Object lvl iblk dblk dsize dnsize lsize %full type
> > 839 2 128K 1M 994M 512 1G 100.00 ZFS plain file
> > 192 bonus System attributes
> > dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED SPILL_BLKPTR
> > dnode maxblkid: 1023
> > path /O/300000401/d3/227
> >
> > ....
> > Indirect blocks:
> > 0 L1 0:3b000e1000:6000 20000L/2000P F=1024 B=7452/7452
> > cksum=00000229556c8638:000a9a2f615cde18:1d4cdd6e53eba13e:bbc5d925c17605f6
> > 0 L0 0:397f896000:201000 100000L/100000P F=1
> > B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> > 100000 L0 0:397fa97000:201000 100000L/100000P F=1
> > B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> > 200000 L0 0:397fc98000:201000 100000L/100000P F=1
> > B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> > 300000 L0 0:398009a000:201000 100000L/100000P F=1
> > B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> > 400000 L0 0:397fe99000:201000 100000L/100000P F=1
> > B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
> >
> >
> >
> > --
> > May the wind guide your path; May the stars light your way
> > _______________________________________________
> > lustre-discuss mailing list
> > lustre-discuss at lists.lustre.org
> > https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!PvDODwlR4mBZyAb0!TYRsnPPco3pZxVAJ_z0a9A3CCP3IIEmwfpDcJZwatwa929qir0JLvzxpHQQpFLmgbHriuLevycbJC-Q0DivC_EVuI8CF1E0$
> >
> >
>
>
> --
> May the wind guide your path; May the stars light your way
--
May the wind guide your path; May the stars light your way
More information about the lustre-discuss
mailing list