<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
Hi Homer,<br>
<br>
Is it possible you are hitting this know bug with Lustre 2.15.7 +
zfs?<br>
<a class="moz-txt-link-freetext" href="https://jira.whamcloud.com/browse/LU-19193">https://jira.whamcloud.com/browse/LU-19193</a><br>
<br>
<br>
David<br>
<br>
<div class="moz-cite-prefix">On 6/22/2026 6:20 AM, Homer Li via
lustre-discuss wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAG1+85ETv_UyDo9DrL58M2_2C_pnUpzgzxzXzUp8WEUTHA9sEA@mail.gmail.com">
<pre wrap="" class="moz-quote-pre">Hi All
Using rclone to read from object storage and write in parallel to
multiple versions of Lustre 2.15 + ZFS causes some space waste. This
issue does not occur with ZFS and any version of Lustre 2.12.
In multi-threaded mode, multiple pwrite64 calls with different offsets
are invoked, while single-threaded sequential write operations
(detailed below) do not result in space amplification. The Lustre
implementation causes issues with ZFS space allocation.
Below are the commands to reproduce this issue:
rclone copy /tmp/test_file /mnt/lustre/zfs_index1/file1 --transfers 1
--multi-thread-streams 32
zfs dnode=648
rclone copy /tmp/test_file /mnt/lustre/zfs_index1/file2 --transfers 1
--multi-thread-streams 1
zfs dnode=839
client$ ls -lhs /mnt/lustre/zfs_index1/file1 /mnt/lustre/zfs_index1/file2
/mnt/lustre/zfs_index1/file1:
total 1.5G
1.5G -rw-r--r-- 1 root root 1.0G Jun 22 20:57 test_file
/mnt/lustre/zfs_index1/file2:
total 994M
994M -rw-r--r-- 1 root root 1.0G Jun 22 20:57 test_file
client$ xxh128sum /mnt/lustre/zfs_index1/file1/test_file
/mnt/lustre/zfs_index1/file2/test_file
16024760318c6298efd1151033ad2e9f /mnt/lustre/zfs_index1/file1/test_file
16024760318c6298efd1151033ad2e9f /mnt/lustre/zfs_index1/file2/test_file
oss$ umount -t lustre
oss$ zfs mount -a
oss$ ls -lihs ./0/d5/229 ./300000401/d3/227
648 1.5G -rw-rw-rw- 1 root root 1.0G Jun 22 20:57 ./0/d5/229
839 994M -rw-rw-rw- 1 root root 1.0G Jun 22 20:57 ./300000401/d3/227
oss$ xxh128sum ./0/d5/229 ./300000401/d3/227
16024760318c6298efd1151033ad2e9f ./0/d5/229
16024760318c6298efd1151033ad2e9f ./300000401/d3/227
What should have been mapped by a single L1 indirect block fractured
into 256 L1 blocks and an additional L2 indirect block, triggering
performance degradations and an ARC footprint explosion.
A 4KB dblk completely fragments orderly sequential writes into a
massive amount of tiny random I/Os. This not only leads to
catastrophic space alignment and parity amplification on RAID-Z
arrays, but also severely wastes ARC memory cache due to the metadata
tree structure.
oss$ zdb -ddddd test_ost1/test_ost1 648
Object lvl iblk dblk dsize dnsize lsize %full type
648 3 128K 4K 1.46G 512 1G 100.00 ZFS plain file
192 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED SPILL_BLKPTR
dnode maxblkid: 262143
path /O/0/d5/229
...
Spill block: 0:3b01ca7000:3000 200L/200P F=1 B=7452/7452
cksum=000000046340a6b6:000001c2df0b4eca:00005d31f69d6ee9:000d32915fcfae51
Indirect blocks:
0 L2 0:3b01d04000:9000 20000L/3000P F=262144
B=7452/7452 cksum=0000049e9f732fed:001a617da1cd2146:68ba901169b7e8d6:1203fa42b8e153ca
0 L1 0:3b01caa000:6000 20000L/2000P F=1024
B=7452/7452 cksum=0000019630118006:0008913b752091c4:18e62b360f606bea:31823a30bb201c6b
0 L0 0:fc08b2000:3000 1000L/1000P F=1 B=7447/7447
cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
1000 L0 0:fc08b5000:3000 1000L/1000P F=1 B=7447/7447
cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
2000 L0 0:fc08b8000:3000 1000L/1000P F=1 B=7447/7447
cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
3000 L0 0:fc08bb000:3000 1000L/1000P F=1 B=7447/7447
cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
4000 L0 0:fc08be000:3000 1000L/1000P F=1 B=7447/7447
cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
oss$ zdb -ddddd test_ost1/test_ost1 839
Object lvl iblk dblk dsize dnsize lsize %full type
839 2 128K 1M 994M 512 1G 100.00 ZFS plain file
192 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED SPILL_BLKPTR
dnode maxblkid: 1023
path /O/300000401/d3/227
....
Indirect blocks:
0 L1 0:3b000e1000:6000 20000L/2000P F=1024 B=7452/7452
cksum=00000229556c8638:000a9a2f615cde18:1d4cdd6e53eba13e:bbc5d925c17605f6
0 L0 0:397f896000:201000 100000L/100000P F=1
B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
100000 L0 0:397fa97000:201000 100000L/100000P F=1
B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
200000 L0 0:397fc98000:201000 100000L/100000P F=1
B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
300000 L0 0:398009a000:201000 100000L/100000P F=1
B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
400000 L0 0:397fe99000:201000 100000L/100000P F=1
B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
--
May the wind guide your path; May the stars light your way
_______________________________________________
lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!PvDODwlR4mBZyAb0!TYRsnPPco3pZxVAJ_z0a9A3CCP3IIEmwfpDcJZwatwa929qir0JLvzxpHQQpFLmgbHriuLevycbJC-Q0DivC_EVuI8CF1E0$">https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!PvDODwlR4mBZyAb0!TYRsnPPco3pZxVAJ_z0a9A3CCP3IIEmwfpDcJZwatwa929qir0JLvzxpHQQpFLmgbHriuLevycbJC-Q0DivC_EVuI8CF1E0$</a>
</pre>
</blockquote>
<br>
</body>
</html>