[lustre-discuss] The pre-allocation and parallel offset writing lead to massive ZFS space waste

[Homer Li]

Hi All
    Using rclone to read from object storage and write in parallel to
multiple versions of Lustre 2.15 + ZFS causes some space waste. This
issue does not occur with ZFS and any version of Lustre 2.12.

In multi-threaded mode, multiple pwrite64 calls with different offsets
are invoked, while single-threaded sequential write operations
(detailed below) do not result in space amplification. The Lustre
implementation causes issues with ZFS space allocation.

Below are the commands to reproduce this issue:
 rclone copy /tmp/test_file /mnt/lustre/zfs_index1/file1 --transfers 1
--multi-thread-streams 32
zfs dnode=648

 rclone copy /tmp/test_file /mnt/lustre/zfs_index1/file2 --transfers 1
--multi-thread-streams 1
zfs dnode=839

client$ ls -lhs /mnt/lustre/zfs_index1/file1 /mnt/lustre/zfs_index1/file2
/mnt/lustre/zfs_index1/file1:
total 1.5G
1.5G -rw-r--r-- 1 root root 1.0G Jun 22 20:57 test_file

/mnt/lustre/zfs_index1/file2:
total 994M
994M -rw-r--r-- 1 root root 1.0G Jun 22 20:57 test_file

client$ xxh128sum /mnt/lustre/zfs_index1/file1/test_file
/mnt/lustre/zfs_index1/file2/test_file
16024760318c6298efd1151033ad2e9f  /mnt/lustre/zfs_index1/file1/test_file
16024760318c6298efd1151033ad2e9f  /mnt/lustre/zfs_index1/file2/test_file

oss$ umount -t lustre
oss$ zfs mount -a

oss$ ls -lihs ./0/d5/229 ./300000401/d3/227
648 1.5G -rw-rw-rw- 1 root root 1.0G Jun 22 20:57 ./0/d5/229
839 994M -rw-rw-rw- 1 root root 1.0G Jun 22 20:57 ./300000401/d3/227


oss$ xxh128sum ./0/d5/229 ./300000401/d3/227
16024760318c6298efd1151033ad2e9f  ./0/d5/229
16024760318c6298efd1151033ad2e9f  ./300000401/d3/227

What should have been mapped by a single L1 indirect block fractured
into 256 L1 blocks and an additional L2 indirect block, triggering
performance degradations and an ARC footprint explosion.

A 4KB dblk completely fragments orderly sequential writes into a
massive amount of tiny random I/Os. This not only leads to
catastrophic space alignment and parity amplification on RAID-Z
arrays, but also severely wastes ARC memory cache due to the metadata
tree structure.

oss$ zdb -ddddd test_ost1/test_ost1 648
    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
       648    3   128K     4K  1.46G     512     1G  100.00  ZFS plain file
                                               192   bonus  System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED SPILL_BLKPTR
dnode maxblkid: 262143
path /O/0/d5/229
...
Spill block: 0:3b01ca7000:3000 200L/200P F=1 B=7452/7452
cksum=000000046340a6b6:000001c2df0b4eca:00005d31f69d6ee9:000d32915fcfae51
Indirect blocks:
               0 L2   0:3b01d04000:9000 20000L/3000P F=262144
B=7452/7452 cksum=0000049e9f732fed:001a617da1cd2146:68ba901169b7e8d6:1203fa42b8e153ca
               0  L1  0:3b01caa000:6000 20000L/2000P F=1024
B=7452/7452 cksum=0000019630118006:0008913b752091c4:18e62b360f606bea:31823a30bb201c6b
               0   L0 0:fc08b2000:3000 1000L/1000P F=1 B=7447/7447
cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
            1000   L0 0:fc08b5000:3000 1000L/1000P F=1 B=7447/7447
cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
            2000   L0 0:fc08b8000:3000 1000L/1000P F=1 B=7447/7447
cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
            3000   L0 0:fc08bb000:3000 1000L/1000P F=1 B=7447/7447
cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
            4000   L0 0:fc08be000:3000 1000L/1000P F=1 B=7447/7447
cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000



oss$ zdb -ddddd test_ost1/test_ost1 839
    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
       839    2   128K     1M   994M     512     1G  100.00  ZFS plain file
                                               192   bonus  System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED SPILL_BLKPTR
dnode maxblkid: 1023
path /O/300000401/d3/227

....
Indirect blocks:
               0 L1  0:3b000e1000:6000 20000L/2000P F=1024 B=7452/7452
cksum=00000229556c8638:000a9a2f615cde18:1d4cdd6e53eba13e:bbc5d925c17605f6
               0  L0 0:397f896000:201000 100000L/100000P F=1
B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
          100000  L0 0:397fa97000:201000 100000L/100000P F=1
B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
          200000  L0 0:397fc98000:201000 100000L/100000P F=1
B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
          300000  L0 0:398009a000:201000 100000L/100000P F=1
B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000
          400000  L0 0:397fe99000:201000 100000L/100000P F=1
B=7449/7449 cksum=0000000000000000:0000000000000000:0000000000000000:0000000000000000



--
May the wind guide your path; May the stars light your way