[lustre-discuss] Project quota and odd mv behavior
Hans Henrik Happe
happe at nbi.dk
Wed Dec 17 13:11:09 PST 2025
Thanks for digging up the LUs that I somehow didn't manage to find.
Also, I noticed that the XFS trace was for the case where Lustre is fast
(walking/creating the tree and rename files).
I've attached the other case (dest: project0/d1) for XFS. If I'm
interpreting it right it seems that on XFS mv is using reflinks for the
copy:
ioctl(4, BTRFS_IOC_CLONE or FICLONE, 3) = 0
That explains why it is fast on XFS and a full copy on Lustre which
doesn't support this operation.
I'm not sure I see the logic in the mv implementation. It should be
possible to handle both cases by walking/creating the tree and rename
files. I will dig deeper into the mv code to figure out what is going on.
Cheers,
Hans Henrik
On 17/12/2025 21.15, Andreas Dilger wrote:
> Looking at the strace output, it appears in all cases that the *directory*
> rename is returning EXDEV, as it should be the case (to preserve the projid
> for that directory tree). In the XFS case, it is refusing to do the file
> rename across projects also (returning EXDEV and forcing userspace to do the
> file copy, which is very inefficient), while Lustre allows the file rename
> to succeed (just reassinging the project quota usage).
>
> In the lustre-mv-copy case, it appears that "mv" is not trying to rename the
> file again after it creates the target directory, as it does in the lustre-mv
> case. Instead it just copies the file, which seems like a bug in "mv".
>
> It looks like the renameat2() flags have not been implemented yet:
> https://jira.whamcloud.com/browse/LU-12272 - RENAME_NOREPLACE
> https://jira.whamcloud.com/browse/LU-11557 - RENAME_EXCHANGE
> https://jira.whamcloud.com/browse/LU-19720 - RENAME_WHITEOUT
>
> Probably implementing RENAME_NOREPLACE support would be the most important
> and useful flag, since "mv" is always trying this first. RENAME_EXCHANGE
> could be useful for some applications, while RENAME_WHITEOUT is for overlayfs.
>
> Cheers, Andreas
>
>> On Dec 17, 2025, at 05:54, Hans Henrik Happe via lustre-discuss<lustre-discuss at lists.lustre.org> wrote:
>>
>> To be more specific:
>>
>> Server:
>> - Lustre 2.15.7
>> - Rocky Linux 8.10 (4.18.0-553.53.1.el8_10.x86_64)
>>
>> Client:
>> - Lustre 2.15.7, but same outcome on 2.15.5
>> - Rocky Linux 9.6 (5.14.0-570.42.2.el9_6.x86_64)
>> - glibc-2.34-168.el9_6.23.x86_64
>> - coreutils-8.32-39.el9.x86_64
>>
>>
>>
>> On 17/12/2025 13.22, Hans Henrik Happe via lustre-discuss wrote:
>>> It's 2.15.7.
>>>
>>> On 17/12/2025 03.36, Andreas Dilger wrote:
>>>> There have definitely been some fixes in this area.
>>>> What version of Lustre are you running?
>>>>
>>>>
>>>>> On Dec 16, 2025, at 07:39, Hans Henrik Happe via lustre-discuss<lustre-discuss at lists.lustre.org> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>> It seems like the inherited project quota issue [1] with mv has been resolved with XFS in RHEL9. However, Lustre is a bit strange.
>>>>>
>>>>> Given a "src" and "dst"(empty) dir these two ways of calling mv behaves differently:
>>>>>
>>>>> mv src dst/ (work: no copy)
>>>>> mv src dst/src (copy, then delete)
>>>>>
>>>>> I've attached an strace output for both Lustre and XFS. It seems like mv is handling the fact that Lustre don't have renameat2 a bit differently.
>>>>>
>>>>> Before I dig further. Is this behavior known?
>>>>>
>>>>> Cheers,
>>>>> Hans Henrik
>>>>>
>>>>> [1]http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2023-February/018511.html
>>>>> <lustre-mv.strace><xfs-mv.strace><lustre-mv-copy.strace>_______________________________________________
>>>>> lustre-discuss mailing list
>>>>> lustre-discuss at lists.lustre.org
>>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>>>
>>>>>
>> Cheers,
>> Hans Henrik
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> Andreas Dilger
> Principal Lustre Architect
> adilger at thelustrecollective.com
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20251217/0ded1ac4/attachment.htm>
-------------- next part --------------
renameat2(AT_FDCWD, "project1/d1", AT_FDCWD, "project0/d1", RENAME_NOREPLACE) = -1 EXDEV (Invalid cross-device link)
newfstatat(AT_FDCWD, "project0/d1", 0x7ffecf195440, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "project1/d1", {st_mode=S_IFDIR|0755, st_size=26, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "project0/d1", 0x7ffecf194fa0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
rmdir("project0/d1") = -1 ENOENT (No such file or directory)
mkdir("project0/d1", 0700) = 0
newfstatat(AT_FDCWD, "project0/d1", {st_mode=S_IFDIR|0700, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "project1/d1", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=26, ...}) = 0
getdents64(3, 0x561e74912300 /* 4 entries */, 32768) = 96
getdents64(3, 0x561e74912300 /* 0 entries */, 32768) = 0
close(3) = 0
newfstatat(AT_FDCWD, "project1/d1/d2", {st_mode=S_IFDIR|0755, st_size=16, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "project0/d1/d2", 0x7ffecf194a50, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
rmdir("project0/d1/d2") = -1 ENOENT (No such file or directory)
mkdir("project0/d1/d2", 0700) = 0
newfstatat(AT_FDCWD, "project0/d1/d2", {st_mode=S_IFDIR|0700, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "project1/d1/d2", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=16, ...}) = 0
getdents64(3, 0x561e74912300 /* 3 entries */, 32768) = 72
getdents64(3, 0x561e74912300 /* 0 entries */, 32768) = 0
close(3) = 0
newfstatat(AT_FDCWD, "project1/d1/d2/f2", {st_mode=S_IFREG|0644, st_size=2147483648, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "project0/d1/d2/f2", 0x7ffecf194500, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
unlink("project0/d1/d2/f2") = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "project1/d1/d2/f2", O_RDONLY|O_NOFOLLOW) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2147483648, ...}) = 0
openat(AT_FDCWD, "project0/d1/d2/f2", O_WRONLY|O_CREAT|O_EXCL, 0600) = 4
fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
ioctl(4, BTRFS_IOC_CLONE or FICLONE, 3) = 0
More information about the lustre-discuss
mailing list