<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-CA" link="blue" vlink="purple" style="word-wrap:break-word;-webkit-nbsp-mode: space;line-break:after-white-space">
<div class="WordSection1">
<p class="MsoNormal">Hi Andreas,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you very much! Yes, this explains it. <o:p></o:p></p>
<p class="MsoNormal">Looks like rsync or lustre-rsync is still the best solution for now (we are on Lustre 2.12).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;color:black">-- </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.5pt;color:black">Grigory Shamov</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.5pt;color:black">Site Lead / HPC Specialist</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:13.5pt;color:black">University of Manitoba and DRI Alliance Canada</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Andreas Dilger <adilger@whamcloud.com><br>
<b>Date: </b>Monday, February 27, 2023 at 11:36 PM<br>
<b>To: </b>Grigory Shamov <Grigory.Shamov@umanitoba.ca><br>
<b>Cc: </b>lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] Renaming or Moving directories on Lustre?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<table class="MsoNormalTable" border="1" cellpadding="0" style="background:#FBE3B3;border:solid black 1.0pt">
<tbody>
<tr>
<td style="border:none;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><strong><span style="font-family:"Calibri",sans-serif;color:#462C1D">Caution:</span></strong><span style="color:#462C1D"> This message was sent from outside the University of Manitoba.</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><br>
On Feb 27, 2023, at 11:57, Grigory Shamov <<a href="mailto:Grigory.Shamov@umanitoba.ca">Grigory.Shamov@umanitoba.ca</a>> wrote:<o:p></o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hi All,<br>
<br>
What happens if a directory on Lustre FS gets moved with a regular CentOS7 mv command, within the same filesystem? On CentOS 7, using mv from the distro, like this, as root:<br>
<br>
mv /project/TEMP/user /project/XYZ/user<br>
<br>
It looks like the content gets copied entirely. Which for large data takes a large amount of time.<br>
Is there a way to rename the Lustre directories (changing the name of the top directory, only without moving every object in these directories)? Thanks!<o:p></o:p></p>
</div>
</div>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Renaming a file or subdirectory tree between "regular" directories in Lustre works as you would expect for a local filesystem, even if the directories are on different MDTs. What you are seeing (full copy of contents between directories)
is really a result of the implementation/design of project quotas, and not directly a Lustre problem. The same would happen if you have two directories using two different project IDs and the "PROJINHERIT" flag set with ext4 or XFS, since they also return
"-EXDEV" if trying to move (rename) a file between directories that do not have the same project ID, and that causes "mv" to copy the whole directory tree.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Running the ext4 "mv" under strace shows this:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> # df -T /mnt/tmp<o:p></o:p></p>
</div>
<p class="MsoNormal"> Filesystem Type 1K-blocks Used Available Use% Mounted on<br>
/dev/mapper/vg_test-lvtest ext4 16337788 52 15482492 1% /mnt/tmp <o:p>
</o:p></p>
<div>
<p class="MsoNormal"> # mkdir /mnt/tmp/{dir1,dir2}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> # chattr -P -p 1000 /mnt/tmp/dir1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> # chattr -P -p 2000 /mnt/tmp/dir2<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> # cp /etc/hosts /mnt/tmp/dir1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> # lsattr /mnt/tmp/dir1<br>
--------------e----P-- /mnt/tmp/dir1/hosts<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> # ls -li /mnt/tmp/dir1<br>
total 8<br>
655365 8 -rw-r--r--. 1 root root 7424 Oct 18 22:42 hosts<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> # strace mv /mnt/tmp/dir1/hosts /mnt/tmp/dir2/hosts<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> :<o:p></o:p></p>
<div>
<p class="MsoNormal"> renameat2(AT_FDCWD, "/mnt/tmp/dir1/hosts", AT_FDCWD, "/mnt/tmp/dir2/hosts", RENAME_NOREPLACE) = -1 EXDEV (Invalid cross-device link)<br>
stat("/mnt/tmp/dir2/hosts", 0x7ffff8a6c2b0) = -1 ENOENT (No such file or directory)<br>
lstat("/mnt/tmp/dir1/hosts", {st_mode=S_IFREG|0644, st_size=7424, ...}) = 0<br>
newfstatat(AT_FDCWD, "/mnt/tmp/dir2/hosts", 0x7ffff8a6bf90, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)<br>
unlink("/mnt/tmp/dir2/hosts") = -1 ENOENT (No such file or directory)<br>
openat(AT_FDCWD, "/mnt/tmp/dir1/hosts", O_RDONLY|O_NOFOLLOW) = 3<br>
openat(AT_FDCWD, "/mnt/tmp/dir2/hosts", O_WRONLY|O_CREAT|O_EXCL, 0600) = 4<br>
read(3, "##\n# Host Database\n#\n# Do not re"..., 131072) = 7424<br>
write(4, "##\n# Host Database\n#\n# Do not re"..., 7424) = 7424<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> :<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> # lsattr -p /mnt/tmp/dir2<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> 2000 --------------e----P-- /mnt/tmp/dir2/hosts<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> # ls -li /mnt/tmp/dir2<o:p></o:p></p>
</div>
<p class="MsoNormal"> total 8<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> 786435 8 -rw-r--r--. 1 root root 7424 Oct 18 22:42 hosts
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The reason for this limitation is that there is no way to atomically update the quota between the two project IDs when a whole subdirectory tree is being moved between projects. There might be thousands of subdirectories and millions of
files that are being moved, and the project ID needs to be updated on all of those files and directories. This is too large to do atomically in a single filesystem transaction. Rather than try to solve this directly in the kernel, the decision of the XFS
developers (copied by ext4) is that cross-project renames will not be done by the kernel and instead be handled in userspace by the "mv" utility, the same way that renames across different filesystems are handled.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In Lustre 2.15.0 and later, this cross-project rename constraint has been removed for *regular file* renames between directories with different project IDs. This means the file is moved between directories and the project ID and associated
quota accounting is updated in a single transaction without doing a data copy. However, *directory* renames with PROJINHERIT still have this issue.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">To work around this behavior, it is possible to use "chattr - p" (or "lfs project -p", they do the same thing) to change the project ID of the source files and directories *before* they are renamed so that the file data copy does not need
to be done, and just the filenames can be moved.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">It might be possible to patch "mv" so that instead of bailing on "rename()" after the first EXDEV return, it creates the target directory and then tries to rename the files within the source directory to the target, before it does the file
copy. It is likely that ext4 could also be patched to allow regular file renames without returning EXDEV.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Cheers, Andreas<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Andreas Dilger<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Lustre Principal Architect<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Whamcloud<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>