<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
The DNE auto-split functionality is disabled by default and not fully completed (e.g. preserve inode numbers) because it had issues with significant performance impact/latency while splitting a directory that was currently in use (which is exactly when you
would want to use it), so I wouldn't recommend to use it at this time.
<div class=""><br class="">
</div>
<div class="">Instead, development efforts were focussed on DNE MDT space balancing. This adds two different features that allow all of the MDTs in a filesystem to be used without user/admin intervention (though it is still possible to manually create directories
on specific MDTs as before).</div>
<div class=""><br class="">
</div>
<div class="">The "round-robin" MDT selection ("lfs setdirstripe -D --max-depth-rr=N -c 1 -i -1") for top-level directories (enabled for the top 3 levels of the filesystem by default) will, as the name suggests, round robin new directories across all of the
available MDTs, when their space is evenly balanced (within 5% free space*inodes by default). That is important to distribute *new* directories across MDTs in new filesystems when e.g. .../home/$user or .../project/$project or .../scratch/$user are being
created. </div>
<div class=""><br class="">
</div>
<div class="">The "space balance" MDT selection ("lctl set_param lmv.*.qos_threshold_rr=N" on the *CLIENT*) kicks in when MDT space usage becomes imbalanced (free space*inodes difference above 5% by default), and then starts selecting the MDT for *new* directories
based on the ratio of free space*inodes. That allows the MDTs to return toward balance over time, without causing a performance imbalance when it isn't necessary.</div>
<div class=""><br class="">
</div>
<div class="">Note that both of these heuristics operate on *single-stripe directories* and not regular files, so the MDT balance will not be perfect if some directory tree has millions more files/subdirectories than another. However, the main issue being
avoided is the *very* common case of MDT0000 getting full and MDT0001..N being (almost) totally unused. These features also make the MDT *usage* balance also pretty good as a result, so it is a win-win. For most filesystems, the MDT capacity is not the
limiting factor (it only makes up a few percent of the total storage).</div>
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Mar 23, 2023, at 15:31, Bertschinger, Thomas Andrew Hjorth via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hello,<br class="">
<br class="">
We've been experimenting with DNEv3 recently and have run into this issue: <a href="https://jira.whamcloud.com/browse/LU-7607" class="">
https://jira.whamcloud.com/browse/LU-7607</a> where the directory inode number changes after auto-split.<br class="">
<br class="">
In addition to the problem noted with backups that track the inode number, we have found that file access through a previously open file descriptor is broken post migration. This can occur when a shell's CWD is the affected directory. For example:<br class="">
<br class="">
mds0 # lctl get_param mdt.mylustre-MDT0000.{dir_split_count,enable_dir_auto_split}<br class="">
mdt.mylustre-MDT0000.dir_split_count=100<br class="">
mdt.mylustre-MDT0000.enable_dir_auto_split=1<br class="">
<br class="">
client $ pwd<br class="">
/mnt/mylustre/dnetest<br class="">
client $ for i in {0..100}; do touch file$i; done<br class="">
client $ ls<br class="">
ls: cannot open directory '.': Operation not permitted<br class="">
client $ ls file0<br class="">
ls: cannot access 'file0': No such file or directory<br class="">
client $ ls /mnt/mylustre/dnetest/file0<br class="">
/mnt/mylustre/dnetest/file0<br class="">
<br class="">
(This is from a build of the current master branch.)<br class="">
<br class="">
We believe users will certainly encounter this, because users monitor output directories of jobs as they run. Therefore this issue is a dealbreaker with DNEv3 for us.<br class="">
<br class="">
I wanted to ask about the status of the linked issue, since it looks like it hasn't been updated in a while. Would the resolution to LU-7607 be expected to fix the file access problem I've noted here or will this require additional changes to resolve?
<br class="">
<br class="">
Thanks!<br class="">
<br class="">
- Thomas Bertschinger<br class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>