<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>
<blockquote type="cite" class="">
<div class="">On Jul 5, 2023, at 07:14, Mike Mosley via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Hello everyone,
<div class=""><br class="">
</div>
<div class="">We have drained some of our OSS/OSTs and plan to deactivate them soon. The process ahead leads us to a couple of questions that we hope somebody can advise us on.</div>
<div class=""><br class="">
</div>
<div class="">Scenario</div>
<div class="">We have fully drained the target OSTs using <b class=""> 'lfs find'</b> to identify all files located on the targets and then feeding the list to '<b class="">lfs migrate</b>. ' A final scan shows there are no files left on the targets.</div>
<div class=""><br class="">
</div>
<div class="">Questions</div>
<div class="">1) Running '<b class="">lfs df -h'</b> still shows some space being used even though we have drained all of the data. Is that normal? i.e.</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">UUID <span class="gmail-Apple-converted-space">
</span>bytes<span class="gmail-Apple-converted-space">
</span>Used <span class="gmail-Apple-converted-space"> </span>Available Use% Mounted on</span></div>
</div>
<div class="">
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo;" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font color="#1008b6" class="">hydra-OST0010_UUID
</font><span class="gmail-Apple-converted-space" style="color:rgb(16,8,182)">
</span><font color="#1008b6" class="">84.7T</font><span class="gmail-Apple-converted-space" style="color:rgb(16,8,182)">
</span><span style="background-color:rgb(255,255,0)" class=""><font color="#0b5394" class="">583.8M</font>
</span><span class="gmail-Apple-converted-space" style="color:rgb(16,8,182)">
</span><font color="#1008b6" class="">80.5T </font><span class="gmail-Apple-converted-space" style="color:rgb(16,8,182)">
</span><font color="#1008b6" class="">1% /dfs/hydra[OST:16]</font></span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST0011_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>581.4M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:17]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST0012_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>581.7M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:18]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST0013_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>582.4M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:19]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST0014_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>584.1M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:20]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST0015_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>583.4M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:21]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST0016_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>583.6M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:22]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST0017_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>581.8M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:23]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST0018_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>582.6M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:24]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST0019_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>582.7M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:25]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST001a_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>580.0M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:26]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST001b_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>580.4M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:27]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST001c_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>582.1M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:28]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST001d_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>583.2M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:29]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST001e_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>583.7M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:30]</span></div>
<div style="margin: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; -webkit-font-kerning: auto; font-feature-settings: normal; font-stretch: normal; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(16, 8, 182);" class="">
<span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">hydra-OST001f_UUID
<span class="gmail-Apple-converted-space"> </span>84.7T<span class="gmail-Apple-converted-space">
</span>587.7M <span class="gmail-Apple-converted-space"> </span>80.5T <span class="gmail-Apple-converted-space">
</span>1% /dfs/hydra[OST:31]</span></div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>I would suggest to unmount the OSTs from Lustre and mount via ldiskfs, then run "find $MOUNT/O -type f -ls" to find if there are any in-use files left. It is likely that the 580M used by all of the OSTs is just residual logs and large directories under
O/*. There might be some hundreds or thousands of zero-length object files that were precreated but never used, that will typically have an unusual file access mode 07666 and can be ignored.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">2) According to some comments, prior to deactivating the OSS/OSTs, we should add the
<b class="">'lazystatfs'</b> option to all of our client mounts so that they do not hang once we deactivate some of the OSTs. Is that correct? If so, why would you not just always have that option set? What are the ramifications of doing it well in advance
of the OST deactivations?</div>
</div>
</div>
</blockquote>
<br class="">
</div>
<div>The lazystatfs feature has been enabled by default since Lustre 2.9 so I don't think you need to do anything with it anymore. The "lfs df" command will automatically skip unconfigured OSTs.<br class="">
<div><br class="">
<blockquote type="cite" class=""></blockquote>
</div>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>