<div dir="ltr">Andreas,<div><br></div><div>Thank you for the information. We appreciate it.</div><div><br></div><div>Mike</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 5, 2023 at 8:46 PM Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">[<strong><font color="red">Caution</font></strong>: Email from External Sender. Do not click or open links or attachments unless you know this sender.]<br><span> </span>
<div>
<blockquote type="cite">
<div>On Jul 5, 2023, at 07:14, Mike Mosley via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br>
<div>
<div dir="ltr">Hello everyone,
<div><br>
</div>
<div>We have drained some of our OSS/OSTs and plan to deactivate them soon. The process ahead leads us to a couple of questions that we hope somebody can advise us on.</div>
<div><br>
</div>
<div>Scenario</div>
<div>We have fully drained the target OSTs using <b> 'lfs find'</b> to identify all files located on the targets and then feeding the list to '<b>lfs migrate</b>. ' A final scan shows there are no files left on the targets.</div>
<div><br>
</div>
<div>Questions</div>
<div>1) Running '<b>lfs df -h'</b> still shows some space being used even though we have drained all of the data. Is that normal? i.e.</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">UUID <span>
</span>bytes<span>
</span>Used <span> </span>Available Use% Mounted on</span></div>
</div>
<div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo">
<span style="font-variant-ligatures:no-common-ligatures"><font color="#1008b6">hydra-OST0010_UUID
</font><span style="color:rgb(16,8,182)">
</span><font color="#1008b6">84.7T</font><span style="color:rgb(16,8,182)">
</span><span style="background-color:rgb(255,255,0)"><font color="#0b5394">583.8M</font>
</span><span style="color:rgb(16,8,182)">
</span><font color="#1008b6">80.5T </font><span style="color:rgb(16,8,182)">
</span><font color="#1008b6">1% /dfs/hydra[OST:16]</font></span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST0011_UUID
<span> </span>84.7T<span>
</span>581.4M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:17]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST0012_UUID
<span> </span>84.7T<span>
</span>581.7M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:18]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST0013_UUID
<span> </span>84.7T<span>
</span>582.4M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:19]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST0014_UUID
<span> </span>84.7T<span>
</span>584.1M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:20]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST0015_UUID
<span> </span>84.7T<span>
</span>583.4M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:21]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST0016_UUID
<span> </span>84.7T<span>
</span>583.6M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:22]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST0017_UUID
<span> </span>84.7T<span>
</span>581.8M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:23]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST0018_UUID
<span> </span>84.7T<span>
</span>582.6M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:24]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST0019_UUID
<span> </span>84.7T<span>
</span>582.7M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:25]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST001a_UUID
<span> </span>84.7T<span>
</span>580.0M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:26]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST001b_UUID
<span> </span>84.7T<span>
</span>580.4M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:27]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST001c_UUID
<span> </span>84.7T<span>
</span>582.1M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:28]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST001d_UUID
<span> </span>84.7T<span>
</span>583.2M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:29]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST001e_UUID
<span> </span>84.7T<span>
</span>583.7M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:30]</span></div>
<div style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)">
<span style="font-variant-ligatures:no-common-ligatures">hydra-OST001f_UUID
<span> </span>84.7T<span>
</span>587.7M <span> </span>80.5T <span>
</span>1% /dfs/hydra[OST:31]</span></div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I would suggest to unmount the OSTs from Lustre and mount via ldiskfs, then run "find $MOUNT/O -type f -ls" to find if there are any in-use files left. It is likely that the 580M used by all of the OSTs is just residual logs and large directories under
O/*. There might be some hundreds or thousands of zero-length object files that were precreated but never used, that will typically have an unusual file access mode 07666 and can be ignored.</div>
<br>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>2) According to some comments, prior to deactivating the OSS/OSTs, we should add the
<b>'lazystatfs'</b> option to all of our client mounts so that they do not hang once we deactivate some of the OSTs. Is that correct? If so, why would you not just always have that option set? What are the ramifications of doing it well in advance
of the OST deactivations?</div>
</div>
</div>
</blockquote>
<br>
</div>
<div>The lazystatfs feature has been enabled by default since Lustre 2.9 so I don't think you need to do anything with it anymore. The "lfs df" command will automatically skip unconfigured OSTs.<br>
<div><br>
<blockquote type="cite"></blockquote>
</div>
</div>
<br>
<div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<br>
</div>
<br>
</div>
</blockquote></div>