<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body dir="auto">
<div>If you mark the OSTs degraded and the MDS will still avoid them for new allocations, though you should make a second scanning pass to verify. </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">Deactivating the OSTs on the MDS will allow it to delete the (now unused) OST objects. <br>
<br>
Cheers, Andreas</div>
<div><br>
On Mar 20, 2017, at 17:03, E.S. Rosenberg <<a href="mailto:esr@cs.huji.ac.il">esr@cs.huji.ac.il</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Mar 20, 2017 at 10:59 PM, Dilger, Andreas <span dir="ltr">
<<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>If you've marked the OST inactive on the MDS then that is not surprising. See <a href="https://jira.hpdd.intel.com/browse/LU-4825" target="_blank">https://jira.hpdd.intel.<wbr>com/browse/LU-4825</a> and the comments in my previous email. <br>
</div>
</div>
</blockquote>
<div>Ah OK.<br>
</div>
<div>But if I re-activate the OST will lfs_migrate still move them away from the device?
<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div><br>
Cheers, Andreas</div>
<div>
<div class="h5">
<div><br>
On Mar 20, 2017, at 16:56, E.S. Rosenberg <<a href="mailto:esr@cs.huji.ac.il" target="_blank">esr@cs.huji.ac.il</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Mar 20, 2017 at 10:50 PM, E.S. Rosenberg <span dir="ltr">
<<a href="mailto:esr+lustre@mail.hebrew.edu" target="_blank">esr+lustre@mail.hebrew.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span>On Mon, Mar 20, 2017 at 10:19 PM, Dilger, Andreas <span dir="ltr">
<<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
<div dir="auto">
<div>The underlying "lfs migrate" command (not the "lfs_migrate" script) in newer Lustre versions (2.9) is capable of migrating files that are in use by using the "--block" option, which prevents other processes from accessing or modifying the file during migration.</div>
<div><br>
</div>
<div>Unfortunately, "lfs_migrate" doesn't pass that argument on, though it wouldn't be hard to change the script. Ideally, the "lfs_migrate" script would pass all unknown options to "lfs migrate". </div>
<div><br>
</div>
<div><br>
</div>
<div>The other item of note is that setting the OST inactive on the MDS will prevent the MDS from deleting objects on the OST (see <a href="https://jira.hpdd.intel.com/browse/LU-4825" target="_blank">https://jira.hpdd.intel.c<wbr>om/browse/LU-4825</a> for details).
In Lustre 2.9 and later it is possible to set on the MDS:</div>
<div><br>
</div>
<div> mds# lctl set_param osp.<OST>.create_count=0</div>
<div><br>
</div>
<div>to stop MDS allocation of new objects on that OST. On older versions it is possible to set on the OSS:</div>
<div><br>
oss# lctl set_param obdfilter.<OST>.degraded=1</div>
<div><br>
</div>
<div>so that it tells the MDS to avoid it if possible, but this isn't a hard exclusion.</div>
<div><br>
</div>
<div>It is also possible to use a testing hack to mark an OST as out of inodes, but that only works for one OST per OSS and it sounds like that won't be useful in this case. </div>
<div><br>
<div>Cheers, Andreas</div>
</div>
</div>
</blockquote>
</span>
<div>You're making me want Lustre 2.9 more :) but for now I'm still stuck on 2.8 and because this is very much production these days I'm more careful with the update (hoping to finally get hw allocated for a test env soon to test the update).<br>
</div>
<div>Thanks,<br>
</div>
<div>Eli <br>
</div>
</div>
</div>
</div>
</blockquote>
<div>Another related question:<br>
</div>
<div>The migration has been running for several hours now on one OST but I am yet to see 1 block being freed from the OSS point of view, is this not mv but cp as far as the original OST is concerned?<br>
</div>
<div>(also man lfs has no lfs migrate that was added in 2.9?)<br>
</div>
<div>Thanks,<br>
</div>
<div>Eli<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div></div>
<div>
<div class="m_-2011267084321588191h5">
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
<div dir="auto">
<div></div>
<div>
<div class="m_-2011267084321588191m_-7665972089601897516gmail-h5">
<div><br>
On Mar 20, 2017, at 13:11, Brett Lee <<a href="mailto:brettlee.lustre@gmail.com" target="_blank">brettlee.lustre@gmail.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>Hi Eli,<br>
<br>
I believe that is still the case with lfs_migrate. If otherwise, we'll probably hear soon.<br>
<br>
You should be able to disable those OSTs while keeping the file system active - via a command on the MDS(s) as well as the clients. My notes have the command as shown below, but please confirm via the appropriate Lustre manual:<br>
<br>
</div>
lctl set_param osc.<fsname>-<OST00xy>-*.activ<wbr>e=0<br>
<div>
<div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">
<div>
<div class="m_-2011267084321588191m_-7665972089601897516gmail-m_2989405016505302045gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Brett</div>
<div>--<br>
</div>
<div dir="ltr"><span style="font-size:small">Protect Yourself </span><span style="font-size:small"><span><span style="font-size:small">Against</span></span> Cybercrime</span><br>
<div>
<div>PDS Software Solutions LLC</div>
<div><a href="https://www.trustpds.com/" target="_blank">https://www.TrustPDS.com</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Mon, Mar 20, 2017 at 10:43 AM, E.S. Rosenberg <span dir="ltr">
<<a href="mailto:esr+lustre@mail.hebrew.edu" target="_blank">esr+lustre@mail.hebrew.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
<div dir="ltr">
<div>
<div>
<div>
<div>In the man page it says the following:<br>
<br>
Because lfs_migrate is not yet closely integrated with the MDS, it cannot determine whether a file is currently open and/or in-use by other applications or nodes. That makes it UNSAFE for use on files that might be modified by other applications, since
the migrated file is only a copy of the current file, and this will result in the old file becoming an open-unlinked file and any modifications to that file will be lost.<br>
<br>
</div>
Is this still the case?<br>
</div>
Is there a better way to disable OSTs while keeping the filesystem live?<br>
<br>
</div>
<div>Background:<br>
</div>
<div>We need to take a OSS enclosure that hosts multiple OSTs offline for hardware maintenance, I'd like to do this without bringing Lustre as a whole down. I made sure there is enough space on the other OSTs to house the contents of the machine going offline
and am now about to move things.<br>
</div>
<div><br>
</div>
Thanks,<br>
</div>
Eli<br>
</div>
<br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.or<wbr>g</a><br>
<a rel="noreferrer" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listin<wbr>fo.cgi/lustre-discuss-lustre.o<wbr>rg</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>______________________________<wbr>_________________</span><br>
<span>lustre-discuss mailing list</span><br>
<span><a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.or<wbr>g</a></span><br>
<span><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listin<wbr>fo.cgi/lustre-discuss-lustre.o<wbr>rg</a></span><br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.<wbr>org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/<wbr>listinfo.cgi/lustre-discuss-<wbr>lustre.org</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</body>
</html>