<p dir="ltr">I observed the same thing with 2.5. I believe it was the unlink object list not being pushed to OST once it came back active. But, I didn't see this issue with version 2.8. It released the space almost immediately after OSTs were activated. </p>
<br><div class="gmail_quote"><div dir="ltr">On Sun, Jul 16, 2017, 10:16 AM Bob Ball <<a href="mailto:ball@umich.edu">ball@umich.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
I agree with Raj. Also, I have noted with Lustre 2.7, that the
space is not actually freed after re-activation of the OST, until
the mgs is restarted. I don't recall the reason for this, or know
if this was fixed in later Lustre versions.<br>
<br>
Remember, this is done on the mgs, not on the clients. If you do it
on a client, the behavior is as you thought.</div><div text="#000000" bgcolor="#FFFFFF"><br>
<br>
bob</div><div text="#000000" bgcolor="#FFFFFF"><br>
<br>
<div class="m_4277987788743248970moz-cite-prefix">On 7/16/2017 11:10 AM, Raj wrote:<br>
</div>
<blockquote type="cite">
<p dir="ltr">No. Deactivating an OST will not allow to create new
objects(file). But, client can read AND modify an existing
objects(append the file). Also, it will not free any space from
deleted objects until the OST is activated again. </p>
<br>
<div class="gmail_quote">
<div dir="ltr">On Sun, Jul 16, 2017, 9:29 AM E.S. Rosenberg <<a href="mailto:esr%2Blustre@mail.hebrew.edu" target="_blank">esr+lustre@mail.hebrew.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Jul 13, 2017 at 5:49 AM,
Bob Ball <span dir="ltr"><<a href="mailto:ball@umich.edu" target="_blank">ball@umich.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> On the mgs/mgt
do something like:<br>
lctl --device <fsname>-OST0019-osc-MDT0000
deactivate<br>
<br>
No further files will be assigned to that OST.
Reverse with "activate". Or reboot the mgs/mdt as
this is not persistent. "lctl dl" will tell you
exactly what that device name should be for you.<span class="m_4277987788743248970m_-1197213167015817042HOEnZb"><font color="#888888"><br>
</font></span></div>
</blockquote>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Doesn't that also disable reads from the OST
though? <br>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span class="m_4277987788743248970m_-1197213167015817042HOEnZb"><font color="#888888"> <br>
bob</font></span>
<div>
<div class="m_4277987788743248970m_-1197213167015817042h5"><br>
<br>
<div class="m_4277987788743248970m_-1197213167015817042m_4682020740092110866moz-cite-prefix">On
7/12/2017 6:04 PM, Alexander I Kulyavtsev
wrote:<br>
</div>
<blockquote type="cite">
<div>You may find advise from Andreas on this
list (also attached below). I did not try
setting fail_loc myself.</div>
<div><br>
</div>
<div>In 2.9 there is setting <span style="color:rgb(112,112,112);font-family:monospace;font-size:14px;background-color:rgb(255,255,255)">osp.*.max_create_count=0</span><font face="Arial, sans-serif" color="#707070"><span style="font-size:14px;background-color:rgb(255,255,255)"> </span></font>described
at LUDOC-305.</div>
<div><br>
</div>
<div>We used to set OST degraded as described
in lustre manual. </div>
<div>It works most of the time but at some
point I saw lustre errors in logs for some
ops. Sorry, I do not recall details.</div>
<div><br>
</div>
<div>I still not sure either of these
approaches will work for you: setting OST
degraded or fail_loc will makes some osts
selected instead of others.</div>
<div>You may want to verify if these settings
will trigger clean error on user side
(instead of blocking) when all OSTs are
degraded.</div>
<div><br>
</div>
<div>The other and also simpler approach would
be to enable lustre quota and set quota
below used space for all users (or groups).</div>
<div><br>
</div>
<div>Alex.</div>
<div><br>
<blockquote type="cite">
<div style="margin:0px"><span><b>From: </b></span><span>"Dilger,
Andreas" <<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>><br>
</span></div>
<div style="margin:0px"><span><b>Subject: </b></span><span><b>Re:
[lustre-discuss] lustre 2.5.3 ost
not draining</b><br>
</span></div>
<div style="margin:0px"><span><b>Date: </b></span><span>July 28,
2015 at 11:51:38 PM CDT</span></div>
<div style="margin:0px"><span><b>Cc: </b></span><span>"<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>"
<<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
</span></div>
<br>
<div><span style="float:none;display:inline!important">Setting
it degraded means the MDS will avoid
allocations on that OST</span><br>
<span style="float:none;display:inline!important">unless
there aren't enough OSTs to meet the
request (e.g. stripe_count =</span><br>
<span style="float:none;display:inline!important">-1),
so it should work.</span><br>
<br>
<span style="float:none;display:inline!important">That
is actually a very interesting
workaround for this problem, and it</span><br>
<span style="float:none;display:inline!important">will
work for older versions of Lustre as
well. It doesn't disable the</span><br>
<span style="float:none;display:inline!important">OST
completely, which is fine if you are
doing space balancing (and may</span><br>
<span style="float:none;display:inline!important">even
be desirable to allow apps that need
more bandwidth for a widely</span><br>
<span style="float:none;display:inline!important">striped
file), but it isn't good if you are
trying to empty the OST</span><br>
<span style="float:none;display:inline!important">completely
to remove it.</span><br>
<br>
<span style="float:none;display:inline!important">It
looks like another approach would be
to mark the OST as having no free</span><br>
<span style="float:none;display:inline!important">space
using OBD_FAIL_OST_ENOINO (0x229)
fault injection on that OST:</span><br>
<br>
<span style="float:none;display:inline!important"> lctl
set_param fail_loc=0x229
fail_val=<ost_index></span><br>
<br>
<span style="float:none;display:inline!important">This
would cause the OST to return 0 free
inodes from OST_STATFS for the</span><br>
<span style="float:none;display:inline!important">specified
OST index, and the MDT would skip this
OST completely. To</span><br>
<span style="float:none;display:inline!important">disable
all of the OSTs on an OSS use
<ost_index> = -1. It isn't
possible</span><br>
<span style="float:none;display:inline!important">to
selectively disable a subset of OSTs
using this method. The</span><br>
<span style="float:none;display:inline!important">OBD_FAIL_OST_ENOINO
fail_loc has been available since
Lustre 2.2, which</span><br>
<span style="float:none;display:inline!important">covers
all of the 2.4+ versions that are
affected by this issue.</span><br>
<br>
<span style="float:none;display:inline!important">If
this mechanism works for you (it
should, as this fail_loc is used</span><br>
<span style="float:none;display:inline!important">during
regular testing) I'd be obliged if
someone could file an LUDOC bug</span><br>
<span style="float:none;display:inline!important">so
the manual can be updated.</span><br>
<br>
<span style="float:none;display:inline!important">Cheers,
Andreas</span></div>
</blockquote>
<br>
</div>
<br>
<div>
<blockquote type="cite">
<div>On Jul 12, 2017, at 4:20 PM, Riccardo
Veraldi <<a href="mailto:Riccardo.Veraldi@cnaf.infn.it" target="_blank">Riccardo.Veraldi@cnaf.infn.it</a>>
wrote:</div>
<br class="m_4277987788743248970m_-1197213167015817042m_4682020740092110866Apple-interchange-newline">
<div>
<div>Hello,<br>
<br>
on one of my lustre FS I need to find
a solution so that users can still<br>
access data on the FS but cannot write
new files on it.<br>
I have hundreds of clients accessing
the FS so remounting it ro is not<br>
really easily feasible.<br>
Is there an option on the OSS side to
allow OSTs to be accessed just to<br>
read data and not to store new data ?<br>
tunefs.lustre could do that ?<br>
thank you<br>
<br>
Rick<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a class="m_4277987788743248970m_-1197213167015817042m_4682020740092110866moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br>
<fieldset class="m_4277987788743248970m_-1197213167015817042m_4682020740092110866mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
lustre-discuss mailing list
<a class="m_4277987788743248970m_-1197213167015817042m_4682020740092110866moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>
<a class="m_4277987788743248970m_-1197213167015817042m_4682020740092110866moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br>
</blockquote>
</div>
</div>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote>
</div>
</blockquote>
<br>
</div></blockquote></div>