<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Thanks for all this, Andreas. Always appreciated.<br>
<br>
bob<br>
<br>
<div class="moz-cite-prefix">On 7/17/2017 12:00 AM, Dilger, Andreas
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:8F568344-5C28-4EC1-A2A5-6B8A07FD48D9@intel.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div>When you write "MGS", you really mean "MDS". The MGS would be
the place for this if you were changing the config to
permanently deactivate the OSTs via "lctl conf_param". To
temporarily do this, the commands should be run on the MDS via
"lctl set_param". In most cases the MDS and MGS are co-located,
so the distinction is irrelevant, but good to get it right for
the record. </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">The problem of objects not being
unlinked until after the MDS is restarted has been fixed. </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">Also, with 2.9 and later it is
possible to use "lctl set_param osp.<OST>.create_count=0"
to stop new file allocation on that OST without blocking
unlinked at all, which is best for emptying old OSTs, rather
than using "deactivate". </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">For marking the OSTs read-only, both
of these solutions will not prevent clients from modifying the
OST filesystems, just from creating new files (assuming all OSTs
are set this way). </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">You might consider to try "mount -o
remount,ro" on the MDT and OST filesystems on the servers to see
if this works (I haven't tested this myself). The problem might
be that this prevents new clients from mounting. </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">It probably makes sense to add
server-side read-only mounting as a feature. Could you please
file a ticket in Jira about this?<br>
<br>
Cheers, Andreas</div>
<div><br>
On Jul 16, 2017, at 09:16, Bob Ball <<a
href="mailto:ball@umich.edu" moz-do-not-send="true">ball@umich.edu</a>>
wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>I agree with Raj. Also, I have noted with Lustre 2.7, that
the space is not actually freed after re-activation of the
OST, until the mgs is restarted. I don't recall the reason
for this, or know if this was fixed in later Lustre versions.<br>
<br>
Remember, this is done on the mgs, not on the clients. If you
do it on a client, the behavior is as you thought.<br>
<br>
bob<br>
<br>
<div class="moz-cite-prefix">On 7/16/2017 11:10 AM, Raj wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CANF66k-n2tJtiwy89apr_zvD_Tn20z31LBGQrXrOt0yGOceYdA@mail.gmail.com">
<p dir="ltr">No. Deactivating an OST will not allow to
create new objects(file). But, client can read AND modify
an existing objects(append the file). Also, it will not
free any space from deleted objects until the OST is
activated again.
</p>
<br>
<div class="gmail_quote">
<div dir="ltr">On Sun, Jul 16, 2017, 9:29 AM E.S.
Rosenberg <<a
href="mailto:esr%2Blustre@mail.hebrew.edu"
moz-do-not-send="true">esr+lustre@mail.hebrew.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Jul 13, 2017 at
5:49 AM, Bob Ball <span dir="ltr">
<<a href="mailto:ball@umich.edu"
target="_blank" moz-do-not-send="true">ball@umich.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">On the
mgs/mgt do something like:<br>
lctl --device
<fsname>-OST0019-osc-MDT0000 deactivate<br>
<br>
No further files will be assigned to that
OST. Reverse with "activate". Or reboot the
mgs/mdt as this is not persistent. "lctl dl"
will tell you exactly what that device name
should be for you.<span
class="m_-1197213167015817042HOEnZb"><font
color="#888888"><br>
</font></span></div>
</blockquote>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Doesn't that also disable reads from the OST
though? <br>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span
class="m_-1197213167015817042HOEnZb"><font
color="#888888"><br>
bob</font></span>
<div>
<div class="m_-1197213167015817042h5"><br>
<br>
<div
class="m_-1197213167015817042m_4682020740092110866moz-cite-prefix">On
7/12/2017 6:04 PM, Alexander I
Kulyavtsev wrote:<br>
</div>
<blockquote type="cite">
<div>You may find advise from Andreas on
this list (also attached below). I did
not try setting fail_loc myself.</div>
<div><br>
</div>
<div>In 2.9 there is setting <span
style="color:rgb(112,112,112);font-family:monospace;font-size:14px;background-color:rgb(255,255,255)">osp.*.max_create_count=0</span><font
face="Arial, sans-serif"
color="#707070"><span
style="font-size:14px;background-color:rgb(255,255,255)"> </span></font>described
at LUDOC-305.</div>
<div><br>
</div>
<div>We used to set OST degraded as
described in lustre manual. </div>
<div>It works most of the time but at
some point I saw lustre errors in logs
for some ops. Sorry, I do not recall
details.</div>
<div><br>
</div>
<div>I still not sure either of these
approaches will work for you: setting
OST degraded or fail_loc will makes
some osts selected instead of others.</div>
<div>You may want to verify if these
settings will trigger clean error on
user side (instead of blocking) when
all OSTs are degraded.</div>
<div><br>
</div>
<div>The other and also simpler approach
would be to enable lustre quota and
set quota below used space for all
users (or groups).</div>
<div><br>
</div>
<div>Alex.</div>
<div><br>
<blockquote type="cite">
<div style="margin:0px"><span
style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif"><b>From: </b></span><span
style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif">"Dilger,
Andreas" <<a
href="mailto:andreas.dilger@intel.com"
target="_blank"
moz-do-not-send="true">andreas.dilger@intel.com</a>><br>
</span></div>
<div style="margin:0px"><span
style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif"><b>Subject: </b></span><span
style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif"><b>Re:
[lustre-discuss] lustre 2.5.3
ost not draining</b><br>
</span></div>
<div style="margin:0px"><span
style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif"><b>Date: </b></span><span
style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif">July
28, 2015 at 11:51:38 PM CDT</span></div>
<div style="margin:0px"><span
style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif"><b>Cc: </b></span><span
style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif">"<a
href="mailto:lustre-discuss@lists.lustre.org" target="_blank"
moz-do-not-send="true">lustre-discuss@lists.lustre.org</a>"
<<a
href="mailto:lustre-discuss@lists.lustre.org"
target="_blank"
moz-do-not-send="true">lustre-discuss@lists.lustre.org</a>><br>
</span></div>
<br>
<div><span
style="float:none;display:inline!important">Setting
it degraded means the MDS will
avoid allocations on that OST</span><br>
<span
style="float:none;display:inline!important">unless
there aren't enough OSTs to meet
the request (e.g. stripe_count =</span><br>
<span
style="float:none;display:inline!important">-1),
so it should work.</span><br>
<br>
<span
style="float:none;display:inline!important">That
is actually a very interesting
workaround for this problem, and
it</span><br>
<span
style="float:none;display:inline!important">will
work for older versions of
Lustre as well. It doesn't
disable the</span><br>
<span
style="float:none;display:inline!important">OST
completely, which is fine if you
are doing space balancing (and
may</span><br>
<span
style="float:none;display:inline!important">even
be desirable to allow apps that
need more bandwidth for a widely</span><br>
<span
style="float:none;display:inline!important">striped
file), but it isn't good if you
are trying to empty the OST</span><br>
<span
style="float:none;display:inline!important">completely
to remove it.</span><br>
<br>
<span
style="float:none;display:inline!important">It
looks like another approach
would be to mark the OST as
having no free</span><br>
<span
style="float:none;display:inline!important">space
using OBD_FAIL_OST_ENOINO
(0x229) fault injection on that
OST:</span><br>
<br>
<span
style="float:none;display:inline!important"> lctl
set_param fail_loc=0x229
fail_val=<ost_index></span><br>
<br>
<span
style="float:none;display:inline!important">This
would cause the OST to return 0
free inodes from OST_STATFS for
the</span><br>
<span
style="float:none;display:inline!important">specified
OST index, and the MDT would
skip this OST completely. To</span><br>
<span
style="float:none;display:inline!important">disable
all of the OSTs on an OSS use
<ost_index> = -1. It
isn't possible</span><br>
<span
style="float:none;display:inline!important">to
selectively disable a subset of
OSTs using this method. The</span><br>
<span
style="float:none;display:inline!important">OBD_FAIL_OST_ENOINO
fail_loc has been available
since Lustre 2.2, which</span><br>
<span
style="float:none;display:inline!important">covers
all of the 2.4+ versions that
are affected by this issue.</span><br>
<br>
<span
style="float:none;display:inline!important">If
this mechanism works for you (it
should, as this fail_loc is used</span><br>
<span
style="float:none;display:inline!important">during
regular testing) I'd be obliged
if someone could file an LUDOC
bug</span><br>
<span
style="float:none;display:inline!important">so
the manual can be updated.</span><br>
<br>
<span
style="float:none;display:inline!important">Cheers,
Andreas</span></div>
</blockquote>
<br>
</div>
<br>
<div>
<blockquote type="cite">
<div>On Jul 12, 2017, at 4:20 PM,
Riccardo Veraldi <<a
href="mailto:Riccardo.Veraldi@cnaf.infn.it"
target="_blank"
moz-do-not-send="true">Riccardo.Veraldi@cnaf.infn.it</a>>
wrote:</div>
<br
class="m_-1197213167015817042m_4682020740092110866Apple-interchange-newline">
<div>
<div>Hello,<br>
<br>
on one of my lustre FS I need to
find a solution so that users
can still<br>
access data on the FS but cannot
write new files on it.<br>
I have hundreds of clients
accessing the FS so remounting
it ro is not<br>
really easily feasible.<br>
Is there an option on the OSS
side to allow OSTs to be
accessed just to<br>
read data and not to store new
data ?<br>
tunefs.lustre could do that ?<br>
thank you<br>
<br>
Rick<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a
href="mailto:lustre-discuss@lists.lustre.org"
target="_blank"
moz-do-not-send="true">lustre-discuss@lists.lustre.org</a><br>
<a
class="m_-1197213167015817042m_4682020740092110866moz-txt-link-freetext"
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
target="_blank"
moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br>
<fieldset
class="m_-1197213167015817042m_4682020740092110866mimeAttachmentHeader">
</fieldset>
<br>
<pre>_______________________________________________
lustre-discuss mailing list
<a class="m_-1197213167015817042m_4682020740092110866moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org" target="_blank" moz-do-not-send="true">lustre-discuss@lists.lustre.org</a>
<a class="m_-1197213167015817042m_4682020740092110866moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank" moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org"
target="_blank" moz-do-not-send="true">lustre-discuss@lists.lustre.org</a><br>
<a
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br>
</blockquote>
</div>
</div>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org"
target="_blank" moz-do-not-send="true">lustre-discuss@lists.lustre.org</a><br>
<a
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote>
</div>
</blockquote>
<br>
</div>
</blockquote>
<blockquote type="cite">
<div><span>_______________________________________________</span><br>
<span>lustre-discuss mailing list</span><br>
<span><a href="mailto:lustre-discuss@lists.lustre.org"
moz-do-not-send="true">lustre-discuss@lists.lustre.org</a></span><br>
<span><a
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span><br>
</div>
</blockquote>
</blockquote>
<br>
</body>
</html>