<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body dir="auto">
<div>When you write "MGS", you really mean "MDS". The MGS would be the place for this if you were changing the config to permanently deactivate the OSTs via "lctl conf_param". To temporarily do this, the commands should be run on the MDS via "lctl set_param".
In most cases the MDS and MGS are co-located, so the distinction is irrelevant, but good to get it right for the record. </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">The problem of objects not being unlinked until after the MDS is restarted has been fixed. </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">Also, with 2.9 and later it is possible to use "lctl set_param osp.<OST>.create_count=0" to stop new file allocation on that OST without blocking unlinked at all, which is best for emptying old OSTs, rather than using "deactivate". </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">For marking the OSTs read-only, both of these solutions will not prevent clients from modifying the OST filesystems, just from creating new files (assuming all OSTs are set this way). </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">You might consider to try "mount -o remount,ro" on the MDT and OST filesystems on the servers to see if this works (I haven't tested this myself). The problem might be that this prevents new clients from mounting. </div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">It probably makes sense to add server-side read-only mounting as a feature. Could you please file a ticket in Jira about this?<br>
<br>
Cheers, Andreas</div>
<div><br>
On Jul 16, 2017, at 09:16, Bob Ball <<a href="mailto:ball@umich.edu">ball@umich.edu</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>I agree with Raj. Also, I have noted with Lustre 2.7, that the space is not actually freed after re-activation of the OST, until the mgs is restarted. I don't recall the reason for this, or know if this was fixed in later Lustre versions.<br>
<br>
Remember, this is done on the mgs, not on the clients. If you do it on a client, the behavior is as you thought.<br>
<br>
bob<br>
<br>
<div class="moz-cite-prefix">On 7/16/2017 11:10 AM, Raj wrote:<br>
</div>
<blockquote type="cite" cite="mid:CANF66k-n2tJtiwy89apr_zvD_Tn20z31LBGQrXrOt0yGOceYdA@mail.gmail.com">
<p dir="ltr">No. Deactivating an OST will not allow to create new objects(file). But, client can read AND modify an existing objects(append the file). Also, it will not free any space from deleted objects until the OST is activated again.
</p>
<br>
<div class="gmail_quote">
<div dir="ltr">On Sun, Jul 16, 2017, 9:29 AM E.S. Rosenberg <<a href="mailto:esr%2Blustre@mail.hebrew.edu" moz-do-not-send="true">esr+lustre@mail.hebrew.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Jul 13, 2017 at 5:49 AM, Bob Ball <span dir="ltr">
<<a href="mailto:ball@umich.edu" target="_blank" moz-do-not-send="true">ball@umich.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">On the mgs/mgt do something like:<br>
lctl --device <fsname>-OST0019-osc-MDT0000 deactivate<br>
<br>
No further files will be assigned to that OST. Reverse with "activate". Or reboot the mgs/mdt as this is not persistent. "lctl dl" will tell you exactly what that device name should be for you.<span class="m_-1197213167015817042HOEnZb"><font color="#888888"><br>
</font></span></div>
</blockquote>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Doesn't that also disable reads from the OST though? <br>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span class="m_-1197213167015817042HOEnZb"><font color="#888888"><br>
bob</font></span>
<div>
<div class="m_-1197213167015817042h5"><br>
<br>
<div class="m_-1197213167015817042m_4682020740092110866moz-cite-prefix">On 7/12/2017 6:04 PM, Alexander I Kulyavtsev wrote:<br>
</div>
<blockquote type="cite">
<div>You may find advise from Andreas on this list (also attached below). I did not try setting fail_loc myself.</div>
<div><br>
</div>
<div>In 2.9 there is setting <span style="color:rgb(112,112,112);font-family:monospace;font-size:14px;background-color:rgb(255,255,255)">osp.*.max_create_count=0</span><font face="Arial, sans-serif" color="#707070"><span style="font-size:14px;background-color:rgb(255,255,255)"> </span></font>described
at LUDOC-305.</div>
<div><br>
</div>
<div>We used to set OST degraded as described in lustre manual. </div>
<div>It works most of the time but at some point I saw lustre errors in logs for some ops. Sorry, I do not recall details.</div>
<div><br>
</div>
<div>I still not sure either of these approaches will work for you: setting OST degraded or fail_loc will makes some osts selected instead of others.</div>
<div>You may want to verify if these settings will trigger clean error on user side (instead of blocking) when all OSTs are degraded.</div>
<div><br>
</div>
<div>The other and also simpler approach would be to enable lustre quota and set quota below used space for all users (or groups).</div>
<div><br>
</div>
<div>Alex.</div>
<div><br>
<blockquote type="cite">
<div style="margin:0px"><span style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif"><b>From: </b></span><span style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif">"Dilger,
Andreas" <<a href="mailto:andreas.dilger@intel.com" target="_blank" moz-do-not-send="true">andreas.dilger@intel.com</a>><br>
</span></div>
<div style="margin:0px"><span style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif"><b>Subject: </b></span><span style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif"><b>Re:
[lustre-discuss] lustre 2.5.3 ost not draining</b><br>
</span></div>
<div style="margin:0px"><span style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif"><b>Date: </b></span><span style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif">July
28, 2015 at 11:51:38 PM CDT</span></div>
<div style="margin:0px"><span style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif"><b>Cc: </b></span><span style="font-family:-webkit-system-font,'Helvetica
Neue',Helvetica,sans-serif">"<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" moz-do-not-send="true">lustre-discuss@lists.lustre.org</a>"
<<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" moz-do-not-send="true">lustre-discuss@lists.lustre.org</a>><br>
</span></div>
<br>
<div><span style="float:none;display:inline!important">Setting it degraded means the MDS will avoid allocations on that OST</span><br>
<span style="float:none;display:inline!important">unless there aren't enough OSTs to meet the request (e.g. stripe_count =</span><br>
<span style="float:none;display:inline!important">-1), so it should work.</span><br>
<br>
<span style="float:none;display:inline!important">That is actually a very interesting workaround for this problem, and it</span><br>
<span style="float:none;display:inline!important">will work for older versions of Lustre as well. It doesn't disable the</span><br>
<span style="float:none;display:inline!important">OST completely, which is fine if you are doing space balancing (and may</span><br>
<span style="float:none;display:inline!important">even be desirable to allow apps that need more bandwidth for a widely</span><br>
<span style="float:none;display:inline!important">striped file), but it isn't good if you are trying to empty the OST</span><br>
<span style="float:none;display:inline!important">completely to remove it.</span><br>
<br>
<span style="float:none;display:inline!important">It looks like another approach would be to mark the OST as having no free</span><br>
<span style="float:none;display:inline!important">space using OBD_FAIL_OST_ENOINO (0x229) fault injection on that OST:</span><br>
<br>
<span style="float:none;display:inline!important"> lctl set_param fail_loc=0x229 fail_val=<ost_index></span><br>
<br>
<span style="float:none;display:inline!important">This would cause the OST to return 0 free inodes from OST_STATFS for the</span><br>
<span style="float:none;display:inline!important">specified OST index, and the MDT would skip this OST completely. To</span><br>
<span style="float:none;display:inline!important">disable all of the OSTs on an OSS use <ost_index> = -1. It isn't possible</span><br>
<span style="float:none;display:inline!important">to selectively disable a subset of OSTs using this method. The</span><br>
<span style="float:none;display:inline!important">OBD_FAIL_OST_ENOINO fail_loc has been available since Lustre 2.2, which</span><br>
<span style="float:none;display:inline!important">covers all of the 2.4+ versions that are affected by this issue.</span><br>
<br>
<span style="float:none;display:inline!important">If this mechanism works for you (it should, as this fail_loc is used</span><br>
<span style="float:none;display:inline!important">during regular testing) I'd be obliged if someone could file an LUDOC bug</span><br>
<span style="float:none;display:inline!important">so the manual can be updated.</span><br>
<br>
<span style="float:none;display:inline!important">Cheers, Andreas</span></div>
</blockquote>
<br>
</div>
<br>
<div>
<blockquote type="cite">
<div>On Jul 12, 2017, at 4:20 PM, Riccardo Veraldi <<a href="mailto:Riccardo.Veraldi@cnaf.infn.it" target="_blank" moz-do-not-send="true">Riccardo.Veraldi@cnaf.infn.it</a>> wrote:</div>
<br class="m_-1197213167015817042m_4682020740092110866Apple-interchange-newline">
<div>
<div>Hello,<br>
<br>
on one of my lustre FS I need to find a solution so that users can still<br>
access data on the FS but cannot write new files on it.<br>
I have hundreds of clients accessing the FS so remounting it ro is not<br>
really easily feasible.<br>
Is there an option on the OSS side to allow OSTs to be accessed just to<br>
read data and not to store new data ?<br>
tunefs.lustre could do that ?<br>
thank you<br>
<br>
Rick<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" moz-do-not-send="true">lustre-discuss@lists.lustre.org</a><br>
<a class="m_-1197213167015817042m_4682020740092110866moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank" moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br>
<fieldset class="m_-1197213167015817042m_4682020740092110866mimeAttachmentHeader">
</fieldset> <br>
<pre>_______________________________________________
lustre-discuss mailing list
<a class="m_-1197213167015817042m_4682020740092110866moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org" target="_blank" moz-do-not-send="true">lustre-discuss@lists.lustre.org</a>
<a class="m_-1197213167015817042m_4682020740092110866moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank" moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" moz-do-not-send="true">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br>
</blockquote>
</div>
</div>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" moz-do-not-send="true">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote>
</div>
</blockquote>
<br>
</div>
</blockquote>
<blockquote type="cite">
<div><span>_______________________________________________</span><br>
<span>lustre-discuss mailing list</span><br>
<span><a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a></span><br>
<span><a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span><br>
</div>
</blockquote>
</body>
</html>