<div><div dir="auto">Jason,</div></div><div dir="auto"><br></div><div dir="auto">If there are files located on the full OST that you can delete then delete them without trying to deactivate the OST. There is a process where you can find files solely located on the full OST using lfs then deactivate the full OST, copy the files to new files with some suffix (.new perhaps) so they end up on other OSTs, delete the original files, rename the new files to the old file names (thereby removing the suffix used previously) and then reactivating the deactivated OST. You might be lucky enough to find a few very large single stripe files located on that OST where moving the file gets you easy gains.</div><div dir="auto"><br></div><div dir="auto">You *should* be able to delete a file that resides on an inactive OST but I’m not over a keyboard where I can do it before making the assertion it is possible.</div><div dir="auto"><br></div><div dir="auto">—Jeff</div><div dir="auto"><br></div><div><br><div class="gmail_quote"><div dir="ltr">On Sat, Jan 5, 2019 at 18:49 Jason Williams <<a href="mailto:jasonw@jhu.edu">jasonw@jhu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_6387161642121510724divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hello, </p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">We have a lustre system (version 2.10.4) that has unfortunately fallen victim to a 100% full OST... Every time we clear some space on it, the system fills it right back up again.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">I have looked around the internet and found you can disable an OST, but when I have tried that, any writes (including deletes) to the OST hang the clients indefinitely. Does anyone know a way to make an OST basically
"read-only" with the exception of deletes so we can work to clear out the OST? Or better yet, a way to "drain" or move files off an OST with a script (keeping in mind it might not be known if the files are in use at the time). Or even a way to tell lustre
"Hey don't write any new data here, but reading and removing data is OK."</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<div id="m_6387161642121510724Signature">
<div id="m_6387161642121510724divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0;margin-bottom:0"><span id="m_6387161642121510724ms-rterangepaste-start"></span></p>
<div style="color:rgb(33,33,33);font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont;font-size:15px;margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">--</span></font></div>
<div style="color:rgb(33,33,33);font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont;font-size:15px;margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Jason Williams</span></font></div>
<div style="color:rgb(33,33,33);font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont;font-size:15px;margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Assistant Director</span></font></div>
<div style="color:rgb(33,33,33);font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont;font-size:15px;margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Systems and Data Center Operations.<br>
Maryland Advanced Research Computing Center (MARCC)<br>
Johns Hopkins University</span></font></div>
<div style="color:rgb(33,33,33);font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont;font-size:15px;margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><a href="mailto:jasonw@jhu.edu" rel="noopener noreferrer" id="m_6387161642121510724LPNoLP" target="_blank"><font color="#0563C1">jasonw@jhu.edu</font></a></span></font></div>
<span id="m_6387161642121510724ms-rterangepaste-end"></span><br>
<p></p>
</div>
</div>
</div>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">------------------------------<br>Jeff Johnson<br>Co-Founder<br>Aeon Computing<br><br><a href="mailto:jeff.johnson@aeoncomputing.com" target="_blank">jeff.johnson@aeoncomputing.com</a><br><a href="http://www.aeoncomputing.com" target="_blank">www.aeoncomputing.com</a><br>t: 858-412-3810 x1001 f: 858-412-3845<br>m: 619-204-9061<br><br>4170 Morena Boulevard, Suite C - San Diego, CA 92117<div><br></div><div>High-Performance Computing / Lustre Filesystems / Scale-out Storage</div></div></div></div></div>