<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Chad hit the nail on the head. I thought about the fact that it was still deactivated yesterday but was afraid to reactivate it until I verified the space was free.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">FWIW, the URL about handling full OSTs does not include the fact that the space will not be free until you reactivate the OST. It actually implies the opposite.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"><a href="http://wiki.lustre.org/Handling_Full_OSTs" class="OWAAutoLink" id="LPlnk293218" previewremoved="true">http://wiki.lustre.org/Handling_Full_OSTs</a><br>
<br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<p style="margin-top:0; margin-bottom:0"><span id="ms-rterangepaste-start"></span></p>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">--</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Jason Williams</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Assistant Director</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Systems and Data Center Operations.<br>
Maryland Advanced Research Computing Center (MARCC)<br>
Johns Hopkins University</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><a href="mailto:jasonw@jhu.edu" target="_blank" rel="noopener noreferrer" id="LPNoLP"><font color="#0563C1">jasonw@jhu.edu</font></a></span></font></div>
<span id="ms-rterangepaste-end"></span><br>
<p></p>
</div>
</div>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Chad DeWitt <ccdewitt@uncc.edu><br>
<b>Sent:</b> Thursday, January 17, 2019 3:07 PM<br>
<b>To:</b> Jason Williams<br>
<b>Cc:</b> Alexander I Kulyavtsev; lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Migrating files doesn't free space on the OST</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">Hi Jason,
<div><br>
</div>
<div>I do not know if this will help you or not, but I had a situation in 2.8.0 where an OST filled up and I marked it as disabled on the MDS:</div>
<div><br>
</div>
<div><span style="font-family:"Courier New"">lctl dl | grep osc</span> </div>
<div>...Grab the <i>device_id</i> of the full OST and then deactivate it...</div>
<div><span style="font-family:"Courier New"">lctl --device </span><i style="font-family:"Courier New"">device_id</i><span style="font-family:"Courier New""> deactivate</span></div>
<div><br>
</div>
<div>IIRC, this allowed the data to be read, but deletes were not processed. When I re-activated the OST, then the deletes were processed and space started clearing. I think you stated you had the OST deactivated. If you still do, try to reactive it. </div>
<div><br>
</div>
<div><font face="monospace, monospace">lctl --device <i>device_id</i> activate</font><br>
</div>
<div><br>
</div>
<div>Once you reactivate the OST, the deletes will start processing within 10 - 30 seconds... Just use
<font face="monospace, monospace">lfs df -h</font> to watch...</div>
<div><br>
</div>
<div>-cd</div>
<div> <br clear="all">
<div>
<div dir="ltr" class="x_m_-5195253070295309020gmail-m_4138295817730096550m_3313866458073280806gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<p style="margin:0in 0in 0.0001pt"><span style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, monospace">------------------------------------------------------------<u></u><u></u></font></span></p>
<p style="margin:0in 0in 0.0001pt"><span style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, monospace">Chad DeWitt, CISSP</font></span></p>
<p style="margin:0in 0in 0.0001pt"><span style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, monospace">UNC Charlotte <b>| </b>ITS – University Research Computing</font></span></p>
<p style="margin:0in 0in 0.0001pt"><font color="#000000" size="2" face="monospace, monospace"><span style="background-color:rgba(255,255,255,0)"><a style="color:rgb(34,34,34)" id="LPlnk498623" class="OWAAutoLink" previewremoved="true">9201 University City Blvd. </a><b><a style="color:rgb(34,34,34)" id="LPlnk442622" class="OWAAutoLink" previewremoved="true">| </a></b><a style="color:rgb(34,34,34)" id="LPlnk295528" class="OWAAutoLink" previewremoved="true">Charlotte,
NC 28223</a></span></font></p>
<p style="margin:0in 0in 0.0001pt"><font color="#000000" size="2" face="monospace, monospace"><span style="background-color:rgba(255,255,255,0)"><a href="mailto:ccdewitt@uncc.edu" target="_blank" style="color:rgb(17,85,204)" id="LPlnk295655" class="OWAAutoLink" previewremoved="true">ccdewitt@uncc.edu</a> <b>| </b><a style="color:rgb(34,34,34)" id="LPlnk329507" class="OWAAutoLink" previewremoved="true">www.uncc.edu</a><u></u><u></u></span></font></p>
<p style="margin:0in 0in 0.0001pt"><span style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, monospace">------------------------------------------------------------</font></span></p>
<p style="margin:0in 0in 0.0001pt"><span style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, monospace"><br>
</font></span></p>
<p style="margin:0in 0in 0.0001pt">If you are not the <span>intended</span> recipient of this transmission or a person responsible for delivering it to the <span>intended</span> recipient, any disclosure, copying, distribution, or other use of any of the information
in this transmission is strictly prohibited. If you have received this transmission in error, please notify me immediately by reply email or by telephone at 704-687-7802. Thank you.<span style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, monospace"><br>
</font></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail_attr">
On Thu, Jan 17, 2019 at 2:38 PM Jason Williams <<a href="mailto:jasonw@jhu.edu" target="_blank" id="LPlnk603160" class="OWAAutoLink" previewremoved="true">jasonw@jhu.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px">Hello Alexander,</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Thank you for your reply. <br>
<br>
- We are not using zfs, it's an LDISKFS backing store, so no snapshots.</p>
<p style="margin-top:0px; margin-bottom:0px">- I have re-run lfs getstripe to make sure the file is indeed moving</p>
<p style="margin-top:0px; margin-bottom:0px">- I just looked for lfsck but I don't seem to have it. We are running 2.10.4 so I don't know what version that appeared in.</p>
<p style="margin-top:0px; margin-bottom:0px">- I will try to have a look into the jobstats and see what I can find, but I made sure the files I moved were not in use when I moved them. </p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830Signature">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px"><span id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830ms-rterangepaste-start"></span></p>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">--</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Jason Williams</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Assistant Director</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Systems and Data Center Operations.<br>
Maryland Advanced Research Computing Center (MARCC)<br>
Johns Hopkins University</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><a href="mailto:jasonw@jhu.edu" rel="noopener noreferrer" id="LPlnk754022" target="_blank" class="OWAAutoLink" previewremoved="true"><font color="#0563C1">jasonw@jhu.edu</font></a></span></font></div>
<span id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830ms-rterangepaste-end"></span><br>
<p></p>
</div>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Alexander I Kulyavtsev <<a href="mailto:aik@fnal.gov" target="_blank" id="LPlnk36318" class="OWAAutoLink" previewremoved="true">aik@fnal.gov</a>><br>
<b>Sent:</b> Thursday, January 17, 2019 12:56 PM<br>
<b>To:</b> Jason Williams; <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" id="LPlnk661491" class="OWAAutoLink" previewremoved="true">
lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> Re: Migrating files doesn't free space on the OST</font>
<div> </div>
</div>
<div dir="ltr">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px">- you can re-run command to find files residing on ost to see if files are new or old.</p>
<p style="margin-top:0px; margin-bottom:0px">- zfs may have snapshots if you ever did snapshots; it takes space.</p>
<p style="margin-top:0px; margin-bottom:0px">- removing data or snapshots has some lag to release the blocks (tens of minutes) but I guess that is completed by now.</p>
<p style="margin-top:0px; margin-bottom:0px">- there are can be orphan objects on OST if you had crashes. On older lustre versions if the ost was emptied out you can mount underlying fs as ext4 or zfs; set mount to readonly and browse ost objects - you may
see if there are some orphan objects left. On newer lustre releases you probably can run lfsck (lustre scanner).</p>
<p style="margin-top:0px; margin-bottom:0px">- to find what hosts / jobs currently writing to lustre you may enable lustre jobstats; clear counters and parse stats files in /proc . There was xltop tool on github for older versions of lustre not having implemented
jobstats but it was not updated for a while.</p>
<p style="margin-top:0px; margin-bottom:0px">- depending on lustre version you have the implementation of lfs migrate is different. The older version copied file with other name to other ost, renamed files and removed old file. If migration done on file open
for write by application the data will not be released until file closed (and data in new file are wrong). Recent implementation of migrate does swap of the file objects with file layout lock taken. I can not tell if it is safe for active write.</p>
<p style="margin-top:0px; margin-bottom:0px">- not releasing space can be a bug - did you check jira on whamcloud? <span style="font-size:12pt">What version of lustre do you have? Is it ldiskfs or zfs based? </span><span style="font-size:12pt">zfs version?</span></p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Alex.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank" id="LPlnk681351" class="OWAAutoLink" previewremoved="true">lustre-discuss-bounces@lists.lustre.org</a>>
on behalf of Jason Williams <<a href="mailto:jasonw@jhu.edu" target="_blank" id="LPlnk351518" class="OWAAutoLink" previewremoved="true">jasonw@jhu.edu</a>><br>
<b>Sent:</b> Wednesday, January 16, 2019 10:25 AM<br>
<b>To:</b> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" id="LPlnk460775" class="OWAAutoLink" previewremoved="true">
lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> [lustre-discuss] Migrating files doesn't free space on the OST</font>
<div> </div>
</div>
<div dir="ltr">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px">I am trying to migrate files I know are not in use off of the full OST that I have using lfs migrate. I have verified up and down that the files I am moving are on that OST and that after the migrate lfs getstripe
indeed shows they are no longer on that OST since it's disabled in the MDS.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">The problem is, the used space on the OST is not going down.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">I see one of at least two issues:</p>
<p style="margin-top:0px; margin-bottom:0px">- the OST is just not freeing the space for some reason or another ( I don't know)</p>
<p style="margin-top:0px; margin-bottom:0px">- Or someone is writing to existing files just as fast as I am clearing the data (possible, but kind of hard to find)</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Is there possibly something else I am missing? Also, does anyone know a good way to see if some client is writing to that OST and determine who it is if it's more probable that that is what is going on?</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_x_Signature">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px"><span id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_x_ms-rterangepaste-start"></span></p>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">--</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Jason Williams</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Assistant Director</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Systems and Data Center Operations.<br>
Maryland Advanced Research Computing Center (MARCC)<br>
Johns Hopkins University</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px">
<font face="Calibri,sans-serif" size="2"><span style="font-size:11pt"><a href="mailto:jasonw@jhu.edu" rel="noopener noreferrer" id="LPlnk422296" target="_blank" class="OWAAutoLink" previewremoved="true"><font color="#0563C1">jasonw@jhu.edu</font></a></span></font></div>
<span id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_x_ms-rterangepaste-end"></span><br>
<p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" id="LPlnk905013" class="OWAAutoLink" previewremoved="true">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank" id="LPlnk726178" class="OWAAutoLink" previewremoved="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>