<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Sorry - final update.<div><br></div><div>It appears that two osts are both still at 100% (don't know how I got that wrong) and 40Mb space.</div><div><br></div><div>I tried:</div><div><br></div><div><div> lfs find /data -O AC3-OST000a_UUID -size +20G | lfs_migrate -y</div><div><br></div><div>Now getting this:</div><div><br></div><div>/data/smarts/ksc_mq/am/03723.am: llapi_semantic_traverse: Failed to open '/data/home/zzhao/workspace/topical_collocation_model/results/ir_evaluation/sjmn2k_tng/DocLDA/k050-alpha0.10-gamma0.01/GibbsRun-2': No such file or directory (2)</div><div>error: find failed for +20G.</div><div>rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)</div><div>rsync: write failed on "/data/smarts/ksc_mq/am/03723.am.tmp.N13504": <font color="#ff2600">No space left on device (28)</font></div><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Tahoma; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div><span><br></span></div><div><span><br></span></div><div><span>Also doing:</span></div><div><span><br></span></div><div><span>lfs_migrate /data/workflow</span></div><div><span><br></span></div><div><span>(8TB in size)</span></div><div><span><br></span></div><div><span>& </span></div><div><span><br></span></div><div><span>lfs_migrate /data/raw</span></div><div><span><br></span></div><div><span>(15TB)</span></div><div><span><br></span></div><div><span>and still:</span></div><div><span><br></span></div><div><span><div>AC3-OST000a_UUID 14.3T 13.6T 46.1M 100% /data[OST:10]</div><div>AC3-OST0010_UUID 7.2T 6.8T 46.1M 100% /data[OST:16]</div><div><br></div><div>We can't run our processes because of the no space on device errors. Help!</div></span></div><div><span><br class="Apple-interchange-newline"><font class="Apple-style-span" face="Arial">Alastair Ferguson<br>IT Manager<br>Capital Markets CRC Limited (CMCRC)<br>Telephone: +61 2 8088 4222<br>Mobile: +61 424 235 159<br>Fax: +61 2 8088 4201<br><a href="http://www.cmcrc.com">www.cmcrc.com</a> </font></span></div><div><span><br></span></div></span><br class="Apple-interchange-newline"><span></span><span></span><span></span><span><img height="87" width="253" id="9d4827e7-af72-4a3b-956b-6bc8ce1a1070" apple-width="yes" apple-height="yes" src="cid:3340953947_834664"></span><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Tahoma; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Tahoma; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Tahoma; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="font-family: Helvetica; "><br></span><font class="Apple-style-span" face="Arial" size="1"><span class="Apple-style-span" style="font-size: 9px; ">Capital Markets CRC Ltd - Confidential Communication<br>The information contained in this e-mail is confidential. It is intended for the addressee only. If you receive this e-mail by mistake please promptly inform us by reply e-mail and then delete the e-mail and destroy any printed copy. You must not disclose or use in any way the information in the e-mail. There is no warranty that this e-mail is error or virus free. It may be a private communication, and if so, does not represent the views of the CMCRC and its associates. If it is a private communication, care should be taken in opening it to ensure that undue offence is not given.</span></font><span class="Apple-style-span" style="font-family: Helvetica; "><br></span><span class="Apple-style-span" style="font-family: Helvetica; "><br></span></span></span></span></span>
</span></div>
<br><div><div>On 18/06/2013, at 9:26 AM, "Dilger, Andreas" <<a href="mailto:andreas.dilger@intel.com">andreas.dilger@intel.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">On 2013/17/06 1:12 AM, "Alastair Ferguson" <<a href="mailto:aferguson@cmcrc.com">aferguson@cmcrc.com</a>> wrote:<br><blockquote type="cite">OK, bit of a weird one, so 3 OSTs are 100%, but there is 30TB of free<br>space around the other OSTs, so I do:<br><br>lfs df -h<br><br>Get this part as one of the OSTs I need to deactivate:<br><br>AC3-OST000c_UUID 14.3T 13.6T 87.4M 100%<br>/data[OST:12]<br><br>then<br><br>lctl dl<br><br>19 UP osc AC3-OST000c-osc AC3-mdtlov_UUID 5<br><br>Then<br><br>lctl --device 19 deactivate<br><br>then<br><br>lctl dl:<br><br>19 IN osc AC3-OST000c-osc AC3-mdtlov_UUID 5<br><br>Should be read only right>>?<br></blockquote><br>Right, this is the MDS OSC device, so no new files should be allocated on<br>that OST.<br><br><blockquote type="cite">Then<br><br>lfs getstripe -O AC3-OST000c_UUID -rv -d /data | grep /data >><br>ost000c_raw.txt<br><br>To find the files in the filesystem (/data) and strip out all the stuff<br>you don't need. Then:<br><br>while read line; do cp -p "$line" "$line.___bak"; rm -f "$line"; mv<br>"$line.___bak" "$line"; done < ost000c_raw.txt<br><br>This should move the data off the OST but it doesn't. I have used this<br>procedure before to remove data from a whole server (which worked) and I<br>can see when I lfs df -h<br>the ost emptying but in this case it goes up and down suggesting it is<br>copying BACK to the same OST despite the fact it is IN not UP when lctl<br>dl is run.<br></blockquote><br>You should look at "lfs_migrate" and its man page, for a more robust<br>mechanism for<br>doing the above migration. Your script is unsafe if interrupted after "rm<br>-f" but<br>before "mv" moves the old file into place. You can also use "lfs_migrate"<br>in a<br>pipeline, so that it only moves new files, while your script would re-move<br>the same<br>files repeatedly if interrupted and restarted.<br><br><blockquote type="cite">How can I get files off this as I get errors saying no space on device??<br></blockquote><br>Your process _should_ be working, but if you are moving small files the<br>effects may<br>be slow. As mentioned in the "lfs_migrate" man page, you should select<br>large files<br>to migrate, since you will get better IO performance, and will free space<br>more quickly.<br><br>Cheers, Andreas<br>-- <br>Andreas Dilger<br><br>Lustre Software Architect<br>Intel High Performance Data Division<br><br><br></blockquote></div><br></div></body></html>