[Lustre-discuss] lctl --device XX deactivate doesn't make OST read only

[Alastair Ferguson]

Sorry - final update.

It appears that two osts are both still at 100% (don't know how I got that wrong) and 40Mb space.

I tried:

 lfs find /data -O AC3-OST000a_UUID -size +20G | lfs_migrate -y

Now getting this:

/data/smarts/ksc_mq/am/03723.am: llapi_semantic_traverse: Failed to open '/data/home/zzhao/workspace/topical_collocation_model/results/ir_evaluation/sjmn2k_tng/DocLDA/k050-alpha0.10-gamma0.01/GibbsRun-2': No such file or directory (2)
error: find failed for +20G.
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: write failed on "/data/smarts/ksc_mq/am/03723.am.tmp.N13504": No space left on device (28)


Also doing:

lfs_migrate /data/workflow

(8TB in size)

& 

lfs_migrate /data/raw

(15TB)

and still:

AC3-OST000a_UUID           14.3T       13.6T       46.1M 100% /data[OST:10]
AC3-OST0010_UUID            7.2T        6.8T       46.1M 100% /data[OST:16]

We can't run our processes because of the no space on device errors. Help!

Alastair Ferguson
IT Manager
Capital Markets CRC Limited (CMCRC)
Telephone: +61 2 8088 4222
Mobile: +61 424 235 159
Fax: +61 2 8088 4201
www.cmcrc.com 



Capital Markets CRC Ltd - Confidential Communication
The information contained in this e-mail is confidential.  It is intended for the addressee only.  If you receive this e-mail by mistake please promptly inform us by reply e-mail and then delete the e-mail and destroy any printed copy. You must not disclose or use in any way the information in the e-mail. There is no warranty that this e-mail is error or virus free. It may be a private communication, and if so, does not represent the views of the CMCRC and its associates.  If it is a private communication, care should be taken in opening it to ensure that undue offence is not given.


On 18/06/2013, at 9:26 AM, "Dilger, Andreas" <andreas.dilger at intel.com> wrote:

> On 2013/17/06 1:12 AM, "Alastair Ferguson" <aferguson at cmcrc.com> wrote:
>> OK, bit of a weird one, so 3 OSTs are 100%, but there is 30TB of free
>> space around the other OSTs, so I do:
>> 
>> lfs df -h
>> 
>> Get this part as one of the OSTs I need to deactivate:
>> 
>> AC3-OST000c_UUID           14.3T       13.6T       87.4M 100%
>> /data[OST:12]
>> 
>> then
>> 
>> lctl dl
>> 
>> 19 UP osc AC3-OST000c-osc AC3-mdtlov_UUID 5
>> 
>> Then
>> 
>> lctl --device 19 deactivate
>> 
>> then
>> 
>> lctl dl:
>> 
>> 19 IN osc AC3-OST000c-osc AC3-mdtlov_UUID 5
>> 
>> Should be read only right>>?
> 
> Right, this is the MDS OSC device, so no new files should be allocated on
> that OST.
> 
>> Then
>> 
>> lfs getstripe -O AC3-OST000c_UUID -rv -d /data | grep /data >>
>> ost000c_raw.txt
>> 
>> To find the files in the filesystem (/data) and strip out all the stuff
>> you don't need.  Then:
>> 
>> while read line; do cp -p "$line" "$line.___bak"; rm -f "$line"; mv
>> "$line.___bak" "$line";  done < ost000c_raw.txt
>> 
>> This should move the data off the OST but it doesn't. I have used this
>> procedure before to remove data from a whole server (which worked) and I
>> can see when I  lfs df -h
>> the ost emptying but in this case it goes up and down suggesting it is
>> copying BACK to the same OST despite the fact it is IN not UP when lctl
>> dl is run.
> 
> You should look at "lfs_migrate" and its man page, for a more robust
> mechanism for
> doing the above migration.  Your script is unsafe if interrupted after "rm
> -f" but
> before "mv" moves the old file into place.  You can also use "lfs_migrate"
> in a
> pipeline, so that it only moves new files, while your script would re-move
> the same
> files repeatedly if interrupted and restarted.
> 
>> How can I get files off this as I get errors saying no space on device??
> 
> Your process _should_ be working, but if you are moving small files the
> effects may
> be slow.  As mentioned in the "lfs_migrate" man page, you should select
> large files
> to migrate, since you will get better IO performance, and will free space
> more quickly.
> 
> Cheers, Andreas
> -- 
> Andreas Dilger
> 
> Lustre Software Architect
> Intel High Performance Data Division
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130619/b81ad7d4/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 13489 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130619/b81ad7d4/attachment.png>