[Lustre-discuss] Moving files off an OST

[Andreas Dilger]

On 2010-04-23, at 11:32, Scott wrote:
> Appolgies if Im missing something obvious here.
> 
> My OSTs are set up in Raid 5 and one of the arrays has a bad stripe so i 
> need to rebuild it.  In preparation for this i want to move all the data 
> off of this OST so i deactivated the OST on the MDS and ran:
> 
> 
> lfs find --recursive --obd nasone-OST0002_UUID --quiet /lustre | while 
> read F;  cp $F $F.tmp && mv $F.tmp $F; done
> 
> This ran for quite a while and after it finished i ran the find command 
> again to confirm there were no more files on the OST.

I did nearly this same thing just yesterday, though I quoted "$F" so that it would handle filenames with spaces, etc. in them.  Note that doing this is only safe for files that are not currently in use by clients/applications.

> However if i look at the OSS i still show there are 3.4TBs of used space 
> on that OST2:
> 
> # df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> 
> /dev/sdd             5765425880 5223515676 249043440  96% /mnt/ost4
> /dev/sdc             4804519904 3479755816 1080708536  77% /mnt/ost2
> 
> Does this make any sense at all or am i missing something obvious here? 
> I was expecting (hoping) to see the used space back to almost zero so 
> does this mean i have quite a bit of lost data?

I got down to virtually no space used, on my 4-year-old OST that I'm moving to a larger disk.

One possibility is that you have open files that are holding this space in use.  If you unmount the MDT (use "umount -f", which will evict all of the clients, though this will cause applications to see IO errors, if that is acceptable) and mount it again, does the space usage go away?

Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.