[Lustre-discuss] redistribute used space to other OSTs (free space)

[Alex]

On Tuesday 04 November 2008 16:37, Brian J. Murrell wrote:
> On Tue, 2008-11-04 at 15:51 +0200, Alex wrote:
> > [root at rs1 ~]# lfs df -h
> > UUID                     bytes      Used Available  Use% Mounted on
> > testfs-MDT0000_UUID     130.4G    460.1M    122.5G    0%
> > /mnt/lustre[MDT:0] testfs-OST0000_UUID      18.3G     17.4G      2.0M  
> > 94% /mnt/lustre[OST:0] testfs-OST0001_UUID      18.3G     15.5G      2.0G
> >   84% /mnt/lustre[OST:1] testfs-OST0002_UUID      36.7G     15.5G    
> > 19.4G   42% /mnt/lustre[OST:2] testfs-OST0003_UUID      36.7G     15.5G  
> >   19.4G   42% /mnt/lustre[OST:3] filesystem summary:     110.0G     63.8G
> >     40.7G   57% /mnt/lustre [root at rs1 ~]#
> >
> > As you can see OST:0 is full!
>
> Indeed.  Is this a 1.6 or a 1.4 installation?  

Hi Brian,

Many thanks for your reply. See my answers inline.

[root at rs1 ~]# rpm -q lustre
lustre-1.6.5.1-2.6.18_53.1.14.el5_lustre.1.6.5.1smp
[root at rs1 ~]#

> As long as you have not 
> forced an allocation policy with striping for example, with 1.6 your
> OSTs should fill in a more-or-less even manner and you should not see
> this type of problem.

I have activated striping on /mnt/lustre after some OSTs has been filled 
arbitray, in order to be able to use entire /mnt/lustre size (110GB), so 
thats the cause of my present problem.

> > Question1: Is any command to use in order to move arround 60% of OST:0
> > data from OST:0 to other OST or to all other OSTes (arround 20% to each
> > other)?

Now i have:
[root at rhclm ~]# cat /proc/fs/lustre/lov/testfs-mdtlov/qos_prio_free
90%
[root at rhclm ~]#

> Deactivate allocation to the full OST on the MDS

How can i do this? lctl conf_param testfs-MDT0000.lov.qos_prio_free=100 will 
do the trick?

> and do some "mv" (or 
> cp/rm if you are more comfortable with that) operations which will
> result in moving the objects for the file to other OSTs.
>

All right. Let me explain better... I found that /mnt/lustre/test3.img is 12Gb 
in size and is striped between all OSTs (arround 3GB/OST). Indeed, if i 
remove /mnt/lustre/test3.img i'll get some free space on OST:0, as well as on 
other OSTs. Now, i want to make up some free space ONLY on OST:0, MOVING some 
objects from it to another OST.

[root at rs1 ~]# lfs getstripe /mnt/lustre/test3.img
OBDS:
0: testfs-OST0000_UUID ACTIVE
1: testfs-OST0001_UUID ACTIVE
2: testfs-OST0002_UUID ACTIVE
3: testfs-OST0003_UUID ACTIVE
/mnt/lustre/test3.img
        obdidx           objid          objid            group
             0               7            0x7                0
             2               6            0x6                0
             3              70           0x46                0
             1              69           0x45                0

[root at rs1 ~]#

So, how can i move objid=7 from OST:0 to OST:2, keeping 
intact /mnt/lustre/test3.img file?

Regards,
Alx