[Lustre-discuss] Re-balance the un-balanced OSTs

[thhsieh]

Dear All,

We are running a lustre-1.6.6 on Linux vanilla kernel 2.6.22.19.
Recently we suffer a serious un-balanced OSTs problem. If I run
the command:

# /opt/lustre-1.8/bin/lfs df
UUID                 1K-blocks      Used Available  Use% Mounted on
cwarp-MDT0000_UUID   119627860    659184 112132220    0% /mnt/src[MDT:0]
cwarp-OST0000_UUID   1441859112 1343925184  24691744   93% /mnt/src[OST:0]
cwarp-OST0001_UUID   1441859128 799739136 568869616   55% /mnt/src[OST:1]
cwarp-OST0002_UUID   1441859128 643666316 724950624   44% /mnt/src[OST:2]
cwarp-OST0003_UUID   1441859112 745288308 623015556   51% /mnt/src[OST:3]
cwarp-OST0004_UUID   1441859128 654020352 714567920   45% /mnt/src[OST:4]
cwarp-OST0005_UUID   1441859128 658416232 709949996   45% /mnt/src[OST:5]

filesystem summary:  8651154736 4845055528 3366045456   56% /mnt/src

It is clear that the cwarp-OST0000_UUID is almost full, but the
other OSTs are still quite empty. No matter what, we are trying
to do something in order to re-balance the OSTs. Our procedure is
(which is indicated in the Lustre manual):

1. In MDS, disable the cwarp-OST0000 so that newly created file
   will go to other OSTs:

   echo 0 > /proc/fs/lustre/osc/cwarp-OST0000-osc/active

2. In one of the client node, we copy and rename files, in hope that
   some files can be pull out of the cwarp-OST0000 and then go to other
   OSTs:

   cp /path/to/some/files /path/to/some/files.tmp
   mv /path/to/some/files.tmp /path/to/some/files

However, this way seems does not help too much. I guess this is because
we do not pull out the files which locate in cwarp-OST0000 exactly.


Actually we are not sure whether this is the correct way to solve this
problem. Any suggestion to re-balance or to prevent un-balanced OSTs
are very appriciated.


Best Regards,

T.H.Hsieh