[lustre-discuss] Using lfs migrate to move files between MDTs

[Jesse Stroik]

Rick, I've run into the same issue.

I don't think this is like LU-11306. Moving a file into a directory 
assigned to another MDT doesn't change its MDT assignment. Copying a the 
same file into that directory does. That behavior is as expected.

However, when I migrated files off of MDT1 with 'lfs migrate -m0 
robinhood' the number of inodes used on MDT1 did not change. If I 
migrated back and forth between the MDTs, the number of inodes used on 
MDT1 increased each time I migrated files to it but never decreased 
unless i deleted the files. And then inodes used decreased only by the 
amount of the most recent migration.

I migrated the directory back to MDT1, mounted the MDTs as ldiskfs and 
examined them:

=====
/mnt/mdt0-ldiskfs/ROOT/robinhood
# exists but is empty

/mnt/mdt1-ldiskfs/REMOTE_PARENT_DIR/0xcc0000402:0x3c:0x0/
# contained the files within robinhood/
=====

Again, that is as I expected it would be. But the IUsed count doesn't 
seem to reflect that.

For our use case, I think we're okay. But it would be nice to know why 
the number of inodes used does not decrease when file are migrated off 
of the MDT using 'lfs migrate -m'.

Best,
Jesse


On 3/29/19 4:36 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
> I have been playing a little bit with DNE today, and I had a question about some odd behavior I saw regarding inode counts.  My Lustre 2.10.6 file system has 2 MDTs. I created a directory (which by default resides on MDT0) and then created 10 files in that directory:
> 
>    [root at sip-mgmt2 test]# lfs df -i /lustre/newfs | grep MDT
>    siplfs-MDT0000_UUID   3906232784         395  3906232389   0% /lustre/newfs[MDT:0]
>    siplfs-MDT0001_UUID   2822275072         276  2822274796   0% /lustre/newfs[MDT:1]
> 
>    [root at sip-mgmt2 test]# mkdir subdir
> 
>    [root at sip-mgmt2 test]# for i in `seq 1 10`; do touch subdir/file.$i; done
> 
>    [root at sip-mgmt2 test]# lfs df -i /lustre/newfs | grep MDT
>    siplfs-MDT0000_UUID   3906232784         406  3906232378   0% /lustre/newfs[MDT:0]
>    siplfs-MDT0001_UUID   2822275072         276  2822274796   0% /lustre/newfs[MDT:1]
> 
> As expected, the inode count for MDT0 increases by about 10.  Then I used “lfs migrate” to move that directory to MDT1:
> 
>    [root at sip-mgmt2 test]# lfs migrate -m 1 subdir
> 
>    [root at sip-mgmt2 test]# lfs df -i /lustre/newfs | grep MDT
>    siplfs-MDT0000_UUID   3906232784         408  3906232376   0% /lustre/newfs[MDT:0]
>    siplfs-MDT0001_UUID   2822275072         289  2822274783   0% /lustre/newfs[MDT:1]
> 
> I expected the inode count for MDT1 to increase by about 10, but I did not see the MDT0 inode count decrease.  So then I decided to migrate the directory back to MDT0:
> 
>    [root at sip-mgmt2 test]# lfs migrate -m 0 subdir
> 
>    [root at sip-mgmt2 test]# lfs df -i /lustre/newfs | grep MDT
>    siplfs-MDT0000_UUID   3906232784         420  3906232364   0% /lustre/newfs[MDT:0]
>    siplfs-MDT0001_UUID   2822275072         290  2822274782   0% /lustre/newfs[MDT:1]
> 
> The inode count increases again on MDT0, but it does not decrease on MDT1.  So then I decided to delete the directory and all the files:
> 
>    [root at sip-mgmt2 test]# rm -rf subdir
> 
>    [root at sip-mgmt2 test]# lfs df -i /lustre/newfs | grep MDT
>    siplfs-MDT0000_UUID   3906232784         409  3906232375   0% /lustre/newfs[MDT:0]
>    siplfs-MDT0001_UUID   2822275072         290  2822274782   0% /lustre/newfs[MDT:1]
> 
> This reclaims about 10 inodes on MDT0, but both MDT0 and MDT1 still seem to have a permanently increased inode count resulting from the previous lfs migrate commands.  Even after waiting a while and remounting the MDTs, the inodes don’t seem to be reclaimed.
> 
> It seems like this is a bug, but since this is the first time I have used DNE, I thought maybe I am just misunderstanding something or using the lfs migrate command incorrectly.  It looks similar to LU-11306, but that issue seemed to deal with renaming files and not migrating them.
> 
>   Any help would be appreciated.
> 
> --
> Rick Mohr
> Senior HPC System Administrator
> National Institute for Computational Sciences
> http://www.nics.tennessee.edu
> 
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3964 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20190805/1358a143/attachment.bin>