[lustre-discuss] Re-format MDT for more inodes

[Tung-Han Hsieh]

Dear All,

We are facing a strange situation. So we are asking for help here.
Any suggestions will be very apprciated.

Our Lustre file system (version 1.8.7) ran out of MDT inodes. So we
have backup the MDT data, reformat MDT with larger partition size
and more inodes, and restore the MDT data. After that, the whole
file system works normally, but we found that the client cannot see
as many inodes as the MDT server. Here are the details of what we
have done.

1. In the beginning, the MDT has a partition with about 200GB. It
   was formatted with default options, and we got more than 48,660,000
   inodes. But then we exhausted all the indoes. So we decided to
   reformat the MDT partiton.

2. We shutdown the Lustre file system, and proceed the following steps
   to backup the MDT data:

   - mount -t ldiskfs /dev/sda2 /mnt/mdt
   - cd /mnt/mdt
   - getfattr -R -d -m '.*' -P . > /tmp/ea.bak
   - tar -cf /tmp/mdt.tar .
   - cd /
   - umount /mnt/mdt

   (Note: /dev/sda1 is the MGS partition. We did not change it at all)

3. We use the "fdisk" to re-partition the hard disk, and enlarge the
   partition to 500GB (which is almost the whole disk size).

4. We reformat this partition with:

   - mkfs.lustre --fsname cfs --mdt --mgsnode=<my_host_name> \
                 --mkfsoptions="-i 1024" /dev/sda2

   Since we use stripe count = 1 (default), we would like to have a
   higher density of inodes in MDT.

   After reformat, and mount it with ldiskfs file system, we really
   get a large number of inodes:

# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
tmpfs                1019522       4 1019518    1% /lib/init/rw
udev                 1019522    2847 1016675    1% /dev
tmpfs                1019522       1 1019521    1% /dev/shm
/dev/shm             1019522       1 1019521    1% /dev/shm
overflow             1019522       2 1019520    1% /tmp
/dev/sda1              61824      42   61782    1% /cfs/mgs
/dev/sda2            486326016     2 486326014  1% /mnt/mdt

5. Then we restore the MDT data to the new partition via:

   - tune2fs -O dir_index /dev/sda2
   - cd /mnt/mdt
   - tar xf /tmp/mdt.tar
   - setfattr --restore=/tmp/ea.bak
   - rm -f OBJECTS/* CATALOGS
   - cd /
   - umount /mnt/mdt

   Then we can mount the lustre file system from client successfully.

6. However, from the client side, it cannot see all the inodes as in
   the MDT server. In MDT server, using "df -i" we see:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
tmpfs                1019522       4 1019518    1% /lib/init/rw
udev                 1019522    2847 1016675    1% /dev
tmpfs                1019522       1 1019521    1% /dev/shm
/dev/shm             1019522       1 1019521    1% /dev/shm
overflow             1019522       2 1019520    1% /tmp
/dev/sda1              61824      42   61782    1% /cfs/mgs
/dev/sda2            486326016 48661002 437665014   11% /cfs/mdt

   But from the client, using "df -i" we see:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            3055616  448561 2607055   15% /
tmpfs                1024174       6 1024168    1% /lib/init/rw
udev                 1024174    4969 1019205    1% /dev
tmpfs                1024174       1 1024173    1% /dev/shm
/dev/shm             1024174       1 1024173    1% /dev/shm
/dev/sda3            26501120 1203655 25297465    5% /home
dfs0:/cfs            90849681 48661011 42188670   54% /work

   Please note that the number 90849681 is actually the inode count
   that is close to the default inode density (--mkfsoptions="-i 4096")
   in a 500GB partition.

Could anyone know what's going on here. Will this situation harm
the operation of the lustre file system? Any suggestions are very
appreciated.

Thank you very much in advance.


Best Regards,

T.H.Hsieh