[lustre-discuss] ran out of MDT inodes

[Liam Forbes]

Today, in our lustre 2.10.3 filesystem, the MDT ran out of inodes. We are
using ZFS as the backing filesystem.

[loforbes at mds02 ~]$ df -i -t lustre
Filesystem                    Inodes    IUsed IFree IUse% Mounted on
digdug-meta/lustre2-mgt-mdt 83703636 83703636     0  100%
/mnt/lustre/local/lustre2-MDT0000
[loforbes at mds02 ~]$ sudo zpool list -v
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
digdug-meta   744G   721G  23.2G         -    86%    96%  1.00x  ONLINE  -
  mirror   372G   368G  4.25G         -    84%    98%
    scsi-35000c5003017156b      -      -      -         -      -      -
    scsi-35000c500301715e7      -      -      -         -      -      -
  mirror   372G   353G  19.0G         -    88%    94%
    scsi-35000c5003017155f      -      -      -         -      -      -
    scsi-35000c500301715a7      -      -      -         -      -      -

When we try to delete files, we get the error message:
  rm: cannot remove XXXXX: No space left on device

Is there a way to unlink files and free up inodes?

Is it possible to expand the existing zpool and filesystem for the MDT?

Is it possible to do a backup of just our MDT? If so, how?

-- 
Regards,
-liam

-There are uncountably more irrational fears than rational ones. -P. Dolan
Liam Forbes           loforbes at alaska.edu                       ph:
907.450.8618
UAF GI Research Computing Systems Manager
https://calendly.com/ualoforbes/30min
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220914/5e9b7fde/attachment.htm>