[lustre-discuss] ran out of MDT inodes (Liam Forbes)

[Brad Merchant]

Yes expanding the pool with another mirrored pair of disks will add inodes.

You can also recover using the ZFS slop space temporarily.

https://wiki.lustre.org/ZFS_MDT_ENOSPC_Recovery


On Thu, Sep 15, 2022 at 4:04 PM <lustre-discuss-request at lists.lustre.org>
wrote:

>
> Message: 1
> Date: Wed, 14 Sep 2022 14:22:33 -0800
> From: Liam Forbes <loforbes at alaska.edu>
> To: lustre-discuss at lists.lustre.org
> Cc: Kevin Galloway <kpgalloway at alaska.edu>, John Mitchell
>         <jpmitchell at alaska.edu>, Liam Forbes <loforbes at alaska.edu>
> Subject: [lustre-discuss] ran out of MDT inodes
> Message-ID:
>         <CACXvBArq1kfb7-oQ+LZ_KdsPLk-psJZZjsLBL=
> b45wj_U_Vdkw at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Today, in our lustre 2.10.3 filesystem, the MDT ran out of inodes. We are
> using ZFS as the backing filesystem.
>
> [loforbes at mds02 ~]$ df -i -t lustre
> Filesystem                    Inodes    IUsed IFree IUse% Mounted on
> digdug-meta/lustre2-mgt-mdt 83703636 83703636     0  100%
> /mnt/lustre/local/lustre2-MDT0000
> [loforbes at mds02 ~]$ sudo zpool list -v
> NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
> digdug-meta   744G   721G  23.2G         -    86%    96%  1.00x  ONLINE  -
>   mirror   372G   368G  4.25G         -    84%    98%
>     scsi-35000c5003017156b      -      -      -         -      -      -
>     scsi-35000c500301715e7      -      -      -         -      -      -
>   mirror   372G   353G  19.0G         -    88%    94%
>     scsi-35000c5003017155f      -      -      -         -      -      -
>     scsi-35000c500301715a7      -      -      -         -      -      -
>
> When we try to delete files, we get the error message:
>   rm: cannot remove XXXXX: No space left on device
>
> Is there a way to unlink files and free up inodes?
>
> Is it possible to expand the existing zpool and filesystem for the MDT?
>
> Is it possible to do a backup of just our MDT? If so, how?
>
> --
> Regards,
> -liam
>
> -There are uncountably more irrational fears than rational ones. -P. Dolan
> Liam Forbes           loforbes at alaska.edu                       ph:
> 907.450.8618
> UAF GI Research Computing Systems Manager
> https://calendly.com/ualoforbes/30min
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220914/5e9b7fde/attachment-0001.htm
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220915/3693d810/attachment.htm>