<div dir="ltr"><div>Yes expanding the pool with another mirrored pair of disks will add inodes.</div><div><br></div><div>You can also recover using the ZFS slop space temporarily.</div><div><br></div><div><a href="https://wiki.lustre.org/ZFS_MDT_ENOSPC_Recovery">https://wiki.lustre.org/ZFS_MDT_ENOSPC_Recovery</a><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 15, 2022 at 4:04 PM <<a href="mailto:lustre-discuss-request@lists.lustre.org">lustre-discuss-request@lists.lustre.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Message: 1<br>
Date: Wed, 14 Sep 2022 14:22:33 -0800<br>
From: Liam Forbes <<a href="mailto:loforbes@alaska.edu" target="_blank">loforbes@alaska.edu</a>><br>
To: <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
Cc: Kevin Galloway <<a href="mailto:kpgalloway@alaska.edu" target="_blank">kpgalloway@alaska.edu</a>>, John Mitchell<br>
<<a href="mailto:jpmitchell@alaska.edu" target="_blank">jpmitchell@alaska.edu</a>>, Liam Forbes <<a href="mailto:loforbes@alaska.edu" target="_blank">loforbes@alaska.edu</a>><br>
Subject: [lustre-discuss] ran out of MDT inodes<br>
Message-ID:<br>
<CACXvBArq1kfb7-oQ+LZ_KdsPLk-psJZZjsLBL=<a href="mailto:b45wj_U_Vdkw@mail.gmail.com" target="_blank">b45wj_U_Vdkw@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Today, in our lustre 2.10.3 filesystem, the MDT ran out of inodes. We are<br>
using ZFS as the backing filesystem.<br>
<br>
[loforbes@mds02 ~]$ df -i -t lustre<br>
Filesystem Inodes IUsed IFree IUse% Mounted on<br>
digdug-meta/lustre2-mgt-mdt 83703636 83703636 0 100%<br>
/mnt/lustre/local/lustre2-MDT0000<br>
[loforbes@mds02 ~]$ sudo zpool list -v<br>
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT<br>
digdug-meta 744G 721G 23.2G - 86% 96% 1.00x ONLINE -<br>
mirror 372G 368G 4.25G - 84% 98%<br>
scsi-35000c5003017156b - - - - - -<br>
scsi-35000c500301715e7 - - - - - -<br>
mirror 372G 353G 19.0G - 88% 94%<br>
scsi-35000c5003017155f - - - - - -<br>
scsi-35000c500301715a7 - - - - - -<br>
<br>
When we try to delete files, we get the error message:<br>
rm: cannot remove XXXXX: No space left on device<br>
<br>
Is there a way to unlink files and free up inodes?<br>
<br>
Is it possible to expand the existing zpool and filesystem for the MDT?<br>
<br>
Is it possible to do a backup of just our MDT? If so, how?<br>
<br>
-- <br>
Regards,<br>
-liam<br>
<br>
-There are uncountably more irrational fears than rational ones. -P. Dolan<br>
Liam Forbes <a href="mailto:loforbes@alaska.edu" target="_blank">loforbes@alaska.edu</a> ph:<br>
907.450.8618<br>
UAF GI Research Computing Systems Manager<br>
<a href="https://calendly.com/ualoforbes/30min" rel="noreferrer" target="_blank">https://calendly.com/ualoforbes/30min</a><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220914/5e9b7fde/attachment-0001.htm" rel="noreferrer" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220914/5e9b7fde/attachment-0001.htm</a>><br><br>
</blockquote></div></div>