<div dir="ltr">OK. Thanks to everyone who answered.<br clear="all"><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><br>.. Lana (<a href="mailto:lana.deere@gmail.com" target="_blank">lana.deere@gmail.com</a>)<br><br><br></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 19, 2020 at 11:24 PM Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">
On Mar 19, 2020, at 12:56, Lana Deere <<a href="mailto:lana.deere@gmail.com" target="_blank">lana.deere@gmail.com</a>> wrote:<br>
<div>
<blockquote type="cite"><br>
<div>
<div dir="ltr">
<div>The MDT shows 6% of storage in use and 9% of inodes in use. The OST, however, shows 46% of storage and 100% of inodes in use (12 free). (There is only one OST on this particular file system.) I suppose if a lot of files are deleted, the system
will recover, but then I'm not sure why I couldn't create any files after deleting a few.</div>
<div><br>
</div>
<div>Is there any way to increase the number of inodes on the OST without losing the data currently on the filesystem?</div>
</div>
</div>
</blockquote>
<div><br>
</div>
That depends on the storage that the OST is on. You can use "resize2fs" on the OST if the underlying stoage is LVM or similar that can be resized. Increasing the OST size adds inodes proportional to the added capacity.</div>
<div><br>
</div>
<div>The more common option for adding capacity to Lustre is to add another OST. Based on your earlier comments, you should probably double the inodes per unit space (reduce the bytes per inode ratio like "mkfs.lustre --mkfsoptions='-i 131072' ..." or similar)
compared to the first OST. You can work out the average bytes per inode on the OST based on the (used OST capacity / used inodes).</div>
<div><br>
</div>
<div>Cheers, Andreas</div>
<div> <br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div dir="ltr"><br>
.. Lana (<a href="mailto:lana.deere@gmail.com" target="_blank">lana.deere@gmail.com</a>)<br>
<br>
<br>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Mar 19, 2020 at 2:13 PM Degremont, Aurelien <<a href="mailto:degremoa@amazon.com" target="_blank">degremoa@amazon.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="FR">
<div>
<p class="MsoNormal"><span lang="EN-US">Hi Lana,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Lustre dispatches the data across several servers, MDTs and OSTs. It is likely that one of this OST is full.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">To see the usage per sub-component, you should check:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">lfs df -h<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">lfs df -ih<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">See if this reports one OSTs or MDT is full.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Aurélien<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div style="border-color:rgb(181,196,223) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b><span style="font-size:12pt">De :
</span></b><span style="font-size:12pt">lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> au nom de Lana Deere <<a href="mailto:lana.deere@gmail.com" target="_blank">lana.deere@gmail.com</a>><br>
<b>Date : </b>jeudi 19 mars 2020 à 19:08<br>
<b>À : </b>"<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
<b>Objet : </b>[EXTERNAL] [lustre-discuss] "no space on device"</span> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I have a Lustre 2.12 setup running on CentOS 7. It has been working fine for some months but last night one of my users tried to untar a large file, which (among other things) created a single directory containing
several million subdirectories. At that point the untar failed, reporting "no space on device". All attempts to create a file on this Lustre system now produce the same error message, but "df" and "df -i" indicate there is plenty of space and inodes left.
I checked the mount point on the metadata node and it appears to have plenty of space left too.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I can list directories and view files on this filesystem. I can delete files or directories on it. But even after removing a few files and a directory I cannot create a new file.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">If anyone can offer some help here it would be appreciated.<u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:12pt;margin-left:35.4pt">
<br>
.. Lana (<a href="mailto:lana.deere@gmail.com" target="_blank">lana.deere@gmail.com</a>)<br>
<br>
<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote>
</div>
<br>
<div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<br>
</div>
<br>
</div>
</blockquote></div>