[lustre-discuss] Full OST

[Alastair Basden]

Hi,

Thanks.  It seems odd that the OST reports having no files on it at all 
(I'd expect several hundred, based on the number of files on the other 
OSTs).

Unless an open-unlinked file would have that effect on lfs find, but I 
don't think it should.

I'm not sure whether lsof would help - the clients could well have files 
open writing to other OSTs.  But have tested, and lsof doesn't return any 
open files that are on that OST on any of the nodes.

Cheers,
Alastair.



On Fri, 3 Sep 2021, Degremont, Aurelien wrote:

> [EXTERNAL EMAIL]
>
> Hi
>
> It could be a bug, but most of the time, this is due to an open-unlinked file, typically a log file which is still in use and some processes keep writing to it until it fills the OSTs it is using.
>
> Look for such files on your clients (use lsof).
>
> Aurélien
>
>
> Le 03/09/2021 09:50, « lustre-discuss au nom de Alastair Basden » <lustre-discuss-bounces at lists.lustre.org au nom de a.g.basden at durham.ac.uk> a écrit :
>
>    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
>    Hi,
>
>    We have a file system where each OST is a single SSD.
>
>    One of those is reporting as 100% full (lfs df -h /snap8):
>    snap8-OST004d_UUID          5.8T        2.0T        3.5T  37% /snap8[OST:77]
>    snap8-OST004e_UUID          5.8T        5.5T        7.5G 100% /snap8[OST:78]
>    snap8-OST004f_UUID          5.8T        2.0T        3.4T  38% /snap8[OST:79]
>
>    However, I can't find any files on it:
>    lfs find --ost snap8-OST004e /snap8/
>    returns nothing.
>
>    I guess that it has filled up, and that there is some bug or other that is
>    now preventing proper behaviour - but I could be wrong.
>
>    Does anyone have any suggestions?
>
>    Essentially, I'd like to find some of the files and delete or migrate
>    some, and thus return it to useful production.
>
>    Cheers,
>    Alastair.
>    _______________________________________________
>    lustre-discuss mailing list
>    lustre-discuss at lists.lustre.org
>    http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>