[lustre-discuss] Full OST, any way of avoiding it without hanging?

Jeff Johnson jeff.johnson at aeoncomputing.com
Sat Jan 5 19:56:03 PST 2019


Jason,

If there are files located on the full OST that you can delete then delete
them without trying to deactivate the OST. There is a process where you can
find files solely located on the full OST using lfs then deactivate the
full OST, copy the files to new files with some suffix (.new perhaps) so
they end up on other OSTs, delete the original files, rename the new files
to the old file names (thereby removing the suffix used previously) and
then reactivating the deactivated OST. You might be lucky enough to find a
few very large single stripe files located on that OST where moving the
file gets you easy gains.

You *should* be able to delete a file that resides on an inactive OST but
I’m not over a keyboard where I can do it before making the assertion it is
possible.

—Jeff


On Sat, Jan 5, 2019 at 18:49 Jason Williams <jasonw at jhu.edu> wrote:

> Hello,
>
>
> We have a lustre system (version 2.10.4) that has unfortunately fallen
> victim to a 100% full OST... Every time we clear some space on it, the
> system fills it right back up again.
>
>
> I have looked around the internet and found you can disable an OST, but
> when I have tried that, any writes (including deletes) to the OST hang the
> clients indefinitely.  Does anyone know a way to make an OST basically
> "read-only" with the exception of deletes so we can work to clear out the
> OST?  Or better yet, a way to "drain" or move files off an OST with a
> script (keeping in mind it might not be known if the files are in use at
> the time).  Or even a way to tell lustre "Hey don't write any new data
> here, but reading and removing data is OK."
>
>
>
>
> --
> Jason Williams
> Assistant Director
> Systems and Data Center Operations.
> Maryland Advanced Research Computing Center (MARCC)
> Johns Hopkins University
> jasonw at jhu.edu
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
-- 
------------------------------
Jeff Johnson
Co-Founder
Aeon Computing

jeff.johnson at aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x1001   f: 858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite C - San Diego, CA 92117

High-Performance Computing / Lustre Filesystems / Scale-out Storage
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20190105/a678a050/attachment.html>


More information about the lustre-discuss mailing list