[lustre-discuss] Full OST, any way of avoiding it without hanging?

Jason Williams jasonw at jhu.edu
Mon Jan 7 08:47:09 PST 2019

So I found this: http://wiki.lustre.org/Handling_Full_OSTs which is what I thought I had followed before but ran into hang issues.  I did some quick testing with this and found that:

1. if I deactivate the OST in the MDS, no new files appear to be created on that OST (expected behavior) and no hangs.

2. If I first create a file on the OST with it activated, then deactivate the OST, and OVERWRITE a file what was spanned on that OST, the indexes stay the same and the file successfully overwrites (the file spanned 4 OSTs, so perhaps a little more testing with a single OST in the index is necessary)

3. Deactivating the OST shows it as inactive in the MDS but UP in the Client. (not expected.)

4. I am able to delete a file that spans that OST with the OST deactivated, no hang.

I think the only thing here that concerns me a bit is #2.

Jason Williams
Assistant Director
Systems and Data Center Operations.
Maryland Advanced Research Computing Center (MARCC)
Johns Hopkins University
jasonw at jhu.edu<mailto:jasonw at jhu.edu>

From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of Jason Williams <jasonw at jhu.edu>
Sent: Sunday, January 6, 2019 5:22:16 PM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: lustre-discuss at lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

Hi Rick,
I thought what I had done was disable it on the MDS, but perhaps I was following the wrong instructions. Do you know where the best instructions for what you are describing can be found? I would be willing to try again.

Sent you tersely from my phone
Jason Williams
From: Mohr Jr, Richard Frank (Rick Mohr) <rmohr at utk.edu>
Sent: Sunday, January 6, 2019 4:56 PM
To: Jason Williams
Cc: lustre-discuss at lists.lustre.org
Subject: Re: [lustre-discuss] Full OST, any way of avoiding it without hanging?

> On Jan 5, 2019, at 9:49 PM, Jason Williams <jasonw at jhu.edu> wrote:
> I have looked around the internet and found you can disable an OST, but when I have tried that, any writes (including deletes) to the OST hang the clients indefinitely. Does anyone know a way to make an OST basically "read-only" with the exception of deletes so we can work to clear out the OST?

What command did you use to disable the OST?

There is a way to disable the OST on all the clients, but there is also a way to deactivate it on the MDS. The latter method should prevent the MDS from allocating any new files to the OST, but still allow clients to read and delete files on that OST.

Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20190107/001d2252/attachment-0001.html>

More information about the lustre-discuss mailing list