[lustre-discuss] lustre 2.5.3 ost not draining

[Shawn Hall]

It sounds like you have a couple of issues that are working against each other then.  You’ll probably need to fight one at a time.



My recommendation of clearing up file system space still stands.  I don’t have scientific proof, but giving Lustre more space to work with definitely helps.

Does your full OST have a lower index than your slow OST?  Then you could disable the slow one (and because of the bug everything above it) and let space clear up on the full one.

Beyond that you might have to get creative and try something similar to Tommy.  Migrate data but manually specify stripe offsets.

Shawn

On 7/10/15, 11:13 AM, "lustre-discuss on behalf of Kurt Strosahl" <lustre-discuss-bounces at lists.lustre.org on behalf of strosahl at jlab.org> wrote:

>No, I'm aware of why the ost is getting new writes... it is because I had to set the qos_threshold_rr to 100 due to https://jira.hpdd.intel.com/browse/LU-5778  (I have an ost that has to be ignored due to terrible write performance...)
>
>w/r,
>Kurt
>
>----- Original Message -----
>From: "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
>To: "Kurt Strosahl" <strosahl at jlab.org>
>Cc: "Patrick Farrell" <paf at cray.com>, "lustre-discuss at lists.lustre.org" <lustre-discuss at lists.lustre.org>
>Sent: Friday, July 10, 2015 11:04:27 AM
>Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining
>
>Dear Kurt,
>
>Apologies.  After leaving it some number of days it did *not* clean itself up, but I feel that some number of days is long enough to verify that it is a problem.
>
>Sounds like you have another issue if the OST is not being marked as full and writes are not being re-allocated to other OSTS .  I also have that second issue on my system as well and I have only workarounds to offer you for the problem.
>
>Thanks,
>Sean
>
>-----Original Message-----
>From: Kurt Strosahl [mailto:strosahl at jlab.org] 
>Sent: 10 July 2015 16:01
>To: Sean Brisbane
>Cc: Patrick Farrell; lustre-discuss at lists.lustre.org
>Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining
>
>The problem there is that I cannot afford to leave it "some number of days"... it is at 97% full, so new writes are going to it faster then it can clean itself off.
>
>w/r,
>Kurt
>
>----- Original Message -----
>From: "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
>To: "Patrick Farrell" <paf at cray.com>, "Kurt Strosahl" <strosahl at jlab.org>
>Cc: lustre-discuss at lists.lustre.org
>Sent: Friday, July 10, 2015 10:44:39 AM
>Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining
>
>Hi,
>
>The 'space not freed' issue also happened to me and I left it 'some number of days'  I don't recall how many, it was a while back.
>
>Cheers,
>Sean
>_______________________________________________
>lustre-discuss mailing list
>lustre-discuss at lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org