[lustre-discuss] lustre 2.5.3 ost not draining

[Kurt Strosahl]

Unfortunately... due to https://jira.hpdd.intel.com/browse/LU-5778, I'm stuck at just a flat round robin.

----- Original Message -----
From: "Shawn Hall" <shawn.hall at nag.com>
To: "Kurt Strosahl" <strosahl at jlab.org>, "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
Cc: lustre-discuss at lists.lustre.org
Sent: Friday, July 10, 2015 11:08:57 AM
Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining

I agree it does take some time to clear delete the objects - unfortunately I don’t know if there’s a way to speed that up. Object allocation, even when weighted, I believe is still probabilistic, so you can never guarantee that OST won’t be written to without deactivating it.  And people can append to files even if new objects aren’t being allocated to that OST.

I see everyone’s run across LU-4825 too, so documenting the issues there will help the cause.  


The only way those objects are going to get removed is by leaving things active.  In the meantime, I would try:

1) Talking with your big users, asking them to remove or archive data. If you have more than one file system lying around you might try moving some data to those too. This will make the biggest difference and what ultimately got us out of our pickle. 
2) Temporarily adjusting some of the object allocator tuning.  The allocator has likely started trying to give preference to the OSTs with more free space, but you can help it to be even more aggressive.  Section 18.5.2 and 18.5.3 of the latest Lustre manual:

https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#idp5261568

Shawn


On 7/10/15, 11:00 AM, "lustre-discuss on behalf of Kurt Strosahl" <lustre-discuss-bounces at lists.lustre.org on behalf of strosahl at jlab.org> wrote:

>The problem there is that I cannot afford to leave it "some number of days"... it is at 97% full, so new writes are going to it faster then it can clean itself off.
>
>w/r,
>Kurt
>
>----- Original Message -----
>From: "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
>To: "Patrick Farrell" <paf at cray.com>, "Kurt Strosahl" <strosahl at jlab.org>
>Cc: lustre-discuss at lists.lustre.org
>Sent: Friday, July 10, 2015 10:44:39 AM
>Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining
>
>Hi,
>
>The 'space not freed' issue also happened to me and I left it 'some number of days'  I don't recall how many, it was a while back.
>
>Cheers,
>Sean
>_______________________________________________
>lustre-discuss mailing list
>lustre-discuss at lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org