[lustre-discuss] lustre 2.5.3 ost not draining

Kurt Strosahl strosahl at jlab.org
Sat Jul 11 18:03:23 PDT 2015


Thanks,

   I'll have to see if I can run this test myself.  Did you notice if the "inactive" status persisted through the unmount/remount?

w/r,
Kurt

----- Original Message -----
From: "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
To: "Kurt Strosahl" <strosahl at jlab.org>, "Shawn Hall" <shawn.hall at nag.com>
Cc: lustre-discuss at lists.lustre.org
Sent: Saturday, July 11, 2015 4:29:42 AM
Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining

Dear Kurt,

I have the same issue as you in that deleted files on deactivated OST could not be cleaned up even after re-activation. It was on my todo list to work out at some point how to get around this. I was told that an unmount/mount cycle on the servers will trigger a clean-up.  

I have just performed the experiment and it was in fact the MDT not the OST which needed to be unmounted and re-mounted in my case.

Unmounting and remounting the OST during this process appeared to make no difference either way.

All the best,
Sean


________________________________________
From: Kurt Strosahl [strosahl at jlab.org]
Sent: 10 July 2015 19:53
To: Shawn Hall
Cc: Sean Brisbane; lustre-discuss at lists.lustre.org
Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining

Yes, there are quite a few issues with lustre 2.5.3 (it would be sad if it wasn't so frustrating... 1.8.x was solid).

The full osts have a higher index then the one that broke the weighted round robin... plus all the ones above the most recent are exceptionally full (>=80%).  I'm not sure how I'm going to go forward, I've heard that maybe an unmount / mount of the osts would push a purge. I'm also compiling a list of all the files on the ost... the idea being that I could then enable it, and launch multiple lfs_migrates... trying to race everyone else using the file system.  I think I'd have the advantage, as my moves would be targeted directly to the ost, while the other writes would just land where ever they could.

w/r,
Kurt

----- Original Message -----
From: "Shawn Hall" <shawn.hall at nag.com>
To: "Kurt Strosahl" <strosahl at jlab.org>, "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
Cc: lustre-discuss at lists.lustre.org
Sent: Friday, July 10, 2015 11:49:06 AM
Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining

It sounds like you have a couple of issues that are working against each other then.  You’ll probably need to fight one at a time.



My recommendation of clearing up file system space still stands.  I don’t have scientific proof, but giving Lustre more space to work with definitely helps.

Does your full OST have a lower index than your slow OST?  Then you could disable the slow one (and because of the bug everything above it) and let space clear up on the full one.

Beyond that you might have to get creative and try something similar to Tommy.  Migrate data but manually specify stripe offsets.

Shawn

On 7/10/15, 11:13 AM, "lustre-discuss on behalf of Kurt Strosahl" <lustre-discuss-bounces at lists.lustre.org on behalf of strosahl at jlab.org> wrote:

>No, I'm aware of why the ost is getting new writes... it is because I had to set the qos_threshold_rr to 100 due to https://jira.hpdd.intel.com/browse/LU-5778  (I have an ost that has to be ignored due to terrible write performance...)
>
>w/r,
>Kurt
>
>----- Original Message -----
>From: "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
>To: "Kurt Strosahl" <strosahl at jlab.org>
>Cc: "Patrick Farrell" <paf at cray.com>, "lustre-discuss at lists.lustre.org" <lustre-discuss at lists.lustre.org>
>Sent: Friday, July 10, 2015 11:04:27 AM
>Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining
>
>Dear Kurt,
>
>Apologies.  After leaving it some number of days it did *not* clean itself up, but I feel that some number of days is long enough to verify that it is a problem.
>
>Sounds like you have another issue if the OST is not being marked as full and writes are not being re-allocated to other OSTS .  I also have that second issue on my system as well and I have only workarounds to offer you for the problem.
>
>Thanks,
>Sean
>
>-----Original Message-----
>From: Kurt Strosahl [mailto:strosahl at jlab.org]
>Sent: 10 July 2015 16:01
>To: Sean Brisbane
>Cc: Patrick Farrell; lustre-discuss at lists.lustre.org
>Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining
>
>The problem there is that I cannot afford to leave it "some number of days"... it is at 97% full, so new writes are going to it faster then it can clean itself off.
>
>w/r,
>Kurt
>
>----- Original Message -----
>From: "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
>To: "Patrick Farrell" <paf at cray.com>, "Kurt Strosahl" <strosahl at jlab.org>
>Cc: lustre-discuss at lists.lustre.org
>Sent: Friday, July 10, 2015 10:44:39 AM
>Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining
>
>Hi,
>
>The 'space not freed' issue also happened to me and I left it 'some number of days'  I don't recall how many, it was a while back.
>
>Cheers,
>Sean
>_______________________________________________
>lustre-discuss mailing list
>lustre-discuss at lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


More information about the lustre-discuss mailing list