[lustre-discuss] lustre 2.5.3 ost not draining

[Shawn Hall]

Note that the “lfs migrate” command (not the lfs_migrate script, which I believe uses the lfs migrate command) has a --block option.  When performing the migrate, this will block all other I/O to that file.  That should help you win the race in case you’re competing for a file.



Shawn

On 7/10/15, 2:53 PM, "Kurt Strosahl" <strosahl at jlab.org> wrote:

>Yes, there are quite a few issues with lustre 2.5.3 (it would be sad if it wasn't so frustrating... 1.8.x was solid).
>
>The full osts have a higher index then the one that broke the weighted round robin... plus all the ones above the most recent are exceptionally full (>=80%).  I'm not sure how I'm going to go forward, I've heard that maybe an unmount / mount of the osts would push a purge. I'm also compiling a list of all the files on the ost... the idea being that I could then enable it, and launch multiple lfs_migrates... trying to race everyone else using the file system.  I think I'd have the advantage, as my moves would be targeted directly to the ost, while the other writes would just land where ever they could.
>
>w/r,
>Kurt
>
>----- Original Message -----
>From: "Shawn Hall" <shawn.hall at nag.com>
>To: "Kurt Strosahl" <strosahl at jlab.org>, "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
>Cc: lustre-discuss at lists.lustre.org
>Sent: Friday, July 10, 2015 11:49:06 AM
>Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining
>
>It sounds like you have a couple of issues that are working against each other then.  You’ll probably need to fight one at a time.
>
>
>
>My recommendation of clearing up file system space still stands.  I don’t have scientific proof, but giving Lustre more space to work with definitely helps.
>
>Does your full OST have a lower index than your slow OST?  Then you could disable the slow one (and because of the bug everything above it) and let space clear up on the full one.
>
>Beyond that you might have to get creative and try something similar to Tommy.  Migrate data but manually specify stripe offsets.
>
>Shawn
>
>On 7/10/15, 11:13 AM, "lustre-discuss on behalf of Kurt Strosahl" <lustre-discuss-bounces at lists.lustre.org on behalf of strosahl at jlab.org> wrote:
>
>>No, I'm aware of why the ost is getting new writes... it is because I had to set the qos_threshold_rr to 100 due to https://jira.hpdd.intel.com/browse/LU-5778  (I have an ost that has to be ignored due to terrible write performance...)
>>
>>w/r,
>>Kurt
>>
>>----- Original Message -----
>>From: "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
>>To: "Kurt Strosahl" <strosahl at jlab.org>
>>Cc: "Patrick Farrell" <paf at cray.com>, "lustre-discuss at lists.lustre.org" <lustre-discuss at lists.lustre.org>
>>Sent: Friday, July 10, 2015 11:04:27 AM
>>Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining
>>
>>Dear Kurt,
>>
>>Apologies.  After leaving it some number of days it did *not* clean itself up, but I feel that some number of days is long enough to verify that it is a problem.
>>
>>Sounds like you have another issue if the OST is not being marked as full and writes are not being re-allocated to other OSTS .  I also have that second issue on my system as well and I have only workarounds to offer you for the problem.
>>
>>Thanks,
>>Sean
>>
>>-----Original Message-----
>>From: Kurt Strosahl [mailto:strosahl at jlab.org] 
>>Sent: 10 July 2015 16:01
>>To: Sean Brisbane
>>Cc: Patrick Farrell; lustre-discuss at lists.lustre.org
>>Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining
>>
>>The problem there is that I cannot afford to leave it "some number of days"... it is at 97% full, so new writes are going to it faster then it can clean itself off.
>>
>>w/r,
>>Kurt
>>
>>----- Original Message -----
>>From: "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
>>To: "Patrick Farrell" <paf at cray.com>, "Kurt Strosahl" <strosahl at jlab.org>
>>Cc: lustre-discuss at lists.lustre.org
>>Sent: Friday, July 10, 2015 10:44:39 AM
>>Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining
>>
>>Hi,
>>
>>The 'space not freed' issue also happened to me and I left it 'some number of days'  I don't recall how many, it was a while back.
>>
>>Cheers,
>>Sean
>>_______________________________________________
>>lustre-discuss mailing list
>>lustre-discuss at lists.lustre.org
>>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org