[lustre-discuss] lustre 2.5.3 ost not draining

Dilger, Andreas andreas.dilger at intel.com
Tue Jul 28 21:51:38 PDT 2015


Setting it degraded means the MDS will avoid allocations on that OST
unless there aren't enough OSTs to meet the request (e.g. stripe_count =
-1), so it should work.

That is actually a very interesting workaround for this problem, and it
will work for older versions of Lustre as well.  It doesn't disable the
OST completely, which is fine if you are doing space balancing (and may
even be desirable to allow apps that need more bandwidth for a widely
striped file), but it isn't good if you are trying to empty the OST
completely to remove it.

It looks like another approach would be to mark the OST as having no free
space using OBD_FAIL_OST_ENOINO (0x229) fault injection on that OST:

   lctl set_param fail_loc=0x229 fail_val=<ost_index>

This would cause the OST to return 0 free inodes from OST_STATFS for the
specified OST index, and the MDT would skip this OST completely.  To
disable all of the OSTs on an OSS use <ost_index> = -1.  It isn't possible
to selectively disable a subset of OSTs using this method.  The
OBD_FAIL_OST_ENOINO fail_loc has been available since Lustre 2.2, which
covers all of the 2.4+ versions that are affected by this issue.

If this mechanism works for you (it should, as this fail_loc is used
during regular testing) I'd be obliged if someone could file an LUDOC bug
so the manual can be updated.

Cheers, Andreas

On 2015/07/10, 10:10 AM, "Alexander I Kulyavtsev"
<lustre-discuss-bounces at lists.lustre.org on behalf of aik at fnal.gov> wrote:

>I think so, try it.
>We do set ost degraded on 1.8 when ost nears 95% and we migrate data to
>another ost.
>On 1.8 lfs_migrate uses 'rm' and objects are indeed deallocated.
>
>Alex
>
>On Jul 10, 2015, at 10:55 AM, Kurt Strosahl <strosahl at jlab.org> wrote:
>
>> Will that let deletes happen against it?
>>w/r,
>> Kurt
>>----- Original Message -----
>> From: "aik" <aik at fnal.gov>
>> To: "Kurt Strosahl" <strosahl at jlab.org>
>> Cc: "aik" <aik at fnal.gov>, "Sean Brisbane"
>><sean.brisbane at physics.ox.ac.uk>, lustre-discuss at lists.lustre.org
>> Sent: Friday, July 10, 2015 11:52:00 AM
>> Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining
>>Hi Kurt,to keep traffic from almost full OST we usually set ost in
>>degraded mode like described in manual:
>>> 
>>> Handling Degraded OST RAID Arrays
>>> 
>>> To mark the OST as degraded, use:
>>> lctl set_param obdfilter.{OST_name}.degraded=1
>>> 
>>> Alex.
>>>On Jul 10, 2015, at 10:13 AM, Kurt Strosahl <strosahl at jlab.org> wrote:
>>> 
>>>> No, I'm aware of why the ost is getting new writes... it is because I
>>>>had to set the qos_threshold_rr to 100 due to
>>>>https://jira.hpdd.intel.com/browse/LU-5778  (I have an ost that has to
>>>>be ignored due to terrible write performance...)
>>>> 
>>>> w/r,
>>>> Kurt
>>>> 
>>>> ----- Original Message -----
>>>> From: "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
>>>> To: "Kurt Strosahl" <strosahl at jlab.org>
>>>> Cc: "Patrick Farrell" <paf at cray.com>,
>>>>"lustre-discuss at lists.lustre.org" <lustre-discuss at lists.lustre.org>
>>>> Sent: Friday, July 10, 2015 11:04:27 AM
>>>> Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining
>>>> 
>>>> Dear Kurt,
>>>> 
>>>> Apologies.  After leaving it some number of days it did *not* clean
>>>>itself up, but I feel that some number of days is long enough to
>>>>verify 
>>>>that it is a problem.
>>>> 
>>>> Sounds like you have another issue if the OST is not being marked as
>>>>full and writes are not being re-allocated to other OSTS .  I also
>>>>have 
>>>>that second issue on my system as well and I have only workarounds to
>>>>offer you for the problem.
>>>> 
>>>> Thanks,
>>>> Sean
>>>> 
>>>> -----Original Message-----
>>>> From: Kurt Strosahl [mailto:strosahl at jlab.org]
>>>> Sent: 10 July 2015 16:01
>>>> To: Sean Brisbane
>>>> Cc: Patrick Farrell; lustre-discuss at lists.lustre.org
>>>> Subject: Re: [lustre-discuss] lustre 2.5.3 ost not draining
>>>> 
>>>> The problem there is that I cannot afford to leave it "some number of
>>>>days"... it is at 97% full, so new writes are going to it faster then
>>>>it can clean itself off.
>>>> 
>>>> w/r,
>>>> Kurt
>>>> 
>>>> ----- Original Message -----
>>>> From: "Sean Brisbane" <sean.brisbane at physics.ox.ac.uk>
>>>> To: "Patrick Farrell" <paf at cray.com>, "Kurt Strosahl"
>>>><strosahl at jlab.org>
>>>> Cc: lustre-discuss at lists.lustre.org
>>>> Sent: Friday, July 10, 2015 10:44:39 AM
>>>> Subject: RE: [lustre-discuss] lustre 2.5.3 ost not draining
>>>> 
>>>> Hi,
>>>> 
>>>> The 'space not freed' issue also happened to me and I left it 'some
>>>>number of days'  I don't recall how many, it was a while back.
>>>> 
>>>> Cheers,
>>>>Sean

Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division




More information about the lustre-discuss mailing list