[Lustre-discuss] Cannot get an OST to activate

[Andreas Dilger]

On 2010-09-10, at 08:21, Bob Ball wrote:
> Now, we move on to the "bad" OST.  First, I did an lfs_find yesterday on just this OST and came up with some 8000 files before it seemed to cease output.  So, I expected to see SOMETHING on the physical disk.  But, in fact, the /tmp/objects.sdc showed no content whatsoever?  Just blank lines, and a direct look at the ls output confirmed that.  And so, the confusion began.
> 
> LAST_ID is, indeed, zero.  
> 
> I am checking with my co-conspirator in this.  It is _possible_ that this OST ID was re-used after a machine was dropped from our system due to non-recoverable disk/system errors.  So, in my mind, that means it is possible that the current OST really IS empty of content.  Is that really what is meant by getting this kind of output from the ls command for all 9 or 10 of these directories?
> 
> /mnt/ost/O/0/d8:
> total 0
> 
> /mnt/ost/O/0/d9:
> total 0

There should be 32 such directories.

> Assuming the disk really is empty then, and LAST_ID really is zero, shall I then leave it at zero, and follow the recommendation of page 23-14, ie, just shut down again, delete the lov_objid file on the MDS, and restart the system?  Certainly the value at the correct index (29) is definitely hosed:
> # od -Ax -td8 /mnt/mdt/lov_objid
> (snip)
> 0000d0               292648               346413
> 0000e0                68225 -7137254917378053186
> 0000f0                59064                59607
> 000100                59227                59414

Yes, that is definitely hosed.  Deleting the lov_objid file from the MDS and remounting the MDS should fix this value.  You could also just binary edit the file and set this to 1.

> Bob Ball wrote:
>> Thank you, Bern.  "df" claims there is some 442MB of data on the volume, compared to neighbors with 285GB.  That could well be a fragment of a single, unsuccessful transfer attempt.  I can run lfs_find on it though and see what comes back.  Was having problems earlier, thought I got files back from that command, but other problems on our cluster confused that result.  We will recheck.
>> 
>> bob
>> 
>> Bernd Schubert wrote:
>>> On Friday, September 03, 2010, Bob Ball wrote:
>>>   
>>> 
>>>> We added a new OSS to our 1.8.4 Lustre installation.  It has 6 OST of
>>>> 8.9TB each.  Within a day of having these on-line, one OST stopped
>>>> accepting new files.  I cannot get it to activate.  The other 5 seem fine.
>>>> 
>>>> On the MDS "lctl dl" shows it IN, but not UP, and files can be read from
>>>> it: 33 IN osc umt3-OST001d-osc umt3-mdtlov_UUID 5
>>>> 
>>>> However, I cannot get it to re-activate:
>>>> lctl --device umt3-OST001d-osc activate
>>>> 
>>>>     
>>>> 
>>> 
>>> [...]
>>> 
>>> 
>>>   
>>> 
>>>> LustreError: 4697:0:(filter.c:3172:filter_handle_precreate())
>>>> umt3-OST001d: ignoring bogus orphan destroy request: obdid
>>>> 11309489156331498430 last_id 0
>>>> 
>>>> Can anyone tell me what must be done to recover this disk volume?
>>>>     
>>>> 
>>> 
>>> Check out section 23.3.9 in the Lustre manual ("How to Fix a Bad LAST_ID on an 
>>> OST).
>>> 
>>> It is on my TODO list to write tool to automatically correct the "lov_objid", 
>>> but as of now I don't have it yet. Somehow your lov_objid file has a 
>>> completely wrong value for this OST.
>>> Now, when you say "files can be read from it", are you sure there are already 
>>> files on that OST? Because the error message says that the last_id is zero and 
>>> so you should not have a single file on it. If that is also wrong, you will 
>>> need to correct it as well. You can do that manually, or you can use a patched 
>>> e2fsprogs version, that will do that for you
>>> 
>>> Patches are here:
>>> 
>>> https://bugzilla.lustre.org/show_bug.cgi?id=22734
>>> 
>>> 
>>> Packages can be found on my home page:
>>> 
>>> http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/e2fsprogs/
>>> 
>>> 
>>> 
>>> If you want to do it automatically, you will need to create a lfsck mdsdb file 
>>> (the hdr file is sufficient, see the lfsck section in the manual) and then you 
>>> will need to run e2fsck for that OST as if you want to create an OSTDB file. 
>>> That will start pass6, and if you then run e2fsck *without* "-n", so in 
>>> correcting mode, it will correct the LAST_ID file to what it finds on disk. 
>>> With "-v" it will also tell you the old and the new value and then you will 
>>> need to put that value properly coded into the MDS lov_objid file.
>>> 
>>> 
>>> Be careful and create backups of the lov_objid and LAST_ID files.
>>> 
>>> 
>>> Hope it helps,
>>> Bern
>>> 
>>> 
>>> 
>>>   
>>> 
>> 
>> _______________________________________________
>> Lustre-discuss mailing list
>> 
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>> 
>>   
>> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss


Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.