<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
OK, I tried this morning to follow the information/procedures from
section 23.3.9 of the user manual, and succeeded in confusing myself
admirably.<br>
<br>
Took lustre completely offline, then first checked the LAST_ID on a
known, good OST. I found this kind of thing:<br>
# od -Ax -td4 /mnt/ost/last_rcvd | more<br>
For ost11, the 8c index value is 28<br>
debugfs -c -R 'dump /O/0/LAST_ID /tmp/LAST_ID' /dev/sdb ; od -Ax -td8
/tmp/LAST_ID<br>
debugfs 1.41.10.sun2 (24-Feb-2010)<br>
/dev/sdb: catastrophic mode - not reading inode or group bitmaps<br>
000000 68321<br>
000008<br>
<br>
(gdb) p /x 68321<br>
$1 = 0x10ae1<br>
(gdb)<br>
[root@umdist03 ~]# cat /tmp/LAST_ID.asc<br>
0000000: e10a 0100 0000 0000 á.......<br>
>From this I can see how to edit the LAST_ID.asc for use in the repair
procedure. All other information was consistent, the /tmp/objects.sdb
ended with this LAST_ID object.<br>
<br>
Now, we move on to the "bad" OST. First, I did an lfs_find yesterday
on just this OST and came up with some 8000 files before it seemed to
cease output. So, I expected to see SOMETHING on the physical disk.
But, in fact, the /tmp/objects.sdc showed no content whatsoever? Just
blank lines, and a direct look at the ls output confirmed that. And
so, the confusion began.<br>
<br>
LAST_ID is, indeed, zero. <br>
<br>
I am checking with my co-conspirator in this. It is _possible_ that
this OST ID was re-used after a machine was dropped from our system due
to non-recoverable disk/system errors. So, in my mind, that means it
is possible that the current OST really IS empty of content. Is that
really what is meant by getting this kind of output from the ls command
for all 9 or 10 of these directories?<br>
<br>
/mnt/ost/O/0/d8:<br>
total 0<br>
<br>
/mnt/ost/O/0/d9:<br>
total 0<br>
<br>
Assuming the disk really is empty then, and LAST_ID really is zero,
shall I then leave it at zero, and follow the recommendation of page
23-14, ie, just shut down again, delete the lov_objid file on the MDS,
and restart the system? Certainly the value at the correct index (29)
is definitely hosed:<br>
# od -Ax -td8 /mnt/mdt/lov_objid<br>
(snip)<br>
0000d0 292648 346413<br>
0000e0 68225 -7137254917378053186<br>
0000f0 59064 59607<br>
000100 59227 59414<br>
<br>
Thanks,<br>
bob<br>
<br>
<br>
Bob Ball wrote:
<blockquote cite="mid:4C81716E.8070107@umich.edu" type="cite">
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
Thank you, Bern. "df" claims there is some 442MB of data on the
volume, compared to neighbors with 285GB. That could well be a
fragment of a single, unsuccessful transfer attempt. I can run
lfs_find on it though and see what comes back. Was having problems
earlier, thought I got files back from that command, but other problems
on our cluster confused that result. We will recheck.<br>
<br>
bob<br>
<br>
Bernd Schubert wrote:
<blockquote cite="mid:201009032322.25265.bs_lists@aakef.fastmail.fm"
type="cite">
<pre wrap="">On Friday, September 03, 2010, Bob Ball wrote:
</pre>
<blockquote type="cite">
<pre wrap="">We added a new OSS to our 1.8.4 Lustre installation. It has 6 OST of
8.9TB each. Within a day of having these on-line, one OST stopped
accepting new files. I cannot get it to activate. The other 5 seem fine.
On the MDS "lctl dl" shows it IN, but not UP, and files can be read from
it: 33 IN osc umt3-OST001d-osc umt3-mdtlov_UUID 5
However, I cannot get it to re-activate:
lctl --device umt3-OST001d-osc activate
</pre>
</blockquote>
<pre wrap=""><!---->
[...]
</pre>
<blockquote type="cite">
<pre wrap="">LustreError: 4697:0:(filter.c:3172:filter_handle_precreate())
umt3-OST001d: ignoring bogus orphan destroy request: obdid
11309489156331498430 last_id 0
Can anyone tell me what must be done to recover this disk volume?
</pre>
</blockquote>
<pre wrap=""><!---->
Check out section 23.3.9 in the Lustre manual ("How to Fix a Bad LAST_ID on an
OST).
It is on my TODO list to write tool to automatically correct the "lov_objid",
but as of now I don't have it yet. Somehow your lov_objid file has a
completely wrong value for this OST.
Now, when you say "files can be read from it", are you sure there are already
files on that OST? Because the error message says that the last_id is zero and
so you should not have a single file on it. If that is also wrong, you will
need to correct it as well. You can do that manually, or you can use a patched
e2fsprogs version, that will do that for you
Patches are here:
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://bugzilla.lustre.org/show_bug.cgi?id=22734">https://bugzilla.lustre.org/show_bug.cgi?id=22734</a>
Packages can be found on my home page:
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/e2fsprogs/">http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/e2fsprogs/</a>
If you want to do it automatically, you will need to create a lfsck mdsdb file
(the hdr file is sufficient, see the lfsck section in the manual) and then you
will need to run e2fsck for that OST as if you want to create an OSTDB file.
That will start pass6, and if you then run e2fsck *without* "-n", so in
correcting mode, it will correct the LAST_ID file to what it finds on disk.
With "-v" it will also tell you the old and the new value and then you will
need to put that value properly coded into the MDS lov_objid file.
Be careful and create backups of the lov_objid and LAST_ID files.
Hope it helps,
Bern
</pre>
</blockquote>
<pre wrap="">
<hr size="4" width="90%">
_______________________________________________
Lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a>
</pre>
</blockquote>
</body>
</html>