<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
Wojciech,<br>
since you have successfully done step #4 can you tell me what is
use in the reformat for the old index id? <br>
I tried to do this a few weeks ago was not succsessful at
reformatting an ost with the old index because<br>
I am not clear on what the index is. I asked on this list at that
time for input and did not get much. <br>
If you could provide the exact command you used that would be good
too. <br>
<br>
lisa<br>
<br>
<br>
On 10/26/10 10:31 AM, Wojciech Turek wrote:
<blockquote
cite="mid:AANLkTinjXFRSUD4uqPWMBxddfwkpVVJc30QyfXtnkmcj@mail.gmail.com"
type="cite">Since some of our users started to recover their data
from backups or by other means (rerunning jobs etc) into the
original locations I don't think it would be good idea to put the
recovered OST back in service as it is, as that may cause some of
users new files to be overwritten by the recovered files. <br>
<br>
To avoid that scenario I decided to reformat the old OST and put
it back into filesystem as empty.<br>
1) First I have created a backup of the recovered object files<br>
2) then using lfs find and lfs getstripe on the client I created
a list of files and object ids from the formatted OST <br>
3) using backup from point 1 and information from point 2 I copied
objects to a new location on the filesystem and renamed them to
their original name. Now users can interrogate those files and
choose which they want to keep.<br>
4) I reformatted old OST with old index id and old label<br>
</blockquote>
<br>
<blockquote
cite="mid:AANLkTinjXFRSUD4uqPWMBxddfwkpVVJc30QyfXtnkmcj@mail.gmail.com"
type="cite"><br>
Before I mount that OST into filesystem I want to make sure that
MDS detects it as empty OST and does not try to recreate missing
objects. Would it be enough to remove lov_objid from MDT and let
it create new lov_objid based on information from OSTs, or do I
need to first unlink all missing files from the client?<br>
<br>
Best regards,<br>
<br>
Wojciech<br>
<br>
<div class="gmail_quote">On 26 October 2010 05:36, Wojciech Turek
<span dir="ltr"><<a moz-do-not-send="true"
href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
Bernd, I would like to clarify if I understood you suggestion
correctly:<br>
<br>
1) create a new OST but using old index and old label<br>
2) mount it as ldiskfs and copy recovered objects (using tar
or rsync with xattrs support) from the old OST to the new OST<br>
3) run --writeconf on MDT and OST of that filesystem<br>
4) mount MDT and all OSTs<br>
<br>
<br>
I guess I could do it also that way:<br>
<br>
1) backup restored object using tar or rsync with xattrs
support<br>
2) format old OST with old index and old label<br>
3) restore Objects from the backup<br>
<br>
Do you think that would work?<br>
<br>
Best regards,<br>
<font color="#888888"><br>
Wojciech</font>
<div>
<div class="h5"><br>
<br>
<br>
<div class="gmail_quote">On 22 October 2010 18:52, Bernd
Schubert <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:bernd.schubert@fastmail.fm"
target="_blank">bernd.schubert@fastmail.fm</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt
0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">Hmm, I would probably format a
small fake device on a ramdisk and copy files<br>
over, run tunefs --writeconf /mdt and then start
everything (inlcuding all<br>
OSTs) again.<br>
<br>
<br>
Cheers,<br>
<div>
<div><br>
On Friday, October 22, 2010, Wojciech Turek wrote:<br>
> I have tried Bernd's suggestion and it seem
to have worked, after running<br>
> e2fsck -D ll_recover_lost_found_objs didn't
cause kernel panic but moved a<br>
> number of objects to O directory. Problem is
that I do not have last_rcvd<br>
> file so the OST has no index at the moment.
What would be the next step to<br>
> enable access to those files in the
filesystem?<br>
><br>
> Best regards,<br>
><br>
> Wojciech<br>
><br>
> On 22 October 2010 17:15, Andreas Dilger <<a
moz-do-not-send="true"
href="mailto:andreas.dilger@oracle.com"
target="_blank">andreas.dilger@oracle.com</a>>
wrote:<br>
> > On 2010-10-22, at 5:42, Bernd Schubert
<<a moz-do-not-send="true"
href="mailto:bernd.schubert@fastmail.fm"
target="_blank">bernd.schubert@fastmail.fm</a>>
wrote:<br>
> > > Hmm, e2fsck didn't catch that?
rec_len is the length of a directory<br>
> ><br>
> > entry, so<br>
> ><br>
> > > after how many bytes the next entry
follows.<br>
> ><br>
> > I agree that e2fsck should have caught
that.<br>
> ><br>
> > > You can try to force e2fsck to do<br>
> > > something about that: e2fsck -D<br>
> ><br>
> > No, I would recommend against using -D
at this point. That will cause it<br>
> > to re-write the directory contents, and
given that the filesystem was<br>
> > previously corrupted I would prefer
making as few changes as possible<br>
> > before the data is estranged.<br>
> ><br>
> > Wojciech,<br>
> > note that if you are able to mount the
filesystem you could just copy all<br>
> > of the objects (with xattrs!) from
lost+found on the bad filesystem,<br>
> > along with the last_rcvd file (if you
can find it) into a new ldiskfs<br>
> > filesystem and then run
ll_recover_lost_found_objs on that.<br>
> ><br>
> > > On Friday, October 22, 2010,
Wojciech Turek wrote:<br>
> > >> Ok, removing and recreating the
journal fixed that problem and I am<br>
> > >> able<br>
> ><br>
> > to<br>
> ><br>
> > >> mount device as ldiskfs
filesystem. Now I hit another wall when trying<br>
> ><br>
> > to<br>
> ><br>
> > >> run ll_recover_lost_found_objs<br>
> > >> When I first time run
ll_recover_lost_found_objs -d<br>
> > >> /mnt/ost/lost+found<br>
> ><br>
> > it<br>
> ><br>
> > >> only creates the O dir and
exits. When I repeat this command again<br>
> ><br>
> > kernel<br>
> ><br>
> > >> panics. Any idea what could be
the problem here?<br>
> > >><br>
> > >><br>
> > >> LDISKFS-fs error (device dm-4):
ldiskfs_readdir: bad entry in<br>
> > >> directory #6831: rec_len is
smaller than minimal - offset=0, inode=0,<br>
> > >> rec_len=0, name_len=0<br>
> > >> Aborting journal on device
dm-4.<br>
> > >> Unable to handle kernel NULL
pointer dereference at 0000000000000000<br>
> ><br>
> > RIP:<br>
> > >> [<ffffffff88033448>]
:jbd:journal_commit_transaction+0xc5b/0x12db<br>
> > >> PGD 1a118d067 PUD 1ce7e7067 PMD
0<br>
> > >> Oops: 0002 [1] SMP<br>
> > >> last sysfs file:
/class/infiniband_mad/umad0/port<br>
> > >> CPU 3<br>
> > >> Modules linked in: ldiskfs(U)
crc16(U) autofs4(U) hidp(U) l2cap(U)<br>
> > >> bluetooth(U) rdma_ucm(U)
rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U)<br>
> > >> ipoib_helper(U) ib_cm(U)
ipv6(U) xfrm_nalgo(U) crypto_api(U)<br>
> ><br>
> > ib_uverbs(U)<br>
> ><br>
> > >> ib_umad(U) mlx4_vnic(U)
mlx4_vnic_helper(U) ib_sa(U) ib_mthca(U)<br>
> ><br>
> > mptctl(U)<br>
> ><br>
> > >> dm_mirror(U) video(U)
backlight(U) sbs(U) power_meter(U) hwmon(U)<br>
> ><br>
> > i2c_ec(U)<br>
> ><br>
> > >> i2c_core(U) dell_wmi(U) wmi(U)
button(U) battery(U) asus_acpi(U)<br>
> > >> acpi_memhotplug(U) ac(U)
parport_pc(U) lp(U) parport(U) sr_mod(U)<br>
> ><br>
> > cdrom(U)<br>
> ><br>
> > >> mlx4_ib(U) ib_mad(U) ib_core(U)
joydev(U) mlx4_core(U) usb_storage(U)<br>
> > >> pcspkr(U) shpchp(U)
serio_raw(U) i5000_edac(U) edac_mc(U) dm_raid45(U)<br>
> > >> dm_message(U) dm_region_hash(U)
dm_log(U) dm_mod(U) dm_mem_cache(U)<br>
> ><br>
> > nfs(U)<br>
> ><br>
> > >> lockd(U) fscache(U) nfs_acl(U)
sunrpc(U) mptsas(U) mptscsih(U)<br>
> ><br>
> > mptbase(U)<br>
> ><br>
> > >> scsi_transport_sas(U)
mppVhba(U) megaraid_sas(U) mppUpper(U) sg(U)<br>
> > >> sd_mod(U) scsi_mod(U) bnx2(U)
ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U)<br>
> > >> ehci_hcd(U) Pid: 11360, comm:
kjournald Tainted: G<br>
> > >> 2.6.18-194.3.1.el5_lustre.1.8.4
#1<br>
> > >> RIP:
0010:[<ffffffff88033448>]
[<ffffffff88033448>]<br>
> > >><br>
> > >>
:jbd:journal_commit_transaction+0xc5b/0x12db<br>
> > >><br>
> > >> RSP: 0018:ffff8101c6481d90
EFLAGS: 00010246<br>
> > >> RAX: 0000000000000000 RBX:
0000000000000000 RCX: 00000000ffffffff<br>
> > >> RDX: 0000000000000000 RSI:
ffff8101e9dab0c0 RDI: ffff81022fa46000<br>
> > >> RBP: ffff81022fa46000 R08:
ffff81022fa46068 R09: 0000000000000000<br>
> > >> R10: ffff810105925b20 R11:
00000000fffffffa R12: 0000000000000000<br>
> > >> R13: 0000000000000000 R14:
ffff8101e9dab0c0 R15: 0000000000000000<br>
> > >> FS: 0000000000000000(0000)
GS:ffff810107b9a4c0(0000)<br>
> > >> knlGS:0000000000000000 CS:
0010 DS: 0018 ES: 0018 CR0:<br>
> > >> 000000008005003b CR2:
0000000000000000 CR3: 00000001eaffb000 CR4:<br>
> > >> 00000000000006e0 Process
kjournald (pid: 11360, threadinfo<br>
> > >> ffff8101c6480000, task
ffff81021c14c0c0)<br>
> > >> Stack: ffff8101a61b9000
000000002b8263c0 ffffffff00000000<br>
> ><br>
> > 0000000000000000<br>
> ><br>
> > >> 0000113b00000001
0000000000000013 0000000000000000 0000000000000111<br>
> > >> 0000000000000000
0000000000000000 0000000001282dd7 00000000000020dd<br>
> > >> Call Trace:<br>
> > >> [<ffffffff8003da91>]
lock_timer_base+0x1b/0x3c<br>
> > >> [<ffffffff8004b347>]
try_to_del_timer_sync+0x7f/0x88<br>
> > >> [<ffffffff88037386>]
:jbd:kjournald+0xc1/0x213<br>
> > >> [<ffffffff800a0ab2>]
autoremove_wake_function+0x0/0x2e<br>
> > >> [<ffffffff800a089a>]
keventd_create_kthread+0x0/0xc4<br>
> > >> [<ffffffff880372c5>]
:jbd:kjournald+0x0/0x213<br>
> > >> [<ffffffff800a089a>]
keventd_create_kthread+0x0/0xc4<br>
> > >> [<ffffffff80032890>]
kthread+0xfe/0x132<br>
> > >> [<ffffffff8005dfb1>]
child_rip+0xa/0x11<br>
> > >> [<ffffffff800a089a>]
keventd_create_kthread+0x0/0xc4<br>
> > >> [<ffffffff8014bcf4>]
deadline_queue_empty+0x0/0x23<br>
> > >> [<ffffffff80032792>]
kthread+0x0/0x132<br>
> > >> [<ffffffff8005dfa7>]
child_rip+0x0/0x11<br>
> > >><br>
> > >><br>
> > >> Code: f0 0f ba 33 01 e8 42 fc
02 f8 8b 03 a8 04 75 07 8b 43 58 85<br>
> > >> RIP [<ffffffff88033448>]
:jbd:journal_commit_transaction+0xc5b/0x12db<br>
> > >> RSP <ffff8101c6481d90><br>
> > >> CR2: 0000000000000000<br>
> > >> <0>Kernel panic - not
syncing: Fatal exception<br>
> > >><br>
> > >> On 22 October 2010 03:09,
Andreas Dilger <<a moz-do-not-send="true"
href="mailto:andreas.dilger@oracle.com"
target="_blank">andreas.dilger@oracle.com</a>><br>
> ><br>
> > wrote:<br>
> > >>> On 2010-10-21, at 18:44,
Wojciech Turek <<a moz-do-not-send="true"
href="mailto:wjt27@cam.ac.uk" target="_blank">wjt27@cam.ac.uk</a>>
wrote:<br>
> > >>><br>
> > >>> fsck has finished and does
not find any more errors to correct.<br>
> > >>> However when I try to mount
the device as ldiskfs kernel panics with<br>
> > >>> following message:<br>
> > >>><br>
> > >>> Assertion failure in
cleanup_journal_tail() at<br>
> > >>> fs/jbd/checkpoint.c:459:
"blocknr != 0"<br>
> > >>><br>
> > >>><br>
> > >>> Hmm, not sure, maybe your
journal is broken? You can delete it with<br>
> > >>> "tune2fs -O ^has_journal"
(maybe after running e2fsck again to clear<br>
> ><br>
> > the<br>
> ><br>
> > >>> journal), then re-create it
with "tune2fs -j".<br>
> > >>><br>
> > >>> ----------- [cut here ]
--------- [please bite here ] ---------<br>
> > >>> Kernel BUG at
fs/jbd/checkpoint.c:459<br>
> > >>> invalid opcode: 0000 [1]
SMP<br>
> > >>> last sysfs file:
/class/infiniband_mad/umad0/<br>
> > >>> port<br>
> > >>> CPU 2<br>
> > >>> Modules linked in:
obdfilter(U) fsfilt_ldiskfs(U) ost(U) mgc(U)<br>
> > >>> ldiskfs(U) crc16(U)
lustre(U) lov(U) mdc(U) lquota(U) osc(U)<br>
> ><br>
> > ksocklnd(U)<br>
> ><br>
> > >>> ko2iblnd(U) ptlrpc(U)
obdclass(U) lnet(U) lvfs(U) libcfs(U)<br>
> > >>> autofs4(U) hidp(U) l2cap(U)
bluetooth(U) rdma_ucm(U) rdma_cm(U)<br>
> > >>> iw_cm(U)<br>
> ><br>
> > ib_addr(U)<br>
> ><br>
> > >>> ib_ipoib(U) ipoib_helper(U)
ib_cm(U) ipv6(U) xfrm_nalgo(U)<br>
> ><br>
> > crypto_api(U)<br>
> ><br>
> > >>> ib_uverbs(U) ib_umad(U)
mlx4_vnic(U) mlx4_vnic_helper(U) ib_sa(U)<br>
> > >>> ib_mthca(U) mptctl(U)
dm_mirror(U) video(U) backlight(U) sbs(U)<br>
> > >>> power_meter(U) hwmon(U)
i2c_ec(U) i2c_core(U) dell_wmi(U) wmi(U)<br>
> > >>> button(U) battery(U)
asus_acpi(U) acpi_memhotplug(U) ac(U)<br>
> ><br>
> > parport_pc(U)<br>
> ><br>
> > >>> lp(U) parport(U) sr_mod(U)
cdrom(U) mlx4_ib(U) ib_mad(U) ib_core(U)<br>
> > >>> joydev(U) mlx4_core(U)
usb_storage(U) shpchp(U) i5000_edac(U)<br>
> ><br>
> > edac_mc(U)<br>
> ><br>
> > >>> serio_raw(U) pcspkr(U)
dm_raid45(U) dm_message(U) dm_region_hash(U)<br>
> > >>> dm_log(U) dm_mod(U)
dm_mem_cache(U) nfs(U) lockd(U) fscache(U)<br>
> > >>> nfs_acl(U) sunrpc(U)
mptsas(U) mptscsih(U) mptbase(U)<br>
> > >>> scsi_transport_sas(U)
mppVhba(U) megaraid_sas(U) mppUpper(U) sg(U)<br>
> > >>> sd_mod(U) scsi_mod(U)
bnx2(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U)<br>
> > >>> ehci_hcd(U) Pid: 13891,
comm: mount Tainted: G<br>
> > >>>
2.6.18-194.3.1.el5_lustre.1.8.4 #1 RIP:
0010:[<ffffffff88034a95>]<br>
> > >>> [<ffffffff88034a95>]<br>
> > >>><br>
> > >>>
:jbd:cleanup_journal_tail+0x9d/0x118<br>
> > >>><br>
> > >>> RSP: 0018:ffff81016f00da68
EFLAGS: 00010286<br>
> > >>> RAX: 000000000000005a RBX:
ffff81012ca12c00 RCX: ffffffff80311da8<br>
> > >>> RDX: ffffffff80311da8 RSI:
0000000000000000 RDI: ffffffff80311da0<br>
> > >>> RBP: 0000000000000000 R08:
ffffffff80311da8 R09: 0000000000000001<br>
> > >>> R10: 0000000000000000 R11:
0000000000000080 R12: 0000000000000002<br>
> > >>> R13: ffff81012ca12d4c R14:
ffff81012ca12c24 R15: ffff81017a8d7400<br>
> > >>> FS: 00002abd7cef1f70(0000)
GS:ffff810107b9acc0(0000)<br>
> > >>> knlGS:0000000000000000<br>
> > >>> CS: 0010 DS: 0000 ES: 0000
CR0: 000000008005003b<br>
> > >>> CR2: 000000000042b000 CR3:
000000012813f000 CR4: 00000000000006e0<br>
> > >>> Process mount (pid: 13891,
threadinfo ffff81016f00c000, task<br>
> > >>> ffff81022e1b7820)<br>
> > >>> Stack: 0000000000000000
ffff81012ca12c00 ffff81017a8d7400<br>
> > >>> ffffffff88037690<br>
> > >>><br>
> > >>> ffff81012ca12c00
ffff8102034ff000 ffff81017a8d7400 0000000000000000<br>
> > >>> ffff8102034ff000
ffffffff88a9be56 0000000001000000 ffff8101bf788000<br>
> > >>><br>
> > >>> Call Trace:<br>
> > >>> [<ffffffff88037690>]
:jbd:journal_flush+0xbe/0x248<br>
> > >>> [<ffffffff88a9be56>]<br>
> > >>>
:ldiskfs:ldiskfs_mark_recovery_complete+0x36/0x90<br>
> > >>> [<ffffffff88aa02e0>]
:ldiskfs:ldiskfs_fill_super+0x1790/0x1950<br>
> > >>> [<ffffffff800eccd2>]
get_filesystem+0x12/0x3b<br>
> > >>> [<ffffffff800e343e>]
test_bdev_super+0x0/0xd<br>
> > >>> [<ffffffff88a9eb50>]
:ldiskfs:ldiskfs_fill_super+0x0/0x1950<br>
> > >>> [<ffffffff800e43fd>]
get_sb_bdev+0x10a/0x16c<br>
> > >>> [<ffffffff800e3d9a>]
vfs_kern_mount+0x93/0x11a<br>
> > >>> [<ffffffff800e3e63>]
do_kern_mount+0x36/0x4d<br>
> > >>> [<ffffffff800ee601>]
do_mount+0x6a9/0x719<br>
> > >>> [<ffffffff800090d2>]
__handle_mm_fault+0x96f/0xfaa<br>
> > >>> [<ffffffff8002c9e0>]
mntput_no_expire+0x19/0x89<br>
> > >>> [<ffffffff8000a72a>]
__link_path_walk+0xf1e/0xf42<br>
> > >>> [<ffffffff800220ce>]
__up_read+0x19/0x7f<br>
> > >>> [<ffffffff80066b88>]
do_page_fault+0x4fe/0x874<br>
> > >>> [<ffffffff8002c9e0>]
mntput_no_expire+0x19/0x89<br>
> > >>> [<ffffffff8000ea45>]
link_path_walk+0xa6/0xb2<br>
> > >>> [<ffffffff800cc329>]
zone_statistics+0x3e/0x6d<br>
> > >>> [<ffffffff8000f2cf>]
__alloc_pages+0x78/0x308<br>
> > >>> [<ffffffff8004c68e>]
sys_mount+0x8a/0xcd<br>
> > >>> [<ffffffff8005d28d>]
tracesys+0xd5/0xe0<br>
> > >>><br>
> > >>> Code: 0f 0b 68 3a 94 03 88
c2 cb 01 44 39 a3 58 01 00 00 75 0e c7<br>
> > >>> RIP
[<ffffffff88034a95>]
:jbd:cleanup_journal_tail+0x9d/0x118<br>
> > >>><br>
> > >>> RSP
<ffff81016f00da68><br>
> > >>> <0>Kernel panic - not
syncing: Fatal exception<br>
> > >>><br>
> > >>> Any idea how to fix this?<br>
> > >>><br>
> > >>> Many thanks<br>
> > >>><br>
> > >>> Wojciech<br>
> > >>><br>
> > >>><br>
> > >>> On 21 October 2010 17:54,
Wojciech Turek < <<a moz-do-not-send="true"
href="mailto:wjt27@cam.ac.uk" target="_blank">wjt27@cam.ac.uk</a>><br>
> > >>><br>
> > >>> <a moz-do-not-send="true"
href="mailto:wjt27@cam.ac.uk" target="_blank">wjt27@cam.ac.uk</a>>
wrote:<br>
> > >>>> Thanks Ken, that
worked.<br>
> > >>>><br>
> > >>>><br>
> > >>>> On 21 October 2010
17:39, Ken Hornstein < <<a
moz-do-not-send="true"
href="mailto:kenh@cmf.nrl.navy.mil"
target="_blank">kenh@cmf.nrl.navy.mil</a>><br>
> > >>>><br>
> > >>>> <a
moz-do-not-send="true"
href="mailto:kenh@cmf.nrl.navy.mil"
target="_blank">kenh@cmf.nrl.navy.mil</a>>
wrote:<br>
> > >>>>>> Now I have
another problem. After last segfault I can not
restart<br>
> ><br>
> > the<br>
> ><br>
> > >>>>> fsck<br>
> > >>>>><br>
> > >>>>>> due to MMP.<br>
> > >>>>>> [...]<br>
> > >>>>>> Also when I try
to access filesystem via debugfs it fails:<br>
> > >>>>>><br>
> > >>>>>> debugfs -c -R
'ls' /dev/scratch2_ost16vg/ost16lv<br>
> > >>>>>> debugfs
1.41.10.sun2 (24-Feb-2010)<br>
> > >>>>>>
/dev/scratch2_ost16vg/ost16lv: MMP: fsck being run
while opening<br>
> > >>>>><br>
> > >>>>> filesystem<br>
> > >>>>><br>
> > >>>>>> ls: Filesystem
not open<br>
> > >>>>>><br>
> > >>>>>> Is there a way
to clear teh MMP flag so it allows fsck to run?<br>
> > >>>>><br>
> > >>>>> You want tune2fs -f
-E clear-mmp<br>
> > >>>>><br>
> > >>>>> --Ken<br>
<br>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
Lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a>
</pre>
</blockquote>
<br>
</body>
</html>