[Lustre-discuss] recovering formatted OST

Wojciech Turek wjt27 at cam.ac.uk
Thu Oct 21 17:44:26 PDT 2010


Hi,

fsck has finished and does not find any more errors to correct. However when
I try to mount the device as ldiskfs kernel panics with following message:

Assertion failure in cleanup_journal_tail() at fs/jbd/checkpoint.c:459:
"blocknr != 0"
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at fs/jbd/checkpoint.c:459
invalid opcode: 0000 [1] SMP
last sysfs file: /class/infiniband_mad/umad0/
port
CPU 2
Modules linked in: obdfilter(U) fsfilt_ldiskfs(U) ost(U) mgc(U) ldiskfs(U)
crc16(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ksocklnd(U) ko2iblnd(U)
ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) autofs4(U) hidp(U) l2cap(U)
bluetooth(U) rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U)
ipoib_helper(U) ib_cm(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) ib_uverbs(U)
ib_umad(U) mlx4_vnic(U) mlx4_vnic_helper(U) ib_sa(U) ib_mthca(U) mptctl(U)
dm_mirror(U) video(U) backlight(U) sbs(U) power_meter(U) hwmon(U) i2c_ec(U)
i2c_core(U) dell_wmi(U) wmi(U) button(U) battery(U) asus_acpi(U)
acpi_memhotplug(U) ac(U) parport_pc(U) lp(U) parport(U) sr_mod(U) cdrom(U)
mlx4_ib(U) ib_mad(U) ib_core(U) joydev(U) mlx4_core(U) usb_storage(U)
shpchp(U) i5000_edac(U) edac_mc(U) serio_raw(U) pcspkr(U) dm_raid45(U)
dm_message(U) dm_region_hash(U) dm_log(U) dm_mod(U) dm_mem_cache(U) nfs(U)
lockd(U) fscache(U) nfs_acl(U) sunrpc(U) mptsas(U) mptscsih(U) mptbase(U)
scsi_transport_sas(U) mppVhba(U) megaraid_sas(U) mppUpper(U) sg(U) sd_mod(U)
scsi_mod(U) bnx2(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U)
Pid: 13891, comm: mount Tainted: G      2.6.18-194.3.1.el5_lustre.1.8.4 #1
RIP: 0010:[<ffffffff88034a95>]  [<ffffffff88034a95>]
:jbd:cleanup_journal_tail+0x9d/0x118
RSP: 0018:ffff81016f00da68  EFLAGS: 00010286
RAX: 000000000000005a RBX: ffff81012ca12c00 RCX: ffffffff80311da8
RDX: ffffffff80311da8 RSI: 0000000000000000 RDI: ffffffff80311da0
RBP: 0000000000000000 R08: ffffffff80311da8 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000080 R12: 0000000000000002
R13: ffff81012ca12d4c R14: ffff81012ca12c24 R15: ffff81017a8d7400
FS:  00002abd7cef1f70(0000) GS:ffff810107b9acc0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000042b000 CR3: 000000012813f000 CR4: 00000000000006e0
Process mount (pid: 13891, threadinfo ffff81016f00c000, task
ffff81022e1b7820)
Stack:  0000000000000000 ffff81012ca12c00 ffff81017a8d7400 ffffffff88037690
 ffff81012ca12c00 ffff8102034ff000 ffff81017a8d7400 0000000000000000
 ffff8102034ff000 ffffffff88a9be56 0000000001000000 ffff8101bf788000
Call Trace:
 [<ffffffff88037690>] :jbd:journal_flush+0xbe/0x248
 [<ffffffff88a9be56>] :ldiskfs:ldiskfs_mark_recovery_complete+0x36/0x90
 [<ffffffff88aa02e0>] :ldiskfs:ldiskfs_fill_super+0x1790/0x1950
 [<ffffffff800eccd2>] get_filesystem+0x12/0x3b
 [<ffffffff800e343e>] test_bdev_super+0x0/0xd
 [<ffffffff88a9eb50>] :ldiskfs:ldiskfs_fill_super+0x0/0x1950
 [<ffffffff800e43fd>] get_sb_bdev+0x10a/0x16c
 [<ffffffff800e3d9a>] vfs_kern_mount+0x93/0x11a
 [<ffffffff800e3e63>] do_kern_mount+0x36/0x4d
 [<ffffffff800ee601>] do_mount+0x6a9/0x719
 [<ffffffff800090d2>] __handle_mm_fault+0x96f/0xfaa
 [<ffffffff8002c9e0>] mntput_no_expire+0x19/0x89
 [<ffffffff8000a72a>] __link_path_walk+0xf1e/0xf42
 [<ffffffff800220ce>] __up_read+0x19/0x7f
 [<ffffffff80066b88>] do_page_fault+0x4fe/0x874
 [<ffffffff8002c9e0>] mntput_no_expire+0x19/0x89
 [<ffffffff8000ea45>] link_path_walk+0xa6/0xb2
 [<ffffffff800cc329>] zone_statistics+0x3e/0x6d
 [<ffffffff8000f2cf>] __alloc_pages+0x78/0x308
 [<ffffffff8004c68e>] sys_mount+0x8a/0xcd
 [<ffffffff8005d28d>] tracesys+0xd5/0xe0


Code: 0f 0b 68 3a 94 03 88 c2 cb 01 44 39 a3 58 01 00 00 75 0e c7
RIP  [<ffffffff88034a95>] :jbd:cleanup_journal_tail+0x9d/0x118
 RSP <ffff81016f00da68>
 <0>Kernel panic - not syncing: Fatal exception

Any idea how to fix this?

Many thanks

Wojciech


On 21 October 2010 17:54, Wojciech Turek <wjt27 at cam.ac.uk> wrote:

> Thanks Ken, that worked.
>
>
> On 21 October 2010 17:39, Ken Hornstein <kenh at cmf.nrl.navy.mil> wrote:
>
>> >Now I have another problem. After last segfault I can not restart the
>> fsck
>> >due to MMP.
>> >[...]
>> >Also when I try to access filesystem via debugfs it fails:
>> >
>> >debugfs -c -R 'ls' /dev/scratch2_ost16vg/ost16lv
>> >debugfs 1.41.10.sun2 (24-Feb-2010)
>> >/dev/scratch2_ost16vg/ost16lv: MMP: fsck being run while opening
>> filesystem
>> >ls: Filesystem not open
>> >
>> >Is there a way to clear teh MMP flag so it allows fsck to run?
>>
>> You want tune2fs -f -E clear-mmp
>>
>> --Ken
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20101022/61489db1/attachment.htm>


More information about the lustre-discuss mailing list