In that case LAST_ID seem to be fine as OST show 2490599 and MDT shows 2490688 so the difference is 89, I don't understand why you said that difference is over 100000<br><br>[root@oss09 ~]# od -Ax -td8 /tmp/LAST_ID<br>
000000 2490599<br>
000008<br><br>[root@mds03 ~]# od -Ax -td8 /tmp/lov_objid<br>
000000 2073842 2100049<br>
000010 2115247 2038471<br>
000020 2119821 2190996<br>
000030 2029234 2354424<br>
000040 2160856 2167105<br>
000050 1970351 2059045<br>
000060 2706486 2571655<br>
000070 2662262 2628346<br>
000080 2490688 2668926<br>
000090 2631587 2643791<br>
0000a0<br><br>What I don't understand is why lctl reports last_id=1 for that OST <br><br>lctl get_param osc.*.prealloc_last_id | grep OST0010<br>osc.scratch2-OST0010-osc.prealloc_last_id=1<br><br><div class="gmail_quote">
On 26 October 2010 19:49, Bernd Schubert <span dir="ltr"><<a href="mailto:bs_lists@aakef.fastmail.fm">bs_lists@aakef.fastmail.fm</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
That is the value in the lov_objid.<br>
<div><div></div><div class="h5"><br>
Cheers,<br>
Bernd<br>
<br>
On Tuesday, October 26, 2010, Wojciech Turek wrote:<br>
> I can not find where MDT stores that LAST_ID value for the OST?<br>
><br>
> On 26 October 2010 19:10, Bernd Schubert <<a href="mailto:bs_lists@aakef.fastmail.fm">bs_lists@aakef.fastmail.fm</a>> wrote:<br>
> > I think the difference is quite huge (over 100000 files). But the MDS has<br>
> > a sanity check and will refuse to activate this OST, if the difference<br>
> > is larger<br>
> > than 20000 files.<br>
> ><br>
> > So one way or the other you need to correct it (either increase LAST_ID<br>
> > value<br>
> > on the OST or on the MDS).<br>
> ><br>
> ><br>
> > Cheers,<br>
> > Bernd<br>
> ><br>
> > On Tuesday, October 26, 2010, Wojciech Turek wrote:<br>
> > > Ok, I have created a filesystem on a loopback device. I mounted it as<br>
> > > ldiskfs and copied CONFIGS directory back to my old OST. Now<br>
> ><br>
> > tunefs.lustre<br>
> ><br>
> > > returns correct info.<br>
> > ><br>
> > > last_id on OST is smaller then number in MDT lov_objid which is good<br>
> > ><br>
> > > Can ignore that lctl get_param osc.*.prealloc_last_id | grep OST0010<br>
> > > osc.scratch2-OST0010-osc.prealloc_last_id=1<br>
> > ><br>
> > > I guess when I restart whole filesystem after writeconf MDT should<br>
> ><br>
> > correct<br>
> ><br>
> > > that?<br>
> > ><br>
> > > best regards,<br>
> > ><br>
> > > Wojciech<br>
> > ><br>
> > > On 26 October 2010 18:05, Bernd Schubert <<a href="mailto:bs_lists@aakef.fastmail.fm">bs_lists@aakef.fastmail.fm</a>><br>
> ><br>
> > wrote:<br>
> > > > Hello Wojciech,<br>
> > > ><br>
> > > > tunefs.lustre has to complain as the files are missing. If you copy<br>
> ><br>
> > over<br>
> ><br>
> > > > the<br>
> > > > files from the loop back device (yes, same index and label),<br>
> > > > tunefs.lustre should work.<br>
> > > ><br>
> > > > Cheers,<br>
> > > > Bernd<br>
> > > ><br>
> > > > On Tuesday, October 26, 2010, Wojciech Turek wrote:<br>
> > > > > Hi Bernd,<br>
> > > > ><br>
> > > > > I am not quite clear how creating new OST on a loopback device<br>
> > > > > would<br>
> > > ><br>
> > > > help:<br>
> > > > > Shall I create new OST on a loopback device formatting it with old<br>
> > > > > index and label and then copy recovered objects to that OST and<br>
> > > > > mount it to the filesystem?<br>
> > > > ><br>
> > > > > I think I need to reformat old OST before mounting it as lustre<br>
> > > > > type filesystem as although fsck recovered some objects (and I can<br>
> > > > > access them mounting OST as ldiskfs) if you run tunefs.lustre on<br>
> > > > > that OST device, tunefs.lustre complaints that it doesn't find any<br>
> > > > > lustre filesystem.<br>
> > > > ><br>
> > > > > As for the EAs I have created a backup of the recovered objects<br>
> > > ><br>
> > > > preserving<br>
> > > ><br>
> > > > > EAs.<br>
> > > > ><br>
> > > > > Best regards,<br>
> > > > ><br>
> > > > > Wojciech<br>
> > > > ><br>
> > > > > On 26 October 2010 16:35, Bernd Schubert<br>
> > > > > <<a href="mailto:bernd.schubert@fastmail.fm">bernd.schubert@fastmail.fm</a><br>
> > > ><br>
> > > > wrote:<br>
> > > > > > Hello Wojciech,<br>
> > > > > ><br>
> > > > > > I think both would work, but why don't just create a small OST<br>
> > > > > > with mkfs.lustre on a loopback device? And then copy over those<br>
> > > > > > files to<br>
> > > ><br>
> > > > your<br>
> > > ><br>
> > > > > > recovered filesystem.<br>
> > > > > > Hmm, well, e2fsck might not have fixed all issues and then a<br>
> ><br>
> > reformat<br>
> ><br>
> > > > > > indeed<br>
> > > > > > might be helpful.<br>
> > > > > ><br>
> > > > > > Also note: EAs on OST objects are a nice to have, but not<br>
> ><br>
> > absolutely<br>
> ><br>
> > > > > > required.<br>
> > > > > ><br>
> > > > > > Cheers,<br>
> > > > > > Bernd<br>
> > > > > ><br>
> > > > > > On Tuesday, October 26, 2010, Wojciech Turek wrote:<br>
> > > > > > > Bernd, I would like to clarify if I understood you suggestion<br>
> > > > > > > correctly:<br>
> > > > > > ><br>
> > > > > > > 1) create a new OST but using old index and old label<br>
> > > > > > > 2) mount it as ldiskfs and copy recovered objects (using tar or<br>
> > > > > > > rsync<br>
> > > > > ><br>
> > > > > > with<br>
> > > > > ><br>
> > > > > > > xattrs support) from the old OST to the new OST<br>
> > > > > > > 3) run --writeconf on MDT and OST of that filesystem<br>
> > > > > > > 4) mount MDT and all OSTs<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > I guess I could do it also that way:<br>
> > > > > > ><br>
> > > > > > > 1) backup restored object using tar or rsync with xattrs<br>
> > > > > > > support 2) format old OST with old index and old label<br>
> > > > > > > 3) restore Objects from the backup<br>
> > > > > > ><br>
> > > > > > > Do you think that would work?<br>
> > > > > > ><br>
> > > > > > > Best regards,<br>
> > > > > > ><br>
> > > > > > > Wojciech<br>
> > > > > > ><br>
> > > > > > > On 22 October 2010 18:52, Bernd Schubert<br>
> > > > > > > <<a href="mailto:bernd.schubert@fastmail.fm">bernd.schubert@fastmail.fm</a><br>
> > > > > ><br>
> > > > > > wrote:<br>
> > > > > > > > Hmm, I would probably format a small fake device on a ramdisk<br>
> ><br>
> > and<br>
> ><br>
> > > > > > > > copy files<br>
> > > > > > > > over, run tunefs --writeconf /mdt and then start everything<br>
> > > > > > > > (inlcuding all OSTs) again.<br>
> > > > > > > ><br>
> > > > > > > ><br>
> > > > > > > > Cheers,<br>
> > > > > > > ><br>
> > > > > > > > On Friday, October 22, 2010, Wojciech Turek wrote:<br>
> > > > > > > > > I have tried Bernd's suggestion and it seem to have worked,<br>
> > > > > > > > > after running e2fsck -D ll_recover_lost_found_objs didn't<br>
> > > > > > > > > cause kernel<br>
> > > > > ><br>
> > > > > > panic<br>
> > > > > ><br>
> > > > > > > > > but moved<br>
> > > > > > > ><br>
> > > > > > > > a<br>
> > > > > > > ><br>
> > > > > > > > > number of objects to O directory. Problem is that I do not<br>
> ><br>
> > have<br>
> ><br>
> > > > > > > > > last_rcvd file so the OST has no index at the moment. What<br>
> > > > > > > > > would<br>
> > > ><br>
> > > > be<br>
> > > ><br>
> > > > > > > > > the next step<br>
> > > > > > > ><br>
> > > > > > > > to<br>
> > > > > > > ><br>
> > > > > > > > > enable access to those files in the filesystem?<br>
> > > > > > > > ><br>
> > > > > > > > > Best regards,<br>
> > > > > > > > ><br>
> > > > > > > > > Wojciech<br>
> > > > > > > > ><br>
> > > > > > > > > On 22 October 2010 17:15, Andreas Dilger<br>
> > > > > > > > > <<a href="mailto:andreas.dilger@oracle.com">andreas.dilger@oracle.com</a>><br>
> > > > > > > ><br>
> > > > > > > > wrote:<br>
> > > > > > > > > > On 2010-10-22, at 5:42, Bernd Schubert<br>
> > > > > > > > > > <<a href="mailto:bernd.schubert@fastmail.fm">bernd.schubert@fastmail.fm</a><br>
> > > > > > > ><br>
> > > > > > > > wrote:<br>
> > > > > > > > > > > Hmm, e2fsck didn't catch that? rec_len is the length of<br>
> > > > > > > > > > > a<br>
> > > > > ><br>
> > > > > > directory<br>
> > > > > ><br>
> > > > > > > > > > entry, so<br>
> > > > > > > > > ><br>
> > > > > > > > > > > after how many bytes the next entry follows.<br>
> > > > > > > > > ><br>
> > > > > > > > > > I agree that e2fsck should have caught that.<br>
> > > > > > > > > ><br>
> > > > > > > > > > > You can try to force e2fsck to do<br>
> > > > > > > > > > > something about that: e2fsck -D<br>
> > > > > > > > > ><br>
> > > > > > > > > > No, I would recommend against using -D at this point.<br>
> > > > > > > > > > That will<br>
> > > > > ><br>
> > > > > > cause<br>
> > > > > ><br>
> > > > > > > > it<br>
> > > > > > > ><br>
> > > > > > > > > > to re-write the directory contents, and given that the<br>
> > > ><br>
> > > > filesystem<br>
> > > ><br>
> > > > > > was<br>
> > > > > ><br>
> > > > > > > > > > previously corrupted I would prefer making as few changes<br>
> ><br>
> > as<br>
> ><br>
> > > > > > possible<br>
> > > > > ><br>
> > > > > > > > > > before the data is estranged.<br>
> > > > > > > > > ><br>
> > > > > > > > > > Wojciech,<br>
> > > > > > > > > > note that if you are able to mount the filesystem you<br>
> > > > > > > > > > could<br>
> > > ><br>
> > > > just<br>
> > > ><br>
> > > > > > copy<br>
> > > > > ><br>
> > > > > > > > all<br>
> > > > > > > ><br>
> > > > > > > > > > of the objects (with xattrs!) from lost+found on the bad<br>
> > > > > ><br>
> > > > > > filesystem,<br>
> > > > > ><br>
> > > > > > > > > > along with the last_rcvd file (if you can find it) into a<br>
> ><br>
> > new<br>
> ><br>
> > > > > > ldiskfs<br>
> > > > > ><br>
> > > > > > > > > > filesystem and then run ll_recover_lost_found_objs on<br>
> > > > > > > > > > that.<br>
> > > > > > > > > ><br>
> > > > > > > > > > > On Friday, October 22, 2010, Wojciech Turek wrote:<br>
> > > > > > > > > > >> Ok, removing and recreating the journal fixed that<br>
> ><br>
> > problem<br>
> ><br>
> > > > and<br>
> > > ><br>
> > > > > > > > > > >> I am able<br>
> > > > > > > > > ><br>
> > > > > > > > > > to<br>
> > > > > > > > > ><br>
> > > > > > > > > > >> mount device as ldiskfs filesystem. Now I hit another<br>
> ><br>
> > wall<br>
> ><br>
> > > > > > > > > > >> when<br>
> > > > > > > ><br>
> > > > > > > > trying<br>
> > > > > > > ><br>
> > > > > > > > > > to<br>
> > > > > > > > > ><br>
> > > > > > > > > > >> run ll_recover_lost_found_objs<br>
> > > > > > > > > > >> When I first time run ll_recover_lost_found_objs -d<br>
> > > > > > > > > > >> /mnt/ost/lost+found<br>
> > > > > > > > > ><br>
> > > > > > > > > > it<br>
> > > > > > > > > ><br>
> > > > > > > > > > >> only creates the O dir and exits. When I repeat this<br>
> > > > > > > > > > >> command<br>
> > > > > ><br>
> > > > > > again<br>
> > > > > ><br>
> > > > > > > > > > kernel<br>
> > > > > > > > > ><br>
> > > > > > > > > > >> panics. Any idea what could be the problem here?<br>
> > > > > > > > > > >><br>
> > > > > > > > > > >><br>
> > > > > > > > > > >> LDISKFS-fs error (device dm-4): ldiskfs_readdir: bad<br>
> ><br>
> > entry<br>
> ><br>
> > > > in<br>
> > > ><br>
> > > > > > > > > > >> directory #6831: rec_len is smaller than minimal -<br>
> > > > > > > > > > >> offset=0,<br>
> > > > > > > ><br>
> > > > > > > > inode=0,<br>
> > > > > > > ><br>
> > > > > > > > > > >> rec_len=0, name_len=0<br>
> > > > > > > > > > >> Aborting journal on device dm-4.<br>
> > > > > > > > > > >> Unable to handle kernel NULL pointer dereference at<br>
> > > > > > > > > > >> 0000000000000000<br>
> > > > > > > > > ><br>
> > > > > > > > > > RIP:<br>
> > > > > > > > > > >> [<ffffffff88033448>]<br>
> > > > > > :<br>
> > > > > > :jbd:journal_commit_transaction+0xc5b/0x12db<br>
> > > > > > :<br>
> > > > > > > > > > >> PGD 1a118d067 PUD 1ce7e7067 PMD 0<br>
> > > > > > > > > > >> Oops: 0002 [1] SMP<br>
> > > > > > > > > > >> last sysfs file: /class/infiniband_mad/umad0/port<br>
> > > > > > > > > > >> CPU 3<br>
> > > > > > > > > > >> Modules linked in: ldiskfs(U) crc16(U) autofs4(U)<br>
> ><br>
> > hidp(U)<br>
> ><br>
> > > > > > l2cap(U)<br>
> > > > > ><br>
> > > > > > > > > > >> bluetooth(U) rdma_ucm(U) rdma_cm(U) iw_cm(U)<br>
> > > > > > > > > > >> ib_addr(U) ib_ipoib(U) ipoib_helper(U) ib_cm(U)<br>
> > > > > > > > > > >> ipv6(U)<br>
> ><br>
> > xfrm_nalgo(U)<br>
> ><br>
> > > > > > > > > > >> crypto_api(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > ib_uverbs(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > >> ib_umad(U) mlx4_vnic(U) mlx4_vnic_helper(U) ib_sa(U)<br>
> > > > > > > > > > >> ib_mthca(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > mptctl(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > >> dm_mirror(U) video(U) backlight(U) sbs(U)<br>
> > > > > > > > > > >> power_meter(U)<br>
> > > > > ><br>
> > > > > > hwmon(U)<br>
> > > > > ><br>
> > > > > > > > > > i2c_ec(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > >> i2c_core(U) dell_wmi(U) wmi(U) button(U) battery(U)<br>
> > > > > > > > > > >> asus_acpi(U) acpi_memhotplug(U) ac(U) parport_pc(U)<br>
> ><br>
> > lp(U)<br>
> ><br>
> > > > > > > > > > >> parport(U)<br>
> > > > > ><br>
> > > > > > sr_mod(U)<br>
> > > > > ><br>
> > > > > > > > > > cdrom(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > >> mlx4_ib(U) ib_mad(U) ib_core(U) joydev(U) mlx4_core(U)<br>
> > > > > > > ><br>
> > > > > > > > usb_storage(U)<br>
> > > > > > > ><br>
> > > > > > > > > > >> pcspkr(U) shpchp(U) serio_raw(U) i5000_edac(U)<br>
> ><br>
> > edac_mc(U)<br>
> ><br>
> > > > > > > > dm_raid45(U)<br>
> > > > > > > ><br>
> > > > > > > > > > >> dm_message(U) dm_region_hash(U) dm_log(U) dm_mod(U)<br>
> > > > > > > > > > >> dm_mem_cache(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > nfs(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > >> lockd(U) fscache(U) nfs_acl(U) sunrpc(U) mptsas(U)<br>
> > > ><br>
> > > > mptscsih(U)<br>
> > > ><br>
> > > > > > > > > > mptbase(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > >> scsi_transport_sas(U) mppVhba(U) megaraid_sas(U)<br>
> > > > > > > > > > >> mppUpper(U)<br>
> > > > > ><br>
> > > > > > sg(U)<br>
> > > > > ><br>
> > > > > > > > > > >> sd_mod(U) scsi_mod(U) bnx2(U) ext3(U) jbd(U)<br>
> > > > > > > > > > >> uhci_hcd(U)<br>
> > > > > > > > > > >><br>
> > > > > > > > > > >> ohci_hcd(U) ehci_hcd(U) Pid: 11360, comm: kjournald<br>
> ><br>
> > Tainted:<br>
> > > > G<br>
> > > ><br>
> > > > > > > > > > >> 2.6.18-194.3.1.el5_lustre.1.8.4 #1<br>
> > > > > > > > > > >> RIP: 0010:[<ffffffff88033448>] [<ffffffff88033448>]<br>
> > > > > > > > > > >><br>
> > > > > > > > > > >> :jbd:journal_commit_transaction+0xc5b/0x12db<br>
> > > > > > > > > > >><br>
> > > > > > > > > > >> RSP: 0018:ffff8101c6481d90 EFLAGS: 00010246<br>
> > > > > ><br>
> > > > > > > > > > >> RAX: 0000000000000000 RBX: 0000000000000000 RCX:<br>
> > > > > > 00000000ffffffff<br>
> > > > > ><br>
> > > > > > > > > > >> RDX: 0000000000000000 RSI: ffff8101e9dab0c0 RDI:<br>
> > > > > > ffff81022fa46000<br>
> > > > > ><br>
> > > > > > > > > > >> RBP: ffff81022fa46000 R08: ffff81022fa46068 R09:<br>
> > > > > > 0000000000000000<br>
> > > > > ><br>
> > > > > > > > > > >> R10: ffff810105925b20 R11: 00000000fffffffa R12:<br>
> > > > > > 0000000000000000<br>
> > > > > ><br>
> > > > > > > > > > >> R13: 0000000000000000 R14: ffff8101e9dab0c0 R15:<br>
> > > > > > 0000000000000000<br>
> > > > > ><br>
> > > > > > > > > > >> FS: 0000000000000000(0000) GS:ffff810107b9a4c0(0000)<br>
> > > > > > > > > > >> knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018<br>
> > > > > > > > > > >> CR0: 000000008005003b CR2: 0000000000000000 CR3:<br>
> > > > > > > > > > >> 00000001eaffb000<br>
> > > > > ><br>
> > > > > > CR4:<br>
> > > > > > > > > > >> 00000000000006e0 Process kjournald (pid: 11360,<br>
> ><br>
> > threadinfo<br>
> ><br>
> > > > > > > > > > >> ffff8101c6480000, task ffff81021c14c0c0)<br>
> > > > > > > > > > >> Stack: ffff8101a61b9000 000000002b8263c0<br>
> ><br>
> > ffffffff00000000<br>
> ><br>
> > > > > > > > > > 0000000000000000<br>
> > > > > > > > > ><br>
> > > > > > > > > > >> 0000113b00000001 0000000000000013 0000000000000000<br>
> > > > > > > > > > >> 0000000000000111 0000000000000000 0000000000000000<br>
> > > > > > > > > > >> 0000000001282dd7 00000000000020dd Call Trace:<br>
> > > > > > > > > > >> [<ffffffff8003da91>] lock_timer_base+0x1b/0x3c<br>
> > > > > > > > > > >> [<ffffffff8004b347>] try_to_del_timer_sync+0x7f/0x88<br>
> > > > > > > > > > >> [<ffffffff88037386>] :jbd:kjournald+0xc1/0x213<br>
> > > > > > > > > > >> [<ffffffff800a0ab2>] autoremove_wake_function+0x0/0x2e<br>
> > > > > > > > > > >> [<ffffffff800a089a>] keventd_create_kthread+0x0/0xc4<br>
> > > > > > > > > > >> [<ffffffff880372c5>] :jbd:kjournald+0x0/0x213<br>
> > > > > > > > > > >> [<ffffffff800a089a>] keventd_create_kthread+0x0/0xc4<br>
> > > > > > > > > > >> [<ffffffff80032890>] kthread+0xfe/0x132<br>
> > > > > > > > > > >> [<ffffffff8005dfb1>] child_rip+0xa/0x11<br>
> > > > > > > > > > >> [<ffffffff800a089a>] keventd_create_kthread+0x0/0xc4<br>
> > > > > > > > > > >> [<ffffffff8014bcf4>] deadline_queue_empty+0x0/0x23<br>
> > > > > > > > > > >> [<ffffffff80032792>] kthread+0x0/0x132<br>
> > > > > > > > > > >> [<ffffffff8005dfa7>] child_rip+0x0/0x11<br>
> > > > > > > > > > >><br>
> > > > > > > > > > >><br>
> > > > > > > > > > >> Code: f0 0f ba 33 01 e8 42 fc 02 f8 8b 03 a8 04 75 07<br>
> > > > > > > > > > >> 8b 43<br>
> > > ><br>
> > > > 58<br>
> > > ><br>
> > > > > > 85<br>
> > > > > ><br>
> > > > > > > > > > >> RIP [<ffffffff88033448>]<br>
> > > > > > > > :<br>
> > > > > > > > :jbd:journal_commit_transaction+0xc5b/0x12db<br>
> > > > > > > > :<br>
> > > > > > > > > > >> RSP <ffff8101c6481d90><br>
> > > > > > > > > > >> CR2: 0000000000000000<br>
> > > > > > > > > > >> <0>Kernel panic - not syncing: Fatal exception<br>
> > > > > > > > > > >><br>
> > > > > > > > > > >> On 22 October 2010 03:09, Andreas Dilger<br>
> > > > > > > > > > >> <<a href="mailto:andreas.dilger@oracle.com">andreas.dilger@oracle.com</a><br>
> > > > > > > > > ><br>
> > > > > > > > > > wrote:<br>
> > > > > > > > > > >>> On 2010-10-21, at 18:44, Wojciech Turek <<br>
> ><br>
> > <a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a>><br>
> ><br>
> > > > > > wrote:<br>
> > > > > > > > > > >>> fsck has finished and does not find any more errors<br>
> > > > > > > > > > >>> to correct. However when I try to mount the device<br>
> > > > > > > > > > >>> as ldiskfs kernel panics<br>
> > > > > > > ><br>
> > > > > > > > with<br>
> > > > > > > ><br>
> > > > > > > > > > >>> following message:<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> Assertion failure in cleanup_journal_tail() at<br>
> > > > > > > > > > >>> fs/jbd/checkpoint.c:459: "blocknr != 0"<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> Hmm, not sure, maybe your journal is broken? You can<br>
> > > ><br>
> > > > delete<br>
> > > ><br>
> > > > > > > > > > >>> it<br>
> > > > > > > ><br>
> > > > > > > > with<br>
> > > > > > > ><br>
> > > > > > > > > > >>> "tune2fs -O ^has_journal" (maybe after running e2fsck<br>
> > > > > > > > > > >>> again to<br>
> > > > > > > ><br>
> > > > > > > > clear<br>
> > > > > > > ><br>
> > > > > > > > > > the<br>
> > > > > > > > > ><br>
> > > > > > > > > > >>> journal), then re-create it with "tune2fs -j".<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> ----------- [cut here ] --------- [please bite here ]<br>
> > > > > > > > > > >>> --------- Kernel BUG at fs/jbd/checkpoint.c:459<br>
> > > > > > > > > > >>> invalid opcode: 0000 [1] SMP<br>
> > > > > > > > > > >>> last sysfs file: /class/infiniband_mad/umad0/<br>
> > > > > > > > > > >>> port<br>
> > > > > > > > > > >>> CPU 2<br>
> > > > > > > > > > >>> Modules linked in: obdfilter(U) fsfilt_ldiskfs(U)<br>
> ><br>
> > ost(U)<br>
> ><br>
> > > > > > > > > > >>> mgc(U) ldiskfs(U) crc16(U) lustre(U) lov(U) mdc(U)<br>
> > > ><br>
> > > > lquota(U)<br>
> > > ><br>
> > > > > > > > > > >>> osc(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > ksocklnd(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > >>> ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U)<br>
> > > > > > > > > > >>> libcfs(U) autofs4(U) hidp(U) l2cap(U) bluetooth(U)<br>
> > > > > > > > > > >>> rdma_ucm(U) rdma_cm(U) iw_cm(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > ib_addr(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > >>> ib_ipoib(U) ipoib_helper(U) ib_cm(U) ipv6(U)<br>
> > > > > > > > > > >>> xfrm_nalgo(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > crypto_api(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > >>> ib_uverbs(U) ib_umad(U) mlx4_vnic(U)<br>
> ><br>
> > mlx4_vnic_helper(U)<br>
> ><br>
> > > > > > ib_sa(U)<br>
> > > > > ><br>
> > > > > > > > > > >>> ib_mthca(U) mptctl(U) dm_mirror(U) video(U)<br>
> ><br>
> > backlight(U)<br>
> ><br>
> > > > > > > > > > >>> sbs(U) power_meter(U) hwmon(U) i2c_ec(U) i2c_core(U)<br>
> > > > > > > > > > >>> dell_wmi(U)<br>
> > > > > ><br>
> > > > > > wmi(U)<br>
> > > > > ><br>
> > > > > > > > > > >>> button(U) battery(U) asus_acpi(U) acpi_memhotplug(U)<br>
> > > > > > > > > > >>> ac(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > parport_pc(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > >>> lp(U) parport(U) sr_mod(U) cdrom(U) mlx4_ib(U)<br>
> ><br>
> > ib_mad(U)<br>
> ><br>
> > > > > > > > > > >>> ib_core(U) joydev(U) mlx4_core(U) usb_storage(U)<br>
> > > > > > > > > > >>> shpchp(U) i5000_edac(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > edac_mc(U)<br>
> > > > > > > > > ><br>
> > > > > > > > > > >>> serio_raw(U) pcspkr(U) dm_raid45(U) dm_message(U)<br>
> > > > > > > > > > >>> dm_region_hash(U) dm_log(U) dm_mod(U) dm_mem_cache(U)<br>
> > > ><br>
> > > > nfs(U)<br>
> > > ><br>
> > > > > > > > > > >>> lockd(U) fscache(U) nfs_acl(U) sunrpc(U) mptsas(U)<br>
> > > > > > > > > > >>> mptscsih(U) mptbase(U)<br>
> > > > > > > > > > >>> scsi_transport_sas(U) mppVhba(U) megaraid_sas(U)<br>
> > > ><br>
> > > > mppUpper(U)<br>
> > > ><br>
> > > > > > > > > > >>> sg(U) sd_mod(U) scsi_mod(U) bnx2(U) ext3(U) jbd(U)<br>
> > > > > > > > > > >>> uhci_hcd(U)<br>
> > > > > > > ><br>
> > > > > > > > ohci_hcd(U)<br>
> > > > > > > ><br>
> > > > > > > > > > >>> ehci_hcd(U) Pid: 13891, comm: mount Tainted: G<br>
> > > > > ><br>
> > > > > > > > > > >>> 2.6.18-194.3.1.el5_lustre.1.8.4 #1 RIP:<br>
> > > > > > 0010:[<ffffffff88034a95>]<br>
> > > > > ><br>
> > > > > > > > > > >>> [<ffffffff88034a95>]<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> :jbd:cleanup_journal_tail+0x9d/0x118<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> RSP: 0018:ffff81016f00da68 EFLAGS: 00010286<br>
> > > > > ><br>
> > > > > > > > > > >>> RAX: 000000000000005a RBX: ffff81012ca12c00 RCX:<br>
> > > > > > ffffffff80311da8<br>
> > > > > ><br>
> > > > > > > > > > >>> RDX: ffffffff80311da8 RSI: 0000000000000000 RDI:<br>
> > > > > > ffffffff80311da0<br>
> > > > > ><br>
> > > > > > > > > > >>> RBP: 0000000000000000 R08: ffffffff80311da8 R09:<br>
> > > > > > 0000000000000001<br>
> > > > > ><br>
> > > > > > > > > > >>> R10: 0000000000000000 R11: 0000000000000080 R12:<br>
> > > > > > 0000000000000002<br>
> > > > > ><br>
> > > > > > > > > > >>> R13: ffff81012ca12d4c R14: ffff81012ca12c24 R15:<br>
> > > > > > ffff81017a8d7400<br>
> > > > > ><br>
> > > > > > > > > > >>> FS: 00002abd7cef1f70(0000) GS:ffff810107b9acc0(0000)<br>
> > > > > > > > > > >>> knlGS:0000000000000000<br>
> > > > > > > > > > >>> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br>
> > > > > ><br>
> > > > > > > > > > >>> CR2: 000000000042b000 CR3: 000000012813f000 CR4:<br>
> > > > > > 00000000000006e0<br>
> > > > > ><br>
> > > > > > > > > > >>> Process mount (pid: 13891, threadinfo<br>
> > > > > > > > > > >>> ffff81016f00c000,<br>
> > > ><br>
> > > > task<br>
> > > ><br>
> > > > > > > > > > >>> ffff81022e1b7820)<br>
> > > > > > > > > > >>> Stack: 0000000000000000 ffff81012ca12c00<br>
> > > > > > > > > > >>> ffff81017a8d7400 ffffffff88037690<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> ffff81012ca12c00 ffff8102034ff000 ffff81017a8d7400<br>
> > > > > > > > > > >>> 0000000000000000 ffff8102034ff000 ffffffff88a9be56<br>
> > > > > > > > > > >>> 0000000001000000 ffff8101bf788000<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> Call Trace:<br>
> > > > > > > > > > >>> [<ffffffff88037690>] :jbd:journal_flush+0xbe/0x248<br>
> > > > > > > > > > >>> [<ffffffff88a9be56>]<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> :ldiskfs:ldiskfs_mark_recovery_complete+0x36/0x90<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> [<ffffffff88aa02e0>]<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> :ldiskfs:ldiskfs_fill_super+0x1790/0x1950<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> [<ffffffff800eccd2>] get_filesystem+0x12/0x3b<br>
> > > > > > > > > > >>> [<ffffffff800e343e>] test_bdev_super+0x0/0xd<br>
> > > > > > > > > > >>> [<ffffffff88a9eb50>]<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> :ldiskfs:ldiskfs_fill_super+0x0/0x1950<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> [<ffffffff800e43fd>] get_sb_bdev+0x10a/0x16c<br>
> > > > > > > > > > >>> [<ffffffff800e3d9a>] vfs_kern_mount+0x93/0x11a<br>
> > > > > > > > > > >>> [<ffffffff800e3e63>] do_kern_mount+0x36/0x4d<br>
> > > > > > > > > > >>> [<ffffffff800ee601>] do_mount+0x6a9/0x719<br>
> > > > > > > > > > >>> [<ffffffff800090d2>] __handle_mm_fault+0x96f/0xfaa<br>
> > > > > > > > > > >>> [<ffffffff8002c9e0>] mntput_no_expire+0x19/0x89<br>
> > > > > > > > > > >>> [<ffffffff8000a72a>] __link_path_walk+0xf1e/0xf42<br>
> > > > > > > > > > >>> [<ffffffff800220ce>] __up_read+0x19/0x7f<br>
> > > > > > > > > > >>> [<ffffffff80066b88>] do_page_fault+0x4fe/0x874<br>
> > > > > > > > > > >>> [<ffffffff8002c9e0>] mntput_no_expire+0x19/0x89<br>
> > > > > > > > > > >>> [<ffffffff8000ea45>] link_path_walk+0xa6/0xb2<br>
> > > > > > > > > > >>> [<ffffffff800cc329>] zone_statistics+0x3e/0x6d<br>
> > > > > > > > > > >>> [<ffffffff8000f2cf>] __alloc_pages+0x78/0x308<br>
> > > > > > > > > > >>> [<ffffffff8004c68e>] sys_mount+0x8a/0xcd<br>
> > > > > > > > > > >>> [<ffffffff8005d28d>] tracesys+0xd5/0xe0<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> Code: 0f 0b 68 3a 94 03 88 c2 cb 01 44 39 a3 58 01 00<br>
> ><br>
> > 00<br>
> ><br>
> > > > > > > > > > >>> 75 0e<br>
> > > > > ><br>
> > > > > > c7<br>
> > > > > ><br>
> > > > > > > > > > >>> RIP [<ffffffff88034a95>]<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> :jbd:cleanup_journal_tail+0x9d/0x118<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> RSP <ffff81016f00da68><br>
> > > > > > > > > > >>> <0>Kernel panic - not syncing: Fatal exception<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> Any idea how to fix this?<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> Many thanks<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> Wojciech<br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>><br>
> > > > > > > > > > >>> On 21 October 2010 17:54, Wojciech Turek < <<br>
> > > ><br>
> > > > <a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a>><br>
> > > ><br>
> > > > > > > > > > >>> <a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a>> wrote:<br>
> > > > > > > > > > >>>> Thanks Ken, that worked.<br>
> > > > > > > > > > >>>><br>
> > > > > > > > > > >>>><br>
> > > > > > > > > > >>>> On 21 October 2010 17:39, Ken Hornstein <<br>
> > > > > > > > > > >>>> <<a href="mailto:kenh@cmf.nrl.navy.mil">kenh@cmf.nrl.navy.mil</a>><br>
> > > > > > > > > > >>>><br>
> > > > > > > > > > >>>> <a href="mailto:kenh@cmf.nrl.navy.mil">kenh@cmf.nrl.navy.mil</a>> wrote:<br>
> > > > > > > > > > >>>>>> Now I have another problem. After last segfault I<br>
> ><br>
> > can<br>
> ><br>
> > > > not<br>
> > > ><br>
> > > > > > > > restart<br>
> > > > > > > ><br>
> > > > > > > > > > the<br>
> > > > > > > > > ><br>
> > > > > > > > > > >>>>> fsck<br>
> > > > > > > > > > >>>>><br>
> > > > > > > > > > >>>>>> due to MMP.<br>
> > > > > > > > > > >>>>>> [...]<br>
> > > > > > > > > > >>>>>> Also when I try to access filesystem via debugfs<br>
> > > > > > > > > > >>>>>> it<br>
> > > ><br>
> > > > fails:<br>
> > > > > > > > > > >>>>>> debugfs -c -R 'ls' /dev/scratch2_ost16vg/ost16lv<br>
> > > > > > > > > > >>>>>> debugfs 1.41.10.sun2 (24-Feb-2010)<br>
> > > > > > > > > > >>>>>> /dev/scratch2_ost16vg/ost16lv: MMP: fsck being run<br>
> > > > > > > > > > >>>>>> while opening<br>
> > > > > > > > > > >>>>><br>
> > > > > > > > > > >>>>> filesystem<br>
> > > > > > > > > > >>>>><br>
> > > > > > > > > > >>>>>> ls: Filesystem not open<br>
> > > > > > > > > > >>>>>><br>
> > > > > > > > > > >>>>>> Is there a way to clear teh MMP flag so it allows<br>
> ><br>
> > fsck<br>
> ><br>
> > > > to<br>
> > > ><br>
> > > > > > run?<br>
> > > > > ><br>
> > > > > > > > > > >>>>> You want tune2fs -f -E clear-mmp<br>
> > > > > > > > > > >>>>><br>
> > > > > > > > > > >>>>> --Ken<br>
> > > ><br>
> > > > --<br>
> > > > Bernd Schubert<br>
> > > > DataDirect Networks<br>
> ><br>
> > --<br>
> > Bernd Schubert<br>
> > DataDirect Networks<br>
<br>
<br>
--<br>
</div></div><font color="#888888">Bernd Schubert<br>
DataDirect Networks<br>
</font></blockquote></div>