[Lustre-discuss] recovering formatted OST

Wojciech Turek wjt27 at cam.ac.uk
Thu Oct 21 09:51:48 PDT 2010


Hi Bernd,

Thanks for the tip.
I don't have high hopes for recovering to much but from where I stand I have
nothing to loose. Failed OST was a part of the scratch filesystem so in
theory the data weren't that sensitive. However some people would be very
happy if they could recover any data.

Best regards,

Wojciech

On 21 October 2010 17:45, Bernd Schubert <bs_lists at aakef.fastmail.fm> wrote:

> Hello Wojciech Turek,
>
>
> On Thursday, October 21, 2010, Wojciech Turek wrote:
> > Hi Andreas,
> >
> > I have restarted fsck after the segfault and it ran for several hours and
> > it segfaulted again.
> >
> > Pass 3A: Optimizing directories
> > Failed to optimize directory ??? (73031): EXT2 directory corrupted
> > Failed to optimize directory ??? (73041): EXT2 directory corrupted
> > Failed to optimize directory ??? (75203): EXT2 directory corrupted
> > Failed to optimize directory ??? (75357): EXT2 directory corrupted
> > Failed to optimize directory ??? (75744): EXT2 directory corrupted
> > Failed to optimize directory ??? (75806): EXT2 directory corrupted
> > Failed to optimize directory ??? (75825): EXT2 directory corrupted
> > Failed to optimize directory ??? (75913): EXT2 directory corrupted
> > Failed to optimize directory ??? (75926): EXT2 directory corrupted
> > Failed to optimize directory ??? (76034): EXT2 directory corrupted
> > Failed to optimize directory ??? (76083): EXT2 directory corrupted
> > Failed to optimize directory ??? (76142): EXT2 directory corrupted
> > Failed to optimize directory ??? (76266): EXT2 directory corrupted
> > Failed to optimize directory ??? (76501): EXT2 directory corrupted
> > Failed to optimize directory ??? (77133): EXT2 directory corrupted
> > Failed to optimize directory ??? (77212): EXT2 directory corrupted
> > Failed to optimize directory ??? (77817): EXT2 directory corrupted
> > Failed to optimize directory ??? (77984): EXT2 directory corrupted
> > Failed to optimize directory ??? (77985): EXT2 directory corrupted
> > Segmentation fault
>
> Maybe try to disable dirindex?
>
> >
> > I noticed that the stack limit was quite low so I now changed it to
> > unlimited, also I increased limit for number of open files (maybe it can
> > help).
> >
> > Now I have another problem. After last segfault I can not restart the
> fsck
> > due to MMP.
> >
> > e2fsck -fy /dev/scratch2_ost16vg/ost16lv
> > e2fsck 1.41.10.sun2 (24-Feb-2010)
> > e2fsck: MMP: fsck being run while trying to open
> > /dev/scratch2_ost16vg/ost16lv
> >
> > The superblock could not be read or does not describe a correct ext2
> > filesystem.  If the device is valid and it really contains an ext2
> > filesystem (and not swap or ufs or something else), then the superblock
> > is corrupt, and you might try running e2fsck with an alternate
> superblock:
> >     e2fsck -b 32768 <device>
> >
> >
> > Also when I try to access filesystem via debugfs it fails:
> >
> > debugfs -c -R 'ls' /dev/scratch2_ost16vg/ost16lv
> > debugfs 1.41.10.sun2 (24-Feb-2010)
> > /dev/scratch2_ost16vg/ost16lv: MMP: fsck being run while opening
> filesystem
> > ls: Filesystem not open
>
> >
> > Is there a way to clear teh MMP flag so it allows fsck to run?
>
> you can try tune2fs -f -E clear-mmp
>
> However, with a corrupted filesystem, that might not work. You can download
> a
> fixed e2fsprogs from my homepage, that does allow to run read-only
> operations
> (such as 'debugfs -c' or 'dumpe2fs -h') in read-only mode. Then you check
> which block is the MMP block and zero that.
>
> http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/e2fsprogs/
>
> (just reminds me, I need to upload it to our DDN download site)
>
>
> Also, do you really want to use data files, that might have been zeroed in
> their middle? I think If at all your recovery will only be useful for small
> human readable text files....
>
>
> Hope it helps,
> Bernd
>
>
> --
> Bernd Schubert
> DataDirect Networkk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20101021/ddf3bef2/attachment.htm>


More information about the lustre-discuss mailing list