[Lustre-discuss] recovering formatted OST
Bernd Schubert
bs_lists at aakef.fastmail.fm
Thu Oct 21 09:45:55 PDT 2010
Hello Wojciech Turek,
On Thursday, October 21, 2010, Wojciech Turek wrote:
> Hi Andreas,
>
> I have restarted fsck after the segfault and it ran for several hours and
> it segfaulted again.
>
> Pass 3A: Optimizing directories
> Failed to optimize directory ??? (73031): EXT2 directory corrupted
> Failed to optimize directory ??? (73041): EXT2 directory corrupted
> Failed to optimize directory ??? (75203): EXT2 directory corrupted
> Failed to optimize directory ??? (75357): EXT2 directory corrupted
> Failed to optimize directory ??? (75744): EXT2 directory corrupted
> Failed to optimize directory ??? (75806): EXT2 directory corrupted
> Failed to optimize directory ??? (75825): EXT2 directory corrupted
> Failed to optimize directory ??? (75913): EXT2 directory corrupted
> Failed to optimize directory ??? (75926): EXT2 directory corrupted
> Failed to optimize directory ??? (76034): EXT2 directory corrupted
> Failed to optimize directory ??? (76083): EXT2 directory corrupted
> Failed to optimize directory ??? (76142): EXT2 directory corrupted
> Failed to optimize directory ??? (76266): EXT2 directory corrupted
> Failed to optimize directory ??? (76501): EXT2 directory corrupted
> Failed to optimize directory ??? (77133): EXT2 directory corrupted
> Failed to optimize directory ??? (77212): EXT2 directory corrupted
> Failed to optimize directory ??? (77817): EXT2 directory corrupted
> Failed to optimize directory ??? (77984): EXT2 directory corrupted
> Failed to optimize directory ??? (77985): EXT2 directory corrupted
> Segmentation fault
Maybe try to disable dirindex?
>
> I noticed that the stack limit was quite low so I now changed it to
> unlimited, also I increased limit for number of open files (maybe it can
> help).
>
> Now I have another problem. After last segfault I can not restart the fsck
> due to MMP.
>
> e2fsck -fy /dev/scratch2_ost16vg/ost16lv
> e2fsck 1.41.10.sun2 (24-Feb-2010)
> e2fsck: MMP: fsck being run while trying to open
> /dev/scratch2_ost16vg/ost16lv
>
> The superblock could not be read or does not describe a correct ext2
> filesystem. If the device is valid and it really contains an ext2
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
> e2fsck -b 32768 <device>
>
>
> Also when I try to access filesystem via debugfs it fails:
>
> debugfs -c -R 'ls' /dev/scratch2_ost16vg/ost16lv
> debugfs 1.41.10.sun2 (24-Feb-2010)
> /dev/scratch2_ost16vg/ost16lv: MMP: fsck being run while opening filesystem
> ls: Filesystem not open
>
> Is there a way to clear teh MMP flag so it allows fsck to run?
you can try tune2fs -f -E clear-mmp
However, with a corrupted filesystem, that might not work. You can download a
fixed e2fsprogs from my homepage, that does allow to run read-only operations
(such as 'debugfs -c' or 'dumpe2fs -h') in read-only mode. Then you check
which block is the MMP block and zero that.
http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/e2fsprogs/
(just reminds me, I need to upload it to our DDN download site)
Also, do you really want to use data files, that might have been zeroed in
their middle? I think If at all your recovery will only be useful for small
human readable text files....
Hope it helps,
Bernd
--
Bernd Schubert
DataDirect Networks
More information about the lustre-discuss
mailing list