[Lustre-discuss] recovering formatted OST

Bernd Schubert bs_lists at aakef.fastmail.fm
Thu Oct 21 09:45:55 PDT 2010


Hello Wojciech Turek,


On Thursday, October 21, 2010, Wojciech Turek wrote:
> Hi Andreas,
> 
> I have restarted fsck after the segfault and it ran for several hours and
> it segfaulted again.
> 
> Pass 3A: Optimizing directories
> Failed to optimize directory ??? (73031): EXT2 directory corrupted
> Failed to optimize directory ??? (73041): EXT2 directory corrupted
> Failed to optimize directory ??? (75203): EXT2 directory corrupted
> Failed to optimize directory ??? (75357): EXT2 directory corrupted
> Failed to optimize directory ??? (75744): EXT2 directory corrupted
> Failed to optimize directory ??? (75806): EXT2 directory corrupted
> Failed to optimize directory ??? (75825): EXT2 directory corrupted
> Failed to optimize directory ??? (75913): EXT2 directory corrupted
> Failed to optimize directory ??? (75926): EXT2 directory corrupted
> Failed to optimize directory ??? (76034): EXT2 directory corrupted
> Failed to optimize directory ??? (76083): EXT2 directory corrupted
> Failed to optimize directory ??? (76142): EXT2 directory corrupted
> Failed to optimize directory ??? (76266): EXT2 directory corrupted
> Failed to optimize directory ??? (76501): EXT2 directory corrupted
> Failed to optimize directory ??? (77133): EXT2 directory corrupted
> Failed to optimize directory ??? (77212): EXT2 directory corrupted
> Failed to optimize directory ??? (77817): EXT2 directory corrupted
> Failed to optimize directory ??? (77984): EXT2 directory corrupted
> Failed to optimize directory ??? (77985): EXT2 directory corrupted
> Segmentation fault

Maybe try to disable dirindex?

> 
> I noticed that the stack limit was quite low so I now changed it to
> unlimited, also I increased limit for number of open files (maybe it can
> help).
> 
> Now I have another problem. After last segfault I can not restart the fsck
> due to MMP.
> 
> e2fsck -fy /dev/scratch2_ost16vg/ost16lv
> e2fsck 1.41.10.sun2 (24-Feb-2010)
> e2fsck: MMP: fsck being run while trying to open
> /dev/scratch2_ost16vg/ost16lv
> 
> The superblock could not be read or does not describe a correct ext2
> filesystem.  If the device is valid and it really contains an ext2
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
>     e2fsck -b 32768 <device>
> 
> 
> Also when I try to access filesystem via debugfs it fails:
> 
> debugfs -c -R 'ls' /dev/scratch2_ost16vg/ost16lv
> debugfs 1.41.10.sun2 (24-Feb-2010)
> /dev/scratch2_ost16vg/ost16lv: MMP: fsck being run while opening filesystem
> ls: Filesystem not open

> 
> Is there a way to clear teh MMP flag so it allows fsck to run?

you can try tune2fs -f -E clear-mmp

However, with a corrupted filesystem, that might not work. You can download a 
fixed e2fsprogs from my homepage, that does allow to run read-only operations 
(such as 'debugfs -c' or 'dumpe2fs -h') in read-only mode. Then you check 
which block is the MMP block and zero that.

http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/e2fsprogs/

(just reminds me, I need to upload it to our DDN download site)


Also, do you really want to use data files, that might have been zeroed in 
their middle? I think If at all your recovery will only be useful for small 
human readable text files....


Hope it helps,
Bernd


-- 
Bernd Schubert
DataDirect Networks



More information about the lustre-discuss mailing list