Hi Alex,<br><br>So if I understand you correctly you have accidentally destroyed your external journals. So it seem that your OSTs are missing journals. Maybe the fix will be to recreate the journal on the OSTs<br><br>regards,<br>
<br>Wojciech<br><br><div class="gmail_quote">On 26 October 2010 20:42, Alexander Bugl <span dir="ltr"><<a href="mailto:alexander.bugl@zmaw.de">alexander.bugl@zmaw.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi,<br>
<br>
we had an accident with a Sun Fire X4540 "Thor" System with 48 HDDs:<br>
<br>
The first two disks sda and sdb contain several partitions, one for the / file<br>
system, one for swap (not used) and 5 small partitions used as external<br>
journals for the OSTs, which reside on the 46 other HDDs.<br>
<br>
[root@soss10 ~]# fdisk -l /dev/sda /dev/sdb<br>
<br>
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes<br>
255 heads, 63 sectors/track, 121601 cylinders<br>
Units = cylinders of 16065 * 512 = 8225280 bytes<br>
Device Boot Start End Blocks Id System<br>
/dev/sda1 * 1 6527 52428096 fd Linux raid autodetect<br>
/dev/sda2 6528 10704 33551752+ fd Linux raid autodetect<br>
/dev/sda3 10705 121601 890780152+ 5 Extended<br>
/dev/sda5 10705 10953 2000061 fd Linux raid autodetect<br>
/dev/sda6 10954 11202 2000061 fd Linux raid autodetect<br>
/dev/sda7 11203 11451 2000061 fd Linux raid autodetect<br>
/dev/sda8 11452 11700 2000061 fd Linux raid autodetect<br>
/dev/sda9 11701 11949 2000061 fd Linux raid autodetect<br>
<br>
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes<br>
255 heads, 63 sectors/track, 121601 cylinders<br>
Units = cylinders of 16065 * 512 = 8225280 bytes<br>
Device Boot Start End Blocks Id System<br>
/dev/sdb1 * 1 6527 52428096 fd Linux raid autodetect<br>
/dev/sdb2 6528 10704 33551752+ fd Linux raid autodetect<br>
/dev/sdb3 10705 121601 890780152+ 5 Extended<br>
/dev/sdb5 10705 10953 2000061 fd Linux raid autodetect<br>
/dev/sdb6 10954 11202 2000061 fd Linux raid autodetect<br>
/dev/sdb7 11203 11451 2000061 fd Linux raid autodetect<br>
/dev/sdb8 11452 11700 2000061 fd Linux raid autodetect<br>
/dev/sdb9 11701 11949 2000061 fd Linux raid autodetect<br>
<br>
The md devices are:<br>
md14 : active raid6 sdw[0] sdav[9] sdan[8] sdaf[7] sdx[6] sdp[5] sdh[4]<br>
sdau[3] sdam[2] sdae[1]<br>
7814099968 blocks level 6, 64k chunk, algorithm 2 [10/10] [UUUUUUUUUU]<br>
<br>
md13 : active raid6 sdak[0] sdo[9] sdg[8] sdat[7] sdal[6] sdad[5] sdv[4]<br>
sdn[3] sdf[2] sdas[1]<br>
7814099968 blocks level 6, 64k chunk, algorithm 2 [10/10] [UUUUUUUUUU]<br>
<br>
md12 : active raid6 sdd[0] sdac[9] sdu[8] sdm[7] sde[6] sdar[5] sdaj[4]<br>
sdab[3] sdt[2] sdl[1]<br>
7814099968 blocks level 6, 64k chunk, algorithm 2 [10/10] [UUUUUUUUUU]<br>
<br>
md11 : active raid6 sdah[0] sdaq[7] sdai[6] sdaa[5] sds[4] sdk[3] sdc[2]<br>
sdap[1]<br>
5860574976 blocks level 6, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]<br>
<br>
md10 : active raid6 sdi[0] sdz[7] sdao[6] sdag[5] sdy[4] sdr[3] sdq[2] sdj[1]<br>
5860574976 blocks level 6, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]<br>
<br>
md1 : active raid1 sdb2[1] sda2[0]<br>
33551680 blocks [2/2] [UU]<br>
<br>
md20 : active raid1 sdb5[1] sda5[0]<br>
1999936 blocks [2/2] [UU]<br>
<br>
md21 : active raid1 sdb6[1] sda6[0]<br>
1999936 blocks [2/2] [UU]<br>
<br>
md22 : active raid1 sdb7[1] sda7[0]<br>
1999936 blocks [2/2] [UU]<br>
<br>
md23 : active raid1 sdb8[1] sda8[0]<br>
1999936 blocks [2/2] [UU]<br>
<br>
md24 : active raid1 sdb9[1] sda9[0]<br>
1999936 blocks [2/2] [UU]<br>
<br>
md0 : active raid1 sdb1[1] sda1[0]<br>
52428032 blocks [2/2] [UU]<br>
<br>
The original OSTs had been created using a command like:<br>
mkfs.lustre --ost --fsname=${FSNAME} --mgsnode=${MGSNODE}@o2ib \<br>
--reformat --mkfsoptions="-m 0 -J device=/dev/md20" \<br>
--param ost.quota_type=ug /dev/md10 &<br>
(the pairs md21/md11, md22/md12, ..., respectively)<br>
<br>
Accidentally we started a fresh installation, which could not be aborted fast<br>
enough -- the partition information on sda and sdb was erased.<br>
The other 46 disks should not have been harmed, though.<br>
<br>
We started a reinstallation which only formatted the first 2 partitions and<br>
which recreated the partition layout on sda and sdb, all of the md devices<br>
resynced without problems.<br>
<br>
When we now try to mount any of the 5 OSTs, we get the following error:<br>
<br>
[root@soss10 ~]# mount /dev/md14<br>
mount.lustre: mount /dev/md14 at /lustre/ost4 failed: Invalid argument<br>
This may have multiple causes.<br>
Are the mount options correct?<br>
Check the syslog for more info.<br>
<br>
syslog says:<br>
Oct 26 21:34:55 soss10 kernel: LDISKFS-fs error (device md14):<br>
ldiskfs_check_descriptors: Block bitmap for group 1920 not in group (block<br>
268482810)!<br>
Oct 26 21:34:55 soss10 kernel: LDISKFS-fs: group descriptors corrupted!<br>
Oct 26 21:34:55 soss10 kernel: LustreError: 10719:0:<br>
(obd_mount.c:1292:server_kernel_mount()) premount /dev/md14:0x0 ldiskfs<br>
failed: -22, ldiskfs2 failed: -19. Is the ldiskfs module available?<br>
Oct 26 21:34:56 soss10 kernel: LustreError: 10719:0:<br>
(obd_mount.c:1618:server_fill_super()) Unable to mount device /dev/md14: -22<br>
Oct 26 21:34:56 soss10 kernel: LustreError: 10719:0:<br>
(obd_mount.c:2050:lustre_fill_super()) Unable to mount (-22)<br>
<br>
Trying to mount the partition as ldiskfs does not work, either:<br>
[root@soss10 ~]# mount -t ldiskfs /dev/md14 /mnt<br>
mount: wrong fs type, bad option, bad superblock on /dev/md14,<br>
missing codepage or other error<br>
In some cases useful info is found in syslog - try<br>
dmesg | tail or so<br>
syslog only says:<br>
Oct 26 21:35:54 soss10 kernel: LDISKFS-fs error (device md14):<br>
ldiskfs_check_descriptors: Block bitmap for group 1920 not in group (block<br>
268482810)!<br>
Oct 26 21:35:54 soss10 kernel: LDISKFS-fs: group descriptors corrupted!<br>
<br>
Trying to run e2fsck -n yields:<br>
[root@soss10 ~]# e2fsck -n /dev/md10<br>
e2fsck 1.41.10.sun2 (24-Feb-2010)<br>
e2fsck: Group descriptors look bad... trying backup blocks...<br>
Error writing block 1 (Attempt to write block from filesystem resulted in<br>
short write). Ignore error? no<br>
Error writing block 2 (Attempt to write block from filesystem resulted in<br>
short write). Ignore error? no<br>
Error writing block 3 (Attempt to write block from filesystem resulted in<br>
short write). Ignore error? no<br>
Error writing block 4 (Attempt to write block from filesystem resulted in<br>
short write). Ignore error? no<br>
... [continues up to block 344]<br>
One or more block group descriptor checksums are invalid. Fix? no<br>
Group descriptor 0 checksum is invalid. IGNORED.<br>
Group descriptor 1 checksum is invalid. IGNORED.<br>
Group descriptor 2 checksum is invalid. IGNORED.<br>
Group descriptor 3 checksum is invalid. IGNORED.<br>
... [continues up to Group descriptor 44712]<br>
squall-OST0019 contains a file system with errors, check forced.<br>
Pass 1: Checking inodes, blocks, and sizes<br>
<br>
(the rest of e2fsck is till running ...)<br>
<br>
Question: What could be the problem, I thought that no data on the OSTs and<br>
insode the journal partitions should have been overwritten. Is there any<br>
chance to repair these problems without data loss?<br>
<br>
Thank you in advance for any suggestions about how to continue ...<br>
With regards, Alex<br>
<br>
--<br>
<font color="#888888">Alexander Bugl, Central IT Services, ZMAW<br>
Max Planck Institute for Meteorology<br>
Bundesstrasse 53, D-20146 Hamburg, Germany<br>
tel +49-40-41173-351, fax -298, room PE048<br>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
</font></blockquote></div><br>