Hi Damiri<br><br>I use more recent e2fsprogs:<br>e2fsprogs-1.41.12.2.ora1-0redhat.x86_64<br><br>I think you can get even more recent version from Whamcloude.<br><br>I remember that your version of e2fsck does not allow access to mounted FS in read only mode which is annoying and not necessary. With more recent version fsck -n should run even on mounted FS.<br>
<br>Anyway for proper fsck'ing you need to umount device that you want to check and make sure that it is not mounted on the other node. Stopping heartbeat should automatically umount your OSTs if your FileSystem resources are properly configured.<br>
<br>Best regards,<br><br>Wojciech<br><br><div class="gmail_quote">On 20 July 2011 16:37, DaMiri Young <span dir="ltr"><<a href="mailto:damiri@unt.edu">damiri@unt.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi Wojciech,<br>
Stopping heartbeat sounds like a logical next step. Before I do that though I tried a fsck dry run using e2fsprogs v1.14.10 and got:<br>
------------------------------<u></u>------------------------------<u></u>---<br>
# e2fsck -n -v /dev/dm-11<br>
e2fsck 1.41.10.sun2 (24-Feb-2010)<br>
device /dev/dm-11 mounted by lustre per /proc/fs/lustre/obdfilter/es1-<u></u>OST000a/mntdev<br>
Warning! /dev/dm-11 is mounted.<br>
e2fsck: MMP: device currently active while trying to open /dev/dm-11<br>
<br>
The superblock could not be read or does not describe a correct ext2<br>
filesystem. If the device is valid and it really contains an ext2<br>
filesystem (and not swap or ufs or something else), then the superblock<br>
is corrupt, and you might try running e2fsck with an alternate superblock:<br>
e2fsck -b 32768 <device><br>
------------------------------<u></u>------------------------------<u></u>------<br>
<br>
Do you suppose stopping heartbeat will allow the OST to be unmounted all the way be lustre? I tried unmounting manually and got:<br>
------------------------------<u></u>------------------------------<u></u>------<br>
# umount /dev/dm-11<br>
umount: /dev/dm-11: not mounted<br>
------------------------------<u></u>------------------------------<u></u>------<br>
<br>
<br>
Wojciech Turek wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi Damiri,<br>
<br><div class="im">
If heartbeat is not able to start(mount) one of the OSTs I would recommend to stop heartbeat on both servers and then mount troubled OST manually. Then you should see why OST is not mounted. In order to check the consistency of the filesystem, in your case I would first run fsck with -n switch to see extent of the damage, this also prevents from damaging your filesystem even more if you have a faulty controller or links corrupting data. In normal situation I use following command: fsck -f -v /dev/<ost_dev> -C0<br>
Make sure that you log output from the fsck which will be essential for the further troubleshooting.<br>
<br>
Best regards,<br>
<br>
Wojciech<br>
<br></div><div class="im">
On 19 July 2011 16:58, Young, Damiri <<a href="mailto:Damiri.Young@unt.edu" target="_blank">Damiri.Young@unt.edu</a> <mailto:<a href="mailto:Damiri.Young@unt.edu" target="_blank">Damiri.Young@unt.edu</a>>> wrote:<br>
<br>
Many thanks for the useful info Turek. I mentioned HA (heartbeat v2)<br>
issues because after the troubled I/O got it's paths back to the<br>
OST's it failed all 4 of the 5 OSTs over to it's sibling server<br>
where they're now mounted. To me it seems the OSTs (we're using<br>
lustre v1.6 btw) won't be released until the failed over node is<br>
reset by it's sibling.<br>
<br>
The OSSs seem to have trouble connecting to the 1 OST I mentioned:<br>
------------------------------<u></u>-- messages<br>
------------------------------<u></u>-------<br>
Jul 19 10:29:02 IO-10 kernel: LustreError: 29429: <br>
</div></blockquote>
<br><div><div></div><div class="h5">
<br>
-- <br>
DaMiri Young<br>
HPC System Engineer<br>
High Performance Computing Team | ACUS/CITC | UNT<br>
</div></div></blockquote></div><br><br>