<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hey folks,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
We had a power "incident". Not sure if it was the cause of our issues or if it just brought previous issues to light. We're a CentOS 7, lustre 2.10.6-1.el7 (from provided binaries) site, SAS direct connect HA paired OSSs running pacemaker to manage failover.
Standard stuff. After the power incident, some OSTS were dropped and remounted but one did not come back. At this point, that OST does not seem to be recognized as a lustre volume.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
First step I took was to disable the pacemaker resource and try to mount it manually to see how it was doing:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
[root@hpcoss02 ~]# mount -t lustre /dev/mapper/mpathg /mnt/OST78
<div>mount.lustre: /dev/mapper/mpathg has not been formatted with mkfs.lustre or the backend filesystem type is not supported by this tool</div>
<div><br>
</div>
<div>The syslog shows the following at that time (syslog is from a subsequent attempt but logs match):</div>
<div>May 07 12:04:15 hpcoss02.adqimr.ad.lan kernel: LDISKFS-fs (dm-6): ldiskfs_check_descriptors: Checksum for group 192 failed (39981!=25867)<br>
May 07 12:04:15 hpcoss02.adqimr.ad.lan kernel: LDISKFS-fs (dm-6): group descriptors corrupted!<br>
</div>
<div><br>
</div>
<div>No fun. Next attempt was to try mounting ldiskfs in case a journal replay would help:</div>
<div><br>
</div>
<div>[root@hpcoss02 ~]# mount -t ldiskfs /dev/mapper/mpathg /mnt/OST78
<div>mount: wrong fs type, bad option, bad superblock on /dev/mapper/mpathg,</div>
<div> missing codepage or helper program, or other error</div>
<div><br>
</div>
<div> In some cases useful info is found in syslog - try</div>
dmesg | tail or so.<br>
</div>
<div><br>
</div>
<div>Unhappy chappie. This OST has been up and running happily for months so it knows it's part of the stack. I ran a "tune2fs -l" against it and, from the "Filesystem magic number: 0xEF53", it looks like it knows it's a lustre volume. I ran an "e2fsck
-n" against it. I'll spare you the details but am happy to answer specifics if you have ideas about what to look for but it does not look good. stdout went to "out" stderr went to "err". "out" shows the following and I can dig deeper or answer questions:</div>
<div><br>
</div>
<div>[root@hpcoss01 fsck]# grep "Group descriptor.*checksum is.*, should be" out |head -n1
<div>Group descriptor 192 checksum is 0x650b, should be 0x9c2d. IGNORED.</div>
<div>[root@hpcoss01 fsck]# grep "Group descriptor.*checksum is.*, should be" out |wc -l</div>
<div>435299</div>
<div>[root@hpcoss01 fsck]# grep "Free blocks count wrong for group" out |head -n1
<div>Free blocks count wrong for group #192 (32768, counted=0).</div>
</div>
[root@hpcoss01 fsck]# grep "Free blocks count wrong for group" out |wc -l
<div>412325</div>
<br>
</div>
<div>"err" showed quote issues but we're not too concerned about them as we don't enforce.</div>
<div><br>
</div>
<div>We are currently replicating the block device of the OST to Logical Volume so we can run non-destructive tests against an LVM snapshot to see what we get (thanks @stu for the suggestion). We're also running an "lfs find mountpoint -obd lustre-OST004e"
to get a list of the files that could be lost. Once we have a usable copy of the OST, we intend to "e2fscl -fy" the snapshot to see of the opject come back or go to lost+found. If they go to lost+found, we're considering replicating the MDT and MGT in a
sandbox, mounting them and the OST and "lfsck"ing the OST to see if the MDT knows how to move the lost objects out of the lost+found to their happy places.</div>
<div><br>
</div>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Are there any other troubleshooting steps we can take while we wait for the OST block device to be copied (that'll take a bit) for our test e2fsck? Is there any output from the "tune2fs -l" or "e2fsck" that we can provide that could shed any light on the issue
and provide possible solutions? Any other tips and tricks? Thanks in advance for any insight.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Cheers</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Scott</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
</body>
</html>