<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi,<div><br></div><div>I repeatedly encounter "invalid magic" in one particular inode of one of my OSS volumes (1 of 4, each 5 TB), with the consequence of lustre remounting R/O.</div><div><br></div><div>I run 2.6.18-53.1.13.el5_lustre.1.6.4.3smp on RHEL5.1 on a cluster with approx. 150 client nodes.</div><div><br></div><div>The error appears on the OSS as:</div><div><div><br></div><div>Jul 1 15:43:58 oss01 kernel: LDISKFS-fs error (device dm-3): ldiskfs_ext_find_extent: bad header in inode #405012501: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)</div> <div>Jul 1 15:43:58 oss01 kernel: Remounting filesystem read-only</div> <div>Jul 1 15:43:58 oss01 kernel: LDISKFS-fs error (device dm-3): ldiskfs_ext_find_extent: bad header in inode #405012501: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)</div> <div>Jul 1 15:43:58 oss01 kernel: LustreError: 25462:0:(fsfilt-ldiskfs.c:417:fsfilt_ldiskfs_brw_start()) can't get handle for 45 credits: rc = -30</div> <div>Jul 1 15:43:58 oss01 kernel: LustreError: 25462:0:(fsfilt-ldiskfs.c:417:fsfilt_ldiskfs_brw_start()) Skipped 6 previous similar messages</div> <div>Jul 1 15:43:58 oss01 kernel: LustreError: 25462:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30</div> <div>Jul 1 15:43:58 oss01 kernel: LustreError: 19569:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30</div> <div>[... many repeats]</div><div><br></div><div><br></div><div>Three login nodes signaled, about 10 .. 15 minutes apart the same wall(8) message:</div><div><br></div><div><pre><font class="Apple-style-span" face="Monaco" size="3"><span class="Apple-style-span" style="font-size: 13px; white-space: normal;">Message from syslogd@ at Tue Jul 1 16:00:02 2008 ...</span></font></pre><pre><font class="Apple-style-span" face="Monaco" size="3"><span class="Apple-style-span" style="font-size: 13px; white-space: normal;">login1 kernel: LustreError: 5612:0:(ptlrpcd.c:72:ptlrpcd_wake()) ASSERTION(pc != NULL) failed</span></font></pre><pre><font class="Apple-style-span" face="Monaco" size="3"><span class="Apple-style-span" style="font-size: 13px; white-space: normal;">Message from syslogd@ at Tue Jul 1 16:00:02 2008 ...</span></font></pre><pre><font class="Apple-style-span" face="Monaco" size="3"><span class="Apple-style-span" style="font-size: 13px; white-space: normal;">login1 kernel: LustreError: 5612:0:(tracefile.c:431:libcfs_assertion_failed()) LBUG</span></font>
</pre><br></div><div><br></div><div><br></div><div>Twice in the past, I followed this recovery procedure from the Manual and the Wiki:</div><div><br></div><div> <a href="http://wiki.lustre.org/index.php?title=Fsck_Support#Using_e2fsck_on_a_backing_filesystem%7Cusing">http://wiki.lustre.org/index.php?title=Fsck_Support#Using_e2fsck_on_a_backing_filesystem%7Cusing</a></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Using e2fsck on a backing filesystem</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>-- nice walk-through<br></div><div><br></div><div> <a href="http://manual.lustre.org/manual/LustreManual16_HTML/Failover.html#50446391_pgfId-1287654">http://manual.lustre.org/manual/LustreManual16_HTML/Failover.html#50446391_pgfId-1287654</a></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>8.4.1 Starting/Stopping a Resource</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>[i.e., simply umounting the device on the OSS - is this correct?]<br></div><div><br></div><div> <a href="http://manual.lustre.org/manual/LustreManual16_HTML/LustreInstallation.html#50446385_43530">http://manual.lustre.org/manual/LustreManual16_HTML/LustreInstallation.html#50446385_43530</a></div><div><div><span class="Apple-tab-span" style="white-space:pre"> </span>4.2.1.5 Stopping a Server</div><div><br></div></div><div><br></div><div>In other words:</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>umount the OSS</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>perform fsck on the block device</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>remount the OSS</div><div><br></div><div>So, last time I did:</div><div><br></div><div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>[root@oss01 ~]# umount /mnt/ost2</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>[root@oss01 ~]# e2fsck -fp /dev/dm-3</div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>lustre-OST0002: recovering journal</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>lustre-OST0002: ext3 recovery flag is clear, but journal has data.</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>lustre-OST0002: Run journal anyway</div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>lustre-OST0002: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.</div><div> <span class="Apple-tab-span" style="white-space: pre; "> </span>(i.e., without -a or -p options)</div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>[root@oss01 ~]# mount -t ldiskfs /dev/dm-3 /mnt/ost2</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>[root@oss01 ~]# umount /mnt/ost2</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>[root@oss01 ~]# e2fsck -fp /dev/dm-3</div><div><span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-style-span" style="font-family: -webkit-monospace; font-size: 12px; line-height: 13px; white-space: pre; ">lustre-OST0002: 342355/427253760 files (4.2% non-contiguous), 139324997/1708984375 blocks</span></div><div><font class="Apple-style-span" face="-webkit-monospace" size="3"><span class="Apple-style-span" style="font-size: 12px; line-height: 13px; white-space: pre;"><br></span></font></div><div><font class="Apple-style-span" face="-webkit-monospace" size="3"><span class="Apple-style-span" style="font-size: 12px; line-height: 13px; white-space: pre;"><br></span></font></div><div>To my surprise, there were no errors. I did the same today after the error above, but left out the "-p" flag; still, fsck did not find an error (except the journal replay??):</div></div><div><br></div><div><div><span class="Apple-tab-span" style="white-space:pre"> </span>[root@oss01 ~]# e2fsck -f /dev/dm-3</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>e2fsck 1.40.4.cfs1 (31-Dec-2007)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>lustre-OST0002: recovering journal</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Pass 1: Checking inodes, blocks, and sizes</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Pass 2: Checking directory structure</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Pass 3: Checking directory connectivity</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Pass 4: Checking reference counts</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Pass 5: Checking group summary information</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>lustre-OST0002: ***** FILE SYSTEM WAS MODIFIED *****</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>lustre-OST0002: 343702/427253760 files (4.4% non-contiguous), 137003893/1708984375 blocks</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>[root@oss01 ~]# </div><div><br></div><div>I haven't mounted back yet for fear this would stall the system again in a couple of days.</div><div><br></div><div><br></div></div><div>How can I locate the "bad" inode - should I try? Is this an inode of the lustre FS or the underlying ext3 on the OST?</div><div><br></div><div>Are there version dependencies of e2fsck with lustre? I am running lustre-1.6.4.3 and e2fsck-1.40.4.</div><div><br></div><div><br></div><div>I would appreciate any pointers.</div><div><br></div><div><br></div><div>Thank you for your attention and help.</div><div>Michael</div><div><br></div></div></body></html>