<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Hi,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">My lustre environment is: 2.6.9-55.0.9.EL_lustre.1.6.3smp</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">One of my OSS's crashed today. Below you can see messages sent by it (storage09) to the syslog (first three lines). Then it died (my guess is with kernel panic) and heartbeat software STONITH that OSS's.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Nov 9 19:08:44 storage09.beowulf.cluster kernel: LDISKFS-fs error (device dm-5): mb_free_blocks: double-free of inode 38887437's block 155560192(bit 10496 in group 4747)</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Nov 9 19:08:44 storage09.beowulf.cluster kernel: Nov 9 19:08:44 storage09.beowulf.cluster kernel: Remounting filesystem read-only</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Nov 9 19:08:44 storage09.beowulf.cluster kernel: LDISKFS-fs error (device dm-5): mb_free_blocks: double-free of inode 38887437's block 155560193(bit 10497 in group 4747)</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Nov 9 19:09:13 storage10.beowulf.cluster heartbeat: [21231]: WARN: node storage09: is dead Nov 9 19:09:13 storage10.beowulf.cluster heartbeat: [21231]: info: Link storage09:eth0 dead.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Nov 9 19:09:13 storage10.beowulf.cluster heartbeat: [21231]: info: Link storage09:eth2 dead. Nov 9 19:09:13 storage10.beowulf.cluster heartbeat: [32414]: info: Resetting node storage09 with [external/ipmi ]</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Do you know how serious are LDISKFS-fs errors? Is that indicates data corruption on the certain block device? Device dm-5 is a DDN LUN. DDN controller S2A9500 says that everything is Healthy there.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Cheers</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Wojciech Turek</div> <br><br><div> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 11.0px Lucida Grande">Mr Wojciech Turek</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Lucida Grande; min-height: 13px; ">Assistant System Manager</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 11.0px Lucida Grande">University of Cambridge</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 11.0px Lucida Grande">High Performance Computing service </font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="'Lucida Grande'" size="3"><span class="Apple-style-span" style="font-size: 11px;">email: <a href="mailto:wjt27@cam.ac.uk">wjt27@cam.ac.uk</a></span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="3" style="font: 11.0px Lucida Grande">tel. +441223763517</font></div></div><div><br class="webkit-block-placeholder"></div></span><br class="Apple-interchange-newline"></span></span> </div><br></body></html>