<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Our MSG/MDT machine went down and when we tried to recover it we experienced success on 3 out of 4 remounts of our MDTs.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">However the last mount of the MDT called ‘tools’ causes a kernel panic, viz:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Snip----------<o:p></o:p></p>
<p class="MsoNormal">0 edi-vf-1-5:~# mount.lustre /dev/tools_mdt/tools_mdt /opt/_lustre_mdt_tools<o:p></o:p></p>
<p class="MsoNormal">mount.lustre: increased /sys/block/dm-20/queue/max_sectors_kb from 512 to 16384<o:p></o:p></p>
<p class="MsoNormal">mount.lustre: increased /sys/block/emcpowerba/queue/max_sectors_kb from 512 to 16384<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Message from syslogd@edi-vf-1-5 at Mar 26 15:43:06 ...<o:p></o:p></p>
<p class="MsoNormal">kernel:LustreError: 20088:0:(llog_osd.c:1075:llog_osd_prev_block()) ASSERTION( last_rec->lrh_index == tail->lrt_index ) failed:
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Message from syslogd@edi-vf-1-5 at Mar 26 15:43:06 ...<o:p></o:p></p>
<p class="MsoNormal">kernel:LustreError: 20088:0:(llog_osd.c:1075:llog_osd_prev_block()) LBUG<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Message from syslogd@edi-vf-1-5 at Mar 26 15:43:06 ...<o:p></o:p></p>
<p class="MsoNormal">kernel:Kernel panic - not syncing: LBUG<o:p></o:p></p>
<p class="MsoNormal">snip------------<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Does anyone have any suggestions how we can recover from this?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are running kernel: 3.10.0-327.3.1.el7_lustre.x86_64 and we are using lustre 2.8, here is the kernel message we get:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Snip---<o:p></o:p></p>
<p class="MsoNormal">[ 262.676650] LDISKFS-fs (dm-20): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=<o:p></o:p></p>
<p class="MsoNormal">remount-ro,no_mbcache<o:p></o:p></p>
<p class="MsoNormal">[ 263.261691] LustreError: 20088:0:(llog_osd.c:1075:llog_osd_prev_block()) ASSERTION( last_rec->lrh_index == tail->lrt_in<o:p></o:p></p>
<p class="MsoNormal">dex ) failed: <o:p></o:p></p>
<p class="MsoNormal">[ 263.261774] LustreError: 20088:0:(llog_osd.c:1075:llog_osd_prev_block()) LBUG<o:p></o:p></p>
<p class="MsoNormal">[ 263.261820] Pid: 20088, comm: mount.lustre<o:p></o:p></p>
<p class="MsoNormal">[ 263.261821] <o:p></o:p></p>
<p class="MsoNormal">Call Trace:<o:p></o:p></p>
<p class="MsoNormal">[ 263.261848] [<ffffffffa08a87d3>] libcfs_debug_dumpstack+0x53/0x80 [libcfs]<o:p></o:p></p>
<p class="MsoNormal">[ 263.261858] [<ffffffffa08a8d75>] lbug_with_loc+0x45/0xc0 [libcfs]<o:p></o:p></p>
<p class="MsoNormal">[ 263.261915] [<ffffffffa09a9917>] llog_osd_prev_block+0x9f7/0xaf0 [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.261936] [<ffffffffa099aee7>] llog_reverse_process+0x147/0xac0 [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.261958] [<ffffffffa099e1f6>] ? llog_cat_id2handle+0x336/0x660 [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.261975] [<ffffffffa11b66c0>] ? changelog_init_cb+0x0/0x1f0 [mdd]<o:p></o:p></p>
<p class="MsoNormal">[ 263.261996] [<ffffffffa099f917>] llog_cat_reverse_process_cb+0x157/0x540 [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262016] [<ffffffffa099b009>] llog_reverse_process+0x269/0xac0 [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262035] [<ffffffffa099f7c0>] ? llog_cat_reverse_process_cb+0x0/0x540 [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262055] [<ffffffffa099be39>] llog_cat_reverse_process+0x199/0x2d0 [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262063] [<ffffffffa11b66c0>] ? changelog_init_cb+0x0/0x1f0 [mdd]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262073] [<ffffffffa11bd0c9>] mdd_prepare+0x1269/0x1a00 [mdd]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262103] [<ffffffffa1079d01>] mdt_prepare+0x51/0x3b0 [mdt]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262144] [<ffffffffa0a0a0c4>] server_start_targets+0x2574/0x2e10 [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262172] [<ffffffffa09d6290>] ? class_config_llog_handler+0x0/0x1a80 [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262210] [<ffffffffa0a0b9ed>] server_fill_super+0x108d/0x184c [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262239] [<ffffffffa09dfe48>] lustre_fill_super+0x328/0x950 [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262263] [<ffffffffa09dfb20>] ? lustre_fill_super+0x0/0x950 [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262273] [<ffffffff811e1ccd>] mount_nodev+0x4d/0xb0<o:p></o:p></p>
<p class="MsoNormal">[ 263.262298] [<ffffffffa09d8108>] lustre_mount+0x38/0x60 [obdclass]<o:p></o:p></p>
<p class="MsoNormal">[ 263.262302] [<ffffffff811e2679>] mount_fs+0x39/0x1b0<o:p></o:p></p>
<p class="MsoNormal">[ 263.262309] [<ffffffff811fdf1f>] vfs_kern_mount+0x5f/0xf0<o:p></o:p></p>
<p class="MsoNormal">[ 263.262314] [<ffffffff8120046e>] do_mount+0x24e/0xa40<o:p></o:p></p>
<p class="MsoNormal">[ 263.262322] [<ffffffff8116defe>] ? __get_free_pages+0xe/0x50<o:p></o:p></p>
<p class="MsoNormal">[ 263.262326] [<ffffffff81200cf6>] SyS_mount+0x96/0xf0<o:p></o:p></p>
<p class="MsoNormal">[ 263.262334] [<ffffffff81645b09>] system_call_fastpath+0x16/0x1b<o:p></o:p></p>
<p class="MsoNormal">[ 263.262336] <o:p></o:p></p>
<p class="MsoNormal">[ 263.262418] Kernel panic - not syncing: LBUG<o:p></o:p></p>
<p class="MsoNormal">[ 263.262452] CPU: 0 PID: 20088 Comm: mount.lustre Tainted: P OE ------------ 3.10.0-327.3.1.el7_lustre.x86_64 #1<o:p></o:p></p>
<p class="MsoNormal">Snip---<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">Yours<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">Faye Gibbins<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#9CC2E5;mso-fareast-language:EN-GB">Snr SysAdmin, Unix Lead Architect<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#9CC2E5;mso-fareast-language:EN-GB">Software Systems and Cloud Services<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#9CC2E5;mso-fareast-language:EN-GB">Cirrus Logic<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#9CC2E5;mso-fareast-language:EN-GB">0131 272 7398<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
This message and any attachments may contain privileged and confidential information that is intended solely for the person(s) to whom it is addressed. If you are not an intended recipient you must not: read; copy; distribute; discuss; take any action in or
make any reliance upon the contents of this message; nor open or read any attachment. If you have received this message in error, please notify us as soon as possible on the following telephone number and destroy this message including any attachments. Thank
you. Cirrus Logic International (UK) Ltd and Cirrus Logic International Semiconductor Ltd are companies registered in Scotland, with registered numbers SC089839 and SC495735 respectively. Our registered office is at 7B Nightingale Way, Quartermile, Edinburgh,
EH3 9EG, UK. Tel: +44 (0)131 272 7000. cirrus.com
</body>
</html>