[lustre-discuss] Mounting MDT causes kernel panic

[Gibbins, Faye]

Hi,

Our MSG/MDT machine went down and when we tried to recover it we experienced success on 3 out of 4 remounts of our MDTs.

However the last mount of the MDT called 'tools' causes a kernel panic, viz:

Snip----------
0 edi-vf-1-5:~# mount.lustre  /dev/tools_mdt/tools_mdt /opt/_lustre_mdt_tools
mount.lustre: increased /sys/block/dm-20/queue/max_sectors_kb from 512 to 16384
mount.lustre: increased /sys/block/emcpowerba/queue/max_sectors_kb from 512 to 16384

Message from syslogd at edi-vf-1-5 at Mar 26 15:43:06 ...
kernel:LustreError: 20088:0:(llog_osd.c:1075:llog_osd_prev_block()) ASSERTION( last_rec->lrh_index == tail->lrt_index ) failed:

Message from syslogd at edi-vf-1-5 at Mar 26 15:43:06 ...
kernel:LustreError: 20088:0:(llog_osd.c:1075:llog_osd_prev_block()) LBUG

Message from syslogd at edi-vf-1-5 at Mar 26 15:43:06 ...
kernel:Kernel panic - not syncing: LBUG
snip------------

Does anyone have any suggestions how we can recover from this?

We are running kernel: 3.10.0-327.3.1.el7_lustre.x86_64 and we are using lustre 2.8, here is the kernel message we get:

Snip---
[  262.676650] LDISKFS-fs (dm-20): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=
remount-ro,no_mbcache
[  263.261691] LustreError: 20088:0:(llog_osd.c:1075:llog_osd_prev_block()) ASSERTION( last_rec->lrh_index == tail->lrt_in
dex ) failed:
[  263.261774] LustreError: 20088:0:(llog_osd.c:1075:llog_osd_prev_block()) LBUG
[  263.261820] Pid: 20088, comm: mount.lustre
[  263.261821]
Call Trace:
[  263.261848]  [<ffffffffa08a87d3>] libcfs_debug_dumpstack+0x53/0x80 [libcfs]
[  263.261858]  [<ffffffffa08a8d75>] lbug_with_loc+0x45/0xc0 [libcfs]
[  263.261915]  [<ffffffffa09a9917>] llog_osd_prev_block+0x9f7/0xaf0 [obdclass]
[  263.261936]  [<ffffffffa099aee7>] llog_reverse_process+0x147/0xac0 [obdclass]
[  263.261958]  [<ffffffffa099e1f6>] ? llog_cat_id2handle+0x336/0x660 [obdclass]
[  263.261975]  [<ffffffffa11b66c0>] ? changelog_init_cb+0x0/0x1f0 [mdd]
[  263.261996]  [<ffffffffa099f917>] llog_cat_reverse_process_cb+0x157/0x540 [obdclass]
[  263.262016]  [<ffffffffa099b009>] llog_reverse_process+0x269/0xac0 [obdclass]
[  263.262035]  [<ffffffffa099f7c0>] ? llog_cat_reverse_process_cb+0x0/0x540 [obdclass]
[  263.262055]  [<ffffffffa099be39>] llog_cat_reverse_process+0x199/0x2d0 [obdclass]
[  263.262063]  [<ffffffffa11b66c0>] ? changelog_init_cb+0x0/0x1f0 [mdd]
[  263.262073]  [<ffffffffa11bd0c9>] mdd_prepare+0x1269/0x1a00 [mdd]
[  263.262103]  [<ffffffffa1079d01>] mdt_prepare+0x51/0x3b0 [mdt]
[  263.262144]  [<ffffffffa0a0a0c4>] server_start_targets+0x2574/0x2e10 [obdclass]
[  263.262172]  [<ffffffffa09d6290>] ? class_config_llog_handler+0x0/0x1a80 [obdclass]
[  263.262210]  [<ffffffffa0a0b9ed>] server_fill_super+0x108d/0x184c [obdclass]
[  263.262239]  [<ffffffffa09dfe48>] lustre_fill_super+0x328/0x950 [obdclass]
[  263.262263]  [<ffffffffa09dfb20>] ? lustre_fill_super+0x0/0x950 [obdclass]
[  263.262273]  [<ffffffff811e1ccd>] mount_nodev+0x4d/0xb0
[  263.262298]  [<ffffffffa09d8108>] lustre_mount+0x38/0x60 [obdclass]
[  263.262302]  [<ffffffff811e2679>] mount_fs+0x39/0x1b0
[  263.262309]  [<ffffffff811fdf1f>] vfs_kern_mount+0x5f/0xf0
[  263.262314]  [<ffffffff8120046e>] do_mount+0x24e/0xa40
[  263.262322]  [<ffffffff8116defe>] ? __get_free_pages+0xe/0x50
[  263.262326]  [<ffffffff81200cf6>] SyS_mount+0x96/0xf0
[  263.262334]  [<ffffffff81645b09>] system_call_fastpath+0x16/0x1b
[  263.262336]
[  263.262418] Kernel panic - not syncing: LBUG
[  263.262452] CPU: 0 PID: 20088 Comm: mount.lustre Tainted: P           OE  ------------   3.10.0-327.3.1.el7_lustre.x86_64 #1
Snip---

Yours
Faye Gibbins

Snr SysAdmin, Unix Lead Architect
Software Systems and Cloud Services
Cirrus Logic
0131 272 7398

This message and any attachments may contain privileged and confidential information that is intended solely for the person(s) to whom it is addressed. If you are not an intended recipient you must not: read; copy; distribute; discuss; take any action in or make any reliance upon the contents of this message; nor open or read any attachment. If you have received this message in error, please notify us as soon as possible on the following telephone number and destroy this message including any attachments. Thank you. Cirrus Logic International (UK) Ltd and Cirrus Logic International Semiconductor Ltd are companies registered in Scotland, with registered numbers SC089839 and SC495735 respectively. Our registered office is at 7B Nightingale Way, Quartermile, Edinburgh, EH3 9EG, UK. Tel: +44 (0)131 272 7000. cirrus.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170326/880ae134/attachment.htm>