<div dir="auto"><div dir="ltr">Hi, <div><br></div><div>Since last monday we have a strange problem on one of our Lustre file-systems, the MDT started to crash with resulted in Kernel panics.</div><div><br></div><div>The configuration is as follows"</div><div>2 metadata servers, one providing the MGS the other the MDT, the metadata servers  are configured in a HA setup with pacemaker.</div><div>18 data servers (oss'es), each if providing 6 OST's, each time 2 OSS's are configured in a HA setup with pacemaker.<br clear="all"><div><br></div><div>We have configured 59 clients to this setup, 50 'normal'-compute nodes 2 head nodes, 4 GPU-nodes, 2 nodes for ingest (data transport to other sites)1 node for Robinhood</div><div>On the OSS'es en metadata servers are runnig lustre 2.7, all are based on ext4</div><div><br></div><div>We did e2fsck on all the volumes, and after that an lfsck, the following command was used for the lfsck:<br><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">lctl lfsck_start -M cep4-fs-MDT0000 -A -t all -r</span><br></span><br></div><div>This all did not brought us back to business, after mount the clients the MDT crashed again.</div><div>We removed the changelog from the metadata and we are able to use the filesystem.</div><div>When we enabled the changelog again on the MDT there was almost an instant crash of the MDT</div><div>At this moment the process for which this storage cluster is in use depends on Robinhood and without the Changelog Robinhood doesn't work.</div><div><br></div><div>The console log provided us with the following call traces:<br><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">[-- MARK -- Mon Jul 23 10:00:00 2018]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Lustre: 12777:0:(osd_internal.h:1014:osd_trans_exec_op()) cep4-fs-MDT0000-osd: Overflow in tracking declares for index, rb = 4</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Pid: 12777, comm: mdt00_010</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Call Trace:</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0502895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0e13d88>] osd_trans_exec_op+0x1f8/0x2e0 [osd_ldiskfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0e28b08>] osd_object_ea_create+0x198/0x8c0 [osd_ldiskfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa063a2eb>] local_object_create+0xdb/0x430 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa061dac2>] llog_osd_create+0x3d2/0x800 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa060c821>] llog_create+0x81/0x1e0 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0613dd2>] llog_cat_new_log+0xe2/0x710 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa061450a>] llog_cat_add_rec+0x10a/0x450 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa060c1e9>] llog_add+0x89/0x1c0 [obdclass]</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div> [<ffffffffa10d94e2>] mdd_changelog_store+0x122/0x290 [mdd]</div><div> [<ffffffffa10ecd0c>] mdd_changelog_data_store+0x16c/0x320 [mdd]</div></span><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa10f0a86>] mdd_xattr_del+0x386/0x3d0 [mdd]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa10f3c40>] mdd_xattr_set+0x3c0/0xe40 [mdd]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0870a34>] ? lustre_msg_get_versions+0xa4/0x120 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0fa804c>] ? mdt_version_save+0x8c/0x1a0 [mdt]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0fb2615>] mdt_reint_setxattr+0x975/0x1810 [mdt]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa051f83c>] ? upcall_cache_get_entry+0x29c/0x880 [libcfs]</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div> [<ffffffffa0fa70cd>] mdt_reint_rec+0x5d/0x200 [mdt]</div><div> [<ffffffffa0f8b23b>] mdt_reint_internal+0x4cb/0x7a0 [mdt]</div><div> [<ffffffffa0f8b9ab>] mdt_reint+0x6b/0x120 [mdt]</div></span><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa08d056e>] tgt_request_handle+0x8be/0x1000 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa08805a1>] ptlrpc_main+0xe41/0x1960 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa087f760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div> [<ffffffff8109e66e>] kthread+0x9e/0xc0</div><div> [<ffffffff8100c20a>] child_rip+0xa/0x20</div><div> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0</div><div> [<ffffffff8100c200>] ? child_rip+0x0/0x20</div><div><br></div></span><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Lustre: 154617:0:(osd_internal.h:1014:osd_trans_exec_op()) cep4-fs-MDT0000-osd: Overflow in tracking declares for index, rb = 4</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Pid: 154617, comm: mdt00_035</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Call Trace:</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0502895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0e13d88>] osd_trans_exec_op+0x1f8/0x2e0 [osd_ldiskfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0e28b08>] osd_object_ea_create+0x198/0x8c0 [osd_ldiskfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa063a2eb>] local_object_create+0xdb/0x430 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa061dac2>] llog_osd_create+0x3d2/0x800 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa060c821>] llog_create+0x81/0x1e0 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0613dd2>] llog_cat_new_log+0xe2/0x710 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa061450a>] llog_cat_add_rec+0x10a/0x450 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa060c1e9>] llog_add+0x89/0x1c0 [obdclass]</div><span class="m_-9197701292137868598gmail-im" style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div style="color:rgb(80,0,80);font-size:small"> [<ffffffffa10d94e2>] mdd_changelog_store+0x122/0x290 [mdd]</div><div><font color="#500050"> [<ffffffffa10ecd0c>] mdd_changelog_data_store+</font><font color="#500050">0x16c/0x320 [mdd][-- MARK -- Mon Jul 23 10:00:00 2018]</font></div></span><div><font color="#500050">Lustre: 12777:0:(osd_internal.h:1014:osd_trans_exec_op()) cep4-fs-MDT0000-osd: Overflow in tracking declares for index, rb = 4</font></div><div><font color="#500050">Pid: 12777, comm: mdt00_010</font></div><div><font color="#500050"><br></font></div><div><font color="#500050">Call Trace:</font></div><div><font color="#500050"> [<ffffffffa0502895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]</font></div><div><font color="#500050"> [<ffffffffa0e13d88>] osd_trans_exec_op+0x1f8/0x2e0 [osd_ldiskfs]</font></div><div><font color="#500050"> [<ffffffffa0e28b08>] osd_object_ea_create+0x198/0x8c0 [osd_ldiskfs]</font></div><div><font color="#500050"> [<ffffffffa063a2eb>] local_object_create+0xdb/0x430 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa061dac2>] llog_osd_create+0x3d2/0x800 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa060c821>] llog_create+0x81/0x1e0 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa0613dd2>] llog_cat_new_log+0xe2/0x710 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa061450a>] llog_cat_add_rec+0x10a/0x450 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa060c1e9>] llog_add+0x89/0x1c0 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa10d94e2>] mdd_changelog_store+0x122/0x290 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10ecd0c>] mdd_changelog_data_store+0x16c/0x320 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10f0a86>] mdd_xattr_del+0x386/0x3d0 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10f3c40>] mdd_xattr_set+0x3c0/0xe40 [mdd]</font></div><div><font color="#500050"> [<ffffffffa0870a34>] ? lustre_msg_get_versions+0xa4/0x120 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa0fa804c>] ? mdt_version_save+0x8c/0x1a0 [mdt]</font></div><div><font color="#500050"> [<ffffffffa0fb2615>] mdt_reint_setxattr+0x975/0x1810 [mdt]</font></div><div><font color="#500050"> [<ffffffffa051f83c>] ? upcall_cache_get_entry+0x29c/0x880 [libcfs]</font></div><div><font color="#500050"> [<ffffffffa0fa70cd>] mdt_reint_rec+0x5d/0x200 [mdt]</font></div><div><font color="#500050"> [<ffffffffa0f8b23b>] mdt_reint_internal+0x4cb/0x7a0 [mdt]</font></div><div><font color="#500050"> [<ffffffffa0f8b9ab>] mdt_reint+0x6b/0x120 [mdt]</font></div><div><font color="#500050"> [<ffffffffa08d056e>] tgt_request_handle+0x8be/0x1000 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa08805a1>] ptlrpc_main+0xe41/0x1960 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa087f760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffff8109e66e>] kthread+0x9e/0xc0</font></div><div><font color="#500050"> [<ffffffff8100c20a>] child_rip+0xa/0x20</font></div><div><font color="#500050"> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0</font></div><div><font color="#500050"> [<ffffffff8100c200>] ? child_rip+0x0/0x20</font></div><div><font color="#500050"><br></font></div><div><font color="#500050">Lustre: 154617:0:(osd_internal.h:1014:osd_trans_exec_op()) cep4-fs-MDT0000-osd: Overflow in tracking declares for index, rb = 4</font></div><div><font color="#500050">Pid: 154617, comm: mdt00_035</font></div><div><font color="#500050"><br></font></div><div><font color="#500050">Call Trace:</font></div><div><font color="#500050"> [<ffffffffa0502895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]</font></div><div><font color="#500050"> [<ffffffffa0e13d88>] osd_trans_exec_op+0x1f8/0x2e0 [osd_ldiskfs]</font></div><div><font color="#500050"> [<ffffffffa0e28b08>] osd_object_ea_create+0x198/0x8c0 [osd_ldiskfs]</font></div><div><font color="#500050"> [<ffffffffa063a2eb>] local_object_create+0xdb/0x430 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa061dac2>] llog_osd_create+0x3d2/0x800 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa060c821>] llog_create+0x81/0x1e0 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa0613dd2>] llog_cat_new_log+0xe2/0x710 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa061450a>] llog_cat_add_rec+0x10a/0x450 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa060c1e9>] llog_add+0x89/0x1c0 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa10d94e2>] mdd_changelog_store+0x122/0x290 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10ecd0c>] mdd_changelog_data_store+0x16c/0x320 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10f0a86>] mdd_xattr_del+0x386/0x3d0 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10f3c40>] mdd_xattr_set+0x3c0/0xe40 [mdd]</font></div><div><font color="#500050"> [<ffffffffa0870a34>] ? lustre_msg_get_versions+0xa4/0x120 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa0fa804c>] ? mdt_version_save+0x8c/0x1a0 [mdt]</font></div><div><font color="#500050"> [<ffffffffa0fb2615>] mdt_reint_setxattr+0x975/0x1810 [mdt]</font></div><div><font color="#500050"> [<ffffffffa051f83c>] ? upcall_cache_get_entry+0x29c/0x880 [libcfs]</font></div><div><font color="#500050"> [<ffffffffa0fa70cd>] mdt_reint_rec+0x5d/0x200 [mdt]</font></div><div><font color="#500050"> [<ffffffffa0f8b23b>] mdt_reint_internal+0x4cb/0x7a0 [mdt]</font></div><div><font color="#500050"> [<ffffffffa0f8b9ab>] mdt_reint+0x6b/0x120 [mdt]</font></div><div><font color="#500050"> [<ffffffffa08d056e>] tgt_request_handle+0x8be/0x1000 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa08805a1>] ptlrpc_main+0xe41/0x1960 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa087f760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffff8109e66e>] kthread+0x9e/0xc0</font></div><div><font color="#500050"> [<ffffffff8100c20a>] child_rip+0xa/0x20</font></div><div><font color="#500050"> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0</font></div><div><font color="#500050"> [<ffffffff8100c200>] ? child_rip+0x0/0x20</font></div><div><font color="#500050"><br></font></div><div><font color="#500050">Lustre: 12813:0:(osd_internal.h:1014:osd_trans_exec_op()) cep4-fs-MDT0000-osd: Overflow in tracking declares for index, rb = 4</font></div><div><font color="#500050">Pid: 12813, comm: mdt00_020</font></div><div><font color="#500050"><br></font></div><div><font color="#500050">Call Trace:</font></div><div><font color="#500050"> [<ffffffffa0502895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]</font></div><div><font color="#500050"> [<ffffffffa0e13d88>] osd_trans_exec_op+0x1f8/0x2e0 [osd_ldiskfs]</font></div><div><font color="#500050"> [<ffffffffa0e28b08>] osd_object_ea_create+0x198/0x8c0 [osd_ldiskfs]</font></div><div><font color="#500050"> [<ffffffffa063a2eb>] local_object_create+0xdb/0x430 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa061dac2>] llog_osd_create+0x3d2/0x800 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa060c821>] llog_create+0x81/0x1e0 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa0613dd2>] llog_cat_new_log+0xe2/0x710 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa061450a>] llog_cat_add_rec+0x10a/0x450 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa060c1e9>] llog_add+0x89/0x1c0 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa10d94e2>] mdd_changelog_store+0x122/0x290 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10ecd0c>] mdd_changelog_data_store+0x16c/0x320 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10f0a86>] mdd_xattr_del+0x386/0x3d0 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10f3c40>] mdd_xattr_set+0x3c0/0xe40 [mdd]</font></div><div><font color="#500050"> [<ffffffffa0870a34>] ? lustre_msg_get_versions+0xa4/0x120 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa0fa804c>] ? mdt_version_save+0x8c/0x1a0 [mdt]</font></div><div><font color="#500050"> [<ffffffffa0fb2615>] mdt_reint_setxattr+0x975/0x1810 [mdt]</font></div><div><font color="#500050"> [<ffffffffa051f83c>] ? upcall_cache_get_entry+0x29c/0x880 [libcfs]</font></div><div><font color="#500050"> [<ffffffffa0fa70cd>] mdt_reint_rec+0x5d/0x200 [mdt]</font></div><div><font color="#500050"> [<ffffffffa0f8b23b>] mdt_reint_internal+0x4cb/0x7a0 [mdt]</font></div><div><font color="#500050"> [<ffffffffa0f8b9ab>] mdt_reint+0x6b/0x120 [mdt]</font></div><div><font color="#500050"> [<ffffffffa08d056e>] tgt_request_handle+0x8be/0x1000 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa08805a1>] ptlrpc_main+0xe41/0x1960 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa087f760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffff8109e66e>] kthread+0x9e/0xc0</font></div><div><font color="#500050"> [<ffffffff8100c20a>] child_rip+0xa/0x20</font></div><div><font color="#500050"> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0</font></div><div><font color="#500050"> [<ffffffff8100c200>] ? child_rip+0x0/0x20</font></div><div><font color="#500050"><br></font></div><div><font color="#500050">LustreError: 15586:0:(osd_handler.c:1017:osd_trans_start()) ASSERTION( get_current()->journal_info == ((void *)0) ) failed: </font></div><div><font color="#500050">LustreError: 15586:0:(osd_handler.c:1017:osd_trans_start()) LBUG</font></div><div><font color="#500050">Pid: 15586, comm: mdt_rdpg00_004</font></div><div><font color="#500050"><br></font></div><div><font color="#500050">Call Trace:</font></div><div><font color="#500050"> [<ffffffffa0502895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]</font></div><div><font color="#500050"> [<ffffffffa0502e97>] lbug_with_loc+0x47/0xb0 [libcfs]</font></div><div><font color="#500050"> [<ffffffffa0e1524d>] osd_trans_start+0x25d/0x660 [osd_ldiskfs]</font></div><div><font color="#500050"> [<ffffffffa061ab4a>] llog_osd_destroy+0x42a/0xd40 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa0613edc>] llog_cat_new_log+0x1ec/0x710 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa061450a>] llog_cat_add_rec+0x10a/0x450 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa060c1e9>] llog_add+0x89/0x1c0 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa0654fdf>] ? keys_fill+0x6f/0x190 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa10d94e2>] mdd_changelog_store+0x122/0x290 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10ecd0c>] mdd_changelog_data_store+0x16c/0x320 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10f18ee>] mdd_close+0x34e/0xc50 [mdd]</font></div><div><font color="#500050"> [<ffffffffa0fba801>] mdt_mfd_close+0x3f1/0xac0 [mdt]</font></div><div><font color="#500050"> [<ffffffffa0636905>] ? class_handle2object+0x95/0x190 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa0fbc313>] mdt_close+0x6f3/0xaa0 [mdt]</font></div><div><font color="#500050"> [<ffffffffa08d056e>] tgt_request_handle+0x8be/0x1000 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa08805a1>] ptlrpc_main+0xe41/0x1960 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa087f760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffff8109e66e>] kthread+0x9e/0xc0</font></div><div><font color="#500050"> [<ffffffff8100c20a>] child_rip+0xa/0x20</font></div><div><font color="#500050"> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0</font></div><div><font color="#500050"> [<ffffffff8100c200>] ? child_rip+0x0/0x20</font></div><div><font color="#500050"><br></font></div><div><font color="#500050">Kernel panic - not syncing: LBUG</font></div><div><font color="#500050">Pid: 15586, comm: mdt_rdpg00_004 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1</font></div><div><font color="#500050">Call Trace:</font></div><div><font color="#500050"> [<ffffffff81529b76>] ? panic+0xa7/0x16f</font></div><div><font color="#500050"> [<ffffffffa0502eeb>] ? lbug_with_loc+0x9b/0xb0 [libcfs]</font></div><div><font color="#500050"> [<ffffffffa0e1524d>] ? osd_trans_start+0x25d/0x660 [osd_ldiskfs]</font></div><div><font color="#500050"> [<ffffffffa061ab4a>] ? llog_osd_destroy+0x42a/0xd40 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa0613edc>] ? llog_cat_new_log+0x1ec/0x710 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa061450a>] ? llog_cat_add_rec+0x10a/0x450 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa060c1e9>] ? llog_add+0x89/0x1c0 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa0654fdf>] ? keys_fill+0x6f/0x190 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa10d94e2>] ? mdd_changelog_store+0x122/0x290 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10ecd0c>] ? mdd_changelog_data_store+0x16c/0x320 [mdd]</font></div><div><font color="#500050"> [<ffffffffa10f18ee>] ? mdd_close+0x34e/0xc50 [mdd]</font></div><div><font color="#500050"> [<ffffffffa0fba801>] ? mdt_mfd_close+0x3f1/0xac0 [mdt]</font></div><div><font color="#500050"> [<ffffffffa0636905>] ? class_handle2object+0x95/0x190 [obdclass]</font></div><div><font color="#500050"> [<ffffffffa0fbc313>] ? mdt_close+0x6f3/0xaa0 [mdt]</font></div><div><font color="#500050"> [<ffffffffa08d056e>] ? tgt_request_handle+0x8be/0x1000 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa08805a1>] ? ptlrpc_main+0xe41/0x1960 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffffa087f760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]</font></div><div><font color="#500050"> [<ffffffff8109e66e>] ? kthread+0x9e/0xc0</font></div><div><font color="#500050"> [<ffffffff8100c20a>] ? child_rip+0xa/0x20</font></div><div><font color="#500050"> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0</font></div><div><font color="#500050"> [<ffffffff8100c200>] ? child_rip+0x0/0x20</font></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa10f0a86>] mdd_xattr_del+0x386/0x3d0 [mdd]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa10f3c40>] mdd_xattr_set+0x3c0/0xe40 [mdd]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0870a34>] ? lustre_msg_get_versions+0xa4/0x120 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0fa804c>] ? mdt_version_save+0x8c/0x1a0 [mdt]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0fb2615>] mdt_reint_setxattr+0x975/0x1810 [mdt]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa051f83c>] ? upcall_cache_get_entry+0x29c/0x880 [libcfs]</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div> [<ffffffffa0fa70cd>] mdt_reint_rec+0x5d/0x200 [mdt]</div><div> [<ffffffffa0f8b23b>] mdt_reint_internal+0x4cb/0x7a0 [mdt]</div><div> [<ffffffffa0f8b9ab>] mdt_reint+0x6b/0x120 [mdt]</div></span><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa08d056e>] tgt_request_handle+0x8be/0x1000 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa08805a1>] ptlrpc_main+0xe41/0x1960 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa087f760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div> [<ffffffff8109e66e>] kthread+0x9e/0xc0</div><div> [<ffffffff8100c20a>] child_rip+0xa/0x20</div><div> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0</div><div> [<ffffffff8100c200>] ? child_rip+0x0/0x20</div><div><br></div></span><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Lustre: 12813:0:(osd_internal.h:1014:osd_trans_exec_op()) cep4-fs-MDT0000-osd: Overflow in tracking declares for index, rb = 4</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Pid: 12813, comm: mdt00_020</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Call Trace:</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0502895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0e13d88>] osd_trans_exec_op+0x1f8/0x2e0 [osd_ldiskfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0e28b08>] osd_object_ea_create+0x198/0x8c0 [osd_ldiskfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa063a2eb>] local_object_create+0xdb/0x430 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa061dac2>] llog_osd_create+0x3d2/0x800 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa060c821>] llog_create+0x81/0x1e0 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0613dd2>] llog_cat_new_log+0xe2/0x710 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa061450a>] llog_cat_add_rec+0x10a/0x450 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa060c1e9>] llog_add+0x89/0x1c0 [obdclass]</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div> [<ffffffffa10d94e2>] mdd_changelog_store+0x122/0x290 [mdd]</div><div> [<ffffffffa10ecd0c>] mdd_changelog_data_store+0x16c/0x320 [mdd]</div></span><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa10f0a86>] mdd_xattr_del+0x386/0x3d0 [mdd]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa10f3c40>] mdd_xattr_set+0x3c0/0xe40 [mdd]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0870a34>] ? lustre_msg_get_versions+0xa4/0x120 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0fa804c>] ? mdt_version_save+0x8c/0x1a0 [mdt]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0fb2615>] mdt_reint_setxattr+0x975/0x1810 [mdt]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa051f83c>] ? upcall_cache_get_entry+0x29c/0x880 [libcfs]</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div> [<ffffffffa0fa70cd>] mdt_reint_rec+0x5d/0x200 [mdt]</div><div> [<ffffffffa0f8b23b>] mdt_reint_internal+0x4cb/0x7a0 [mdt]</div><div> [<ffffffffa0f8b9ab>] mdt_reint+0x6b/0x120 [mdt]</div></span><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa08d056e>] tgt_request_handle+0x8be/0x1000 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa08805a1>] ptlrpc_main+0xe41/0x1960 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa087f760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div> [<ffffffff8109e66e>] kthread+0x9e/0xc0</div><div> [<ffffffff8100c20a>] child_rip+0xa/0x20</div><div> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0</div><div> [<ffffffff8100c200>] ? child_rip+0x0/0x20</div><div><br></div></span><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">LustreError: 15586:0:(osd_handler.c:1017:osd_trans_start()) ASSERTION( get_current()->journal_info == ((void *)0) ) failed: </div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">LustreError: 15586:0:(osd_handler.c:1017:osd_trans_start()) LBUG</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Pid: 15586, comm: mdt_rdpg00_004</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Call Trace:</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0502895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0502e97>] lbug_with_loc+0x47/0xb0 [libcfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0e1524d>] osd_trans_start+0x25d/0x660 [osd_ldiskfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa061ab4a>] llog_osd_destroy+0x42a/0xd40 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0613edc>] llog_cat_new_log+0x1ec/0x710 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa061450a>] llog_cat_add_rec+0x10a/0x450 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa060c1e9>] llog_add+0x89/0x1c0 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0654fdf>] ? keys_fill+0x6f/0x190 [obdclass]</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div> [<ffffffffa10d94e2>] mdd_changelog_store+0x122/0x290 [mdd]</div><div> [<ffffffffa10ecd0c>] mdd_changelog_data_store+0x16c/0x320 [mdd]</div></span><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa10f18ee>] mdd_close+0x34e/0xc50 [mdd]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0fba801>] mdt_mfd_close+0x3f1/0xac0 [mdt]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0636905>] ? class_handle2object+0x95/0x190 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0fbc313>] mdt_close+0x6f3/0xaa0 [mdt]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa08d056e>] tgt_request_handle+0x8be/0x1000 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa08805a1>] ptlrpc_main+0xe41/0x1960 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa087f760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div> [<ffffffff8109e66e>] kthread+0x9e/0xc0</div><div> [<ffffffff8100c20a>] child_rip+0xa/0x20</div><div> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0</div><div> [<ffffffff8100c200>] ? child_rip+0x0/0x20</div><div><br></div><div>Kernel panic - not syncing: LBUG</div></span><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Pid: 15586, comm: mdt_rdpg00_004 Not tainted 2.6.32-504.8.1.el6_lustre.x86_64 #1</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div>Call Trace:</div><div> [<ffffffff81529b76>] ? panic+0xa7/0x16f</div></span><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0502eeb>] ? lbug_with_loc+0x9b/0xb0 [libcfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0e1524d>] ? osd_trans_start+0x25d/0x660 [osd_ldiskfs]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa061ab4a>] ? llog_osd_destroy+0x42a/0xd40 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0613edc>] ? llog_cat_new_log+0x1ec/0x710 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa061450a>] ? llog_cat_add_rec+0x10a/0x450 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa060c1e9>] ? llog_add+0x89/0x1c0 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0654fdf>] ? keys_fill+0x6f/0x190 [obdclass]</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div> [<ffffffffa10d94e2>] ? mdd_changelog_store+0x122/0x290 [mdd]</div><div> [<ffffffffa10ecd0c>] ? mdd_changelog_data_store+0x16c/0x320 [mdd]</div></span><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa10f18ee>] ? mdd_close+0x34e/0xc50 [mdd]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0fba801>] ? mdt_mfd_close+0x3f1/0xac0 [mdt]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0636905>] ? class_handle2object+0x95/0x190 [obdclass]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa0fbc313>] ? mdt_close+0x6f3/0xaa0 [mdt]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa08d056e>] ? tgt_request_handle+0x8be/0x1000 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa08805a1>] ? ptlrpc_main+0xe41/0x1960 [ptlrpc]</div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> [<ffffffffa087f760>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]</div><span class="m_-9197701292137868598gmail-im" style="color:rgb(80,0,80);font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div> [<ffffffff8109e66e>] ? kthread+0x9e/0xc0</div><div> [<ffffffff8100c20a>] ? child_rip+0xa/0x20</div><div> [<ffffffff8109e5d0>] ? kthread+0x0/0xc0</div><div> [<ffffffff8100c200>] ? child_rip+0x0/0x20</div></span></div><div><br></div><div>We hope there is an solution for this problem and that we can go back to production.</div><div><br></div><div>Best regards,</div>-- <br><div dir="ltr" class="m_-9197701292137868598gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="font-size:small;margin-bottom:0.0001pt;line-height:normal"><b style="font-family:georgia,serif;font-size:12.8px"><span lang="EN-US" style="font-size:10pt;color:black">Hopko Meijering<br></span></b></p><p style="font-size:small;margin-bottom:0.0001pt;line-height:12pt;background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:14pt;color:rgb(204,0,0);font-family:georgia,serif">University of Groningen<br></span><span style="color:rgb(204,0,0);font-family:georgia,serif;font-size:10pt">Center for Information Technology (CIT)</span></p><p style="font-size:small;line-height:12pt;background-image:initial;background-position:initial;background-repeat:initial"><a href="http://www.rug.nl/cit" style="font-family:georgia,serif" target="_blank" rel="noreferrer">www.rug.nl/cit</a><br></p></div></div></div></div></div></div></div></div></div></div></div></div></div></div>