<div dir="ltr"><div>yeah I don't know how successful lctl --device abort_recovery is going to be vs abort_recov on the device itself, I think probably by the time you get to aborting it via lctl it's already too late.</div><div><br></div><div>But to confirm you're back online again? (Also, time to upgrade!) =)<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 10, 2020 at 7:57 AM <<a href="mailto:s_b@mit.edu">s_b@mit.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Hi Colin,<br>
<br>
Thank you. That was the tip I needed!<br>
<br>
We are running IEEL so I did the following...<br>
<br>
* mount the mdt by hand with -o abort_recov<br>
<br>
mount -v -t lustre -o abort_recov /dev/mapper/mpatha /mnt/lustre02-MDT0000<br>
<br>
* after it mounted up, umount it<br>
* start the mdt via IEEL<br>
* mount the file system on clients.<br>
<br>
I also tried to start the mdt with IEEL then use<br>
<br>
lctl --device 4 abort_recovery<br>
<br>
but that didn't work.<br>
<br>
Cheers,<br>
<br>
sb. Scott Blomquist<br>
<br>
<br>
Colin Faber <<a href="mailto:cfaber@gmail.com" target="_blank">cfaber@gmail.com</a>> writes:<br>
<br>
> Scott,<br>
><br>
> Have you tried aborting recovery on mount?<br>
><br>
> On Mon, Nov 9, 2020 at 1:15 PM <<a href="mailto:s_b@mit.edu" target="_blank">s_b@mit.edu</a>> wrote:<br>
><br>
>><br>
>> Hi All,<br>
>><br>
>> After the recent power glitch last week one of our lustre file systems<br>
>> failed to come up.<br>
>><br>
>> We diagnosed the problem down to a file system error on the MDT. This<br>
>> is an old IEEL systems running on Dell equipment.<br>
>><br>
>> Here are the facts...<br>
>><br>
>> * the raid 6 array running on an Dell MD32xx is ok.<br>
>><br>
>> * when we bring up the MDT it goes read-only then the MDS host crashes<br>
>><br>
>> * after this the MDT file system is dirty and we have to e2fsck it<br>
>><br>
>> * I have tried multiple combinations of MDS up/down, OSS up/down with<br>
>> nothing changing the results.<br>
>><br>
>> * This seems to be lustre 2.7.15<br>
>><br>
>> I think this may be<br>
>><br>
>> <a href="https://jira.whamcloud.com/browse/LU-7045" rel="noreferrer" target="_blank">https://jira.whamcloud.com/browse/LU-7045</a><br>
>><br>
>> or something like that.<br>
>><br>
>> Is there a way to LFSCK (or something) this error away? Or is this a<br>
>> please update lustre error.<br>
>><br>
>> Thanks for any help.<br>
>><br>
>> I have attached the error below.<br>
>><br>
>> Thanks for any insight,<br>
>><br>
>> sb. Scott Blomquist<br>
>><br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: ------------[ cut here<br>
>> ]------------<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: WARNING: at<br>
>> /tmp/rpmbuild-lustre-jenkins-U6NXEPsD/BUILD/lustre-2.7.15.3/ldiskfs/ext4_jbd2.c:266<br>
>> __ldiskfs_handle_dirty_metadata+0x1c2/0x220 [ldiskfs]()<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: Modules linked in: osp(OE)<br>
>> mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE)<br>
>> ldiskfs(OE) lquota(OE) vfat fat usb_storage mpt3sas mptctl mptbase dell_rbu<br>
>> lustre(OE) lmv(OE) mdc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE)<br>
>> obdclass(OE) lnet(OE) sha512_generic fuse crypto_null libcfs(OE)<br>
>> rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE)<br>
>> ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_en(OE) vxlan<br>
>> ip6_udp_tunnel udp_tunnel intel_powerclamp coretemp intel_rapl kvm_intel<br>
>> kvm crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper<br>
>> ablk_helper iTCO_wdt dcdbas cryptd iTCO_vendor_support dm_round_robin<br>
>> pcspkr sg ipmi_devintf sb_edac edac_core acpi_power_meter ntb ipmi_si wm<br>
>> i shpchp acpi_pad ipmi_msghandler lpc_ich mei_me mei mfd_core<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: knem(OE) nfsd auth_rpcgss<br>
>> nfs_acl lockd grace sunrpc dm_multipath ip_tables ext4 mbcache jbd2<br>
>> mlx4_ib(OE) ib_sa(OE) ib_mad(OE) ib_core(OE) ib_addr(OE) sr_mod cdrom<br>
>> sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect<br>
>> sysimgblt i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common<br>
>> mpt2sas crc32c_intel ttm ahci raid_class drm libahci scsi_transport_sas<br>
>> mlx4_core(OE) mlx_compat(OE) libata tg3 i2c_core ptp megaraid_sas pps_core<br>
>> dm_mirror dm_region_hash dm_log dm_mod<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: CPU: 10 PID: 6577 Comm:<br>
>> mdt01_003 Tainted: G OE ------------<br>
>> 3.10.0-327.el7_lustre.gd4cb884.x86_64 #1<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: Hardware name: Dell Inc.<br>
>> PowerEdge R620/0PXXHP, BIOS 2.5.4 01/22/2016<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: 0000000000000000<br>
>> 00000000fff8486a ffff881f815234f0 ffffffff81635429<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: ffff881f81523528<br>
>> ffffffff8107b200 ffff880fc708f1a0 ffff880fe535b060<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: ffff881fa4aff7c8<br>
>> ffffffffa10b9a9c 0000000000000325 ffff881f81523538<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: Call Trace:<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffff81635429>]<br>
>> dump_stack+0x19/0x1b<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffff8107b200>]<br>
>> warn_slowpath_common+0x70/0xb0<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffff8107b34a>]<br>
>> warn_slowpath_null+0x1a/0x20<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa1057052>]<br>
>> __ldiskfs_handle_dirty_metadata+0x1c2/0x220 [ldiskfs]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa10a88c1>]<br>
>> ldiskfs_getblk+0x131/0x200 [ldiskfs]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa10a89b7>]<br>
>> ldiskfs_bread+0x27/0xc0 [ldiskfs]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa11ff069>]<br>
>> osd_ldiskfs_write_record+0x169/0x360 [osd_ldiskfs]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa11ff358>]<br>
>> osd_write+0xf8/0x230 [osd_ldiskfs]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0952cd5>]<br>
>> dt_record_write+0x45/0x130 [obdclass]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0c0e6c2>]<br>
>> tgt_last_rcvd_update+0x732/0xef0 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa033c1f8>] ?<br>
>> start_this_handle+0xa8/0x5d0 [jbd2]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0c13542>]<br>
>> tgt_txn_stop_cb+0x1a2/0x4a0 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffff811c115e>] ?<br>
>> kmem_cache_alloc_trace+0x1ce/0x1f0<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0952c23>]<br>
>> dt_txn_hook_stop+0x63/0x80 [obdclass]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa11dc9f2>]<br>
>> osd_trans_stop+0x112/0x3d0 [osd_ldiskfs]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa11daa9a>] ?<br>
>> osd_trans_start+0x1ba/0x670 [osd_ldiskfs]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa1338bf9>]<br>
>> mdt_empty_transno+0x109/0x790 [mdt]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa133baed>]<br>
>> mdt_mfd_open+0x91d/0xeb0 [mdt]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa133c5fb>]<br>
>> mdt_finish_open+0x57b/0x9d0 [mdt]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa133d8a0>]<br>
>> mdt_reint_open+0xe50/0x2e00 [mdt]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0bb1bc7>] ?<br>
>> lustre_msg_add_version+0x27/0xa0 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0baf670>] ?<br>
>> lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa096f3ee>] ?<br>
>> lu_ucred+0x1e/0x30 [obdclass]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffff812fbd92>] ?<br>
>> strlcpy+0x42/0x60<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa1331ce0>]<br>
>> mdt_reint_rec+0x80/0x210 [mdt]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa1313699>]<br>
>> mdt_reint_internal+0x5d9/0xb40 [mdt]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa1313d62>]<br>
>> mdt_intent_reint+0x162/0x420 [mdt]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0baf687>] ?<br>
>> lustre_msg_buf+0x17/0x60 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa1317655>]<br>
>> mdt_intent_opc+0x215/0x9b0 [mdt]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0bb3d90>] ?<br>
>> lustre_swab_ldlm_policy_data+0x30/0x30 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa131e6f8>]<br>
>> mdt_intent_policy+0x138/0x320 [mdt]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0b641f7>]<br>
>> ldlm_lock_enqueue+0x357/0x9c0 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0b8a772>]<br>
>> ldlm_handle_enqueue0+0x4f2/0x16f0 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0bb3e10>] ?<br>
>> lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0c16762>]<br>
>> tgt_enqueue+0x62/0x210 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0c1b7cb>]<br>
>> tgt_request_handle+0x8fb/0x11f0 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0bbe96b>]<br>
>> ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa07c7d98>] ?<br>
>> lc_watchdog_touch+0x68/0x180 [libcfs]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0bbba38>] ?<br>
>> ptlrpc_wait_event+0x98/0x330 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffff810af018>] ?<br>
>> __wake_up_common+0x58/0x90<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0bc2290>]<br>
>> ptlrpc_main+0xc00/0x1f50 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffffa0bc1690>] ?<br>
>> ptlrpc_register_service+0x1070/0x1070 [ptlrpc]<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffff810a5aef>]<br>
>> kthread+0xcf/0xe0<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffff810a5a20>] ?<br>
>> kthread_create_on_node+0x140/0x140<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffff81645a98>]<br>
>> ret_from_fork+0x58/0x90<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: [<ffffffff810a5a20>] ?<br>
>> kthread_create_on_node+0x140/0x140<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: ---[ end trace<br>
>> 522ffb7aaa9346b5 ]---<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: LDISKFS-fs:<br>
>> ldiskfs_getblk:805: aborting transaction: error 28 in<br>
>> __ldiskfs_handle_dirty_metadata<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: LDISKFS-fs error (device<br>
>> dm-1): ldiskfs_getblk:805: inode #85: block 607448: comm mdt01_003:<br>
>> journal_dirty_metadata failed: handle type 0 started at line 1141, credits<br>
>> 4/0, errcode -28<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: Aborting journal on device<br>
>> dm-1-8.<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: LDISKFS-fs (dm-1):<br>
>> Remounting filesystem read-only<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: LustreError:<br>
>> 6577:0:(osd_io.c:1655:osd_ldiskfs_write_record()) dm-1: error reading<br>
>> offset 45056 (block 11): rc = -28<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: LustreError:<br>
>> 6577:0:(tgt_lastrcvd.c:1191:tgt_last_rcvd_update()) lustre02-MDT0000: can't<br>
>> update reply_data file: rc = -28<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: LustreError:<br>
>> 6577:0:(osd_handler.c:1219:osd_trans_stop()) lustre02-MDT0000-osd: failed<br>
>> in transaction hook: rc = -28<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: LDISKFS-fs error (device<br>
>> dm-1) in osd_trans_stop:1225: error 28<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: LustreError:<br>
>> 6052:0:(osd_handler.c:993:osd_trans_commit_cb()) transaction<br>
>> @0xffff880fcde5c180 commit error: 2<br>
>> Nov 4 09:29:09 mds1-002.lustre.cluster kernel: LustreError:<br>
>> 6577:0:(osd_handler.c:1228:osd_trans_stop()) lustre02-MDT0000-osd: failed<br>
>> to stop transaction: rc = -28<br>
>><br>
>><br>
>><br>
>> <a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a> writes:<br>
>><br>
>> > You, or someone posing as you, has requested a password reminder for<br>
>> > your membership on the mailing list <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>.<br>
>> > You will need this password in order to change your membership options<br>
>> > (e.g. do you want regular delivery or digest delivery), and having<br>
>> > this password makes it easier for you to unsubscribe from the mailing<br>
>> > list.<br>
>> ><br>
>> > You are subscribed with the address: <a href="mailto:s_b@mit.edu" target="_blank">s_b@mit.edu</a><br>
>> ><br>
>> > Your lustre-discuss password is: graknond<br>
>> ><br>
>> > To make changes to your membership options, log in and visit your<br>
>> > options web page:<br>
>> ><br>
>> ><br>
>> <a href="http://lists.lustre.org/options.cgi/lustre-discuss-lustre.org/s_b%40mit.edu" rel="noreferrer" target="_blank">http://lists.lustre.org/options.cgi/lustre-discuss-lustre.org/s_b%40mit.edu</a><br>
>> ><br>
>> ><br>
>> > You can also make such changes via email by sending a message to:<br>
>> ><br>
>> > <a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank">lustre-discuss-request@lists.lustre.org</a><br>
>> ><br>
>> > with the text "help" in the subject or body. The automatic reply will<br>
>> > contain more detailed instructions.<br>
>> ><br>
>> > Questions or comments? Please send them to the lustre-discuss mailing<br>
>> > list administrator at <a href="mailto:lustre-discuss-owner@lists.lustre.org" target="_blank">lustre-discuss-owner@lists.lustre.org</a>.<br>
>><br>
>> _______________________________________________<br>
>> lustre-discuss mailing list<br>
>> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
>> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
>><br>
<br>
</blockquote></div>