<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">To follow up here, the LU appears to duplicate a DNE-tirggered issue as well. There is a patch available which resolved the issue for us when using 2.10.5 in our environment.</div>
<div class="">For details and a link to the patch see <a href="https://jira.whamcloud.com/browse/LU-11411" class="">https://jira.whamcloud.com/browse/LU-11411</a></div>
<div class=""><br class="">
</div>
-Ben
<div class=""><br class="">
</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Sep 20, 2018, at 1:53 PM, Kirk, Benjamin (JSC-EG311) <<a href="mailto:benjamin.kirk@nasa.gov" class="">benjamin.kirk@nasa.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
I just opened an LU on the issue <a href="https://jira.whamcloud.com/browse/LU-11411" class="">https://jira.whamcloud.com/browse/LU-11411</a> for anyone interested.
<div class=""><br class="">
</div>
<div class="">Thanks a lot!</div>
<div class=""><br class="">
<div class="">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">
<div class=""><span style="orphans: 2; widows: 2;" class="">-Ben</span></div>
</div>
<div class=""><span style="orphans: 2; widows: 2;" class=""><br class="">
</span></div>
<div class=""><span style="orphans: 2; widows: 2;" class=""><br class="">
</span></div>
<div class=""><span style="orphans: 2; widows: 2;" class=""><br class="">
</span></div>
</div>
</div>
<div class="">
<blockquote type="cite" class="">
<div class="">On Aug 27, 2018, at 4:56 PM, Andreas Dilger <<a href="mailto:adilger@whamcloud.com" class="">adilger@whamcloud.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">It's probably best to file an LU ticket for this issue.<br class="">
<br class="">
It looks like there is something with the log processing at mount that is trying to modify the configuration files. I'm not sure whether that should be allowed or not.<br class="">
<br class="">
Does fab have the same MGS as fsA? Does it have the same MDS node as fsA?<br class="">
If it has a different MDS, you might consider to give it its own MGS as well.<br class="">
That doesn't have to be a separate MGS node, just a separate filesystem (ZFS fileset in the same zpool) on the MDS node.<br class="">
<br class="">
Cheers, Andreas<br class="">
<br class="">
<blockquote type="cite" class="">On Aug 27, 2018, at 10:18, Kirk, Benjamin (JSC-EG311) <<a href="mailto:benjamin.kirk@nasa.gov" class="">benjamin.kirk@nasa.gov</a>> wrote:<br class="">
<br class="">
Hi all,<br class="">
<br class="">
We have two filesystems, fsA & fsB (eadc below). Both of which get snapshots taken daily, rotated over a week. It’s a beautiful feature we’ve been using in production ever since it was introduced with 2.10.<br class="">
<br class="">
-) We’ve got Lustre/ZFS 2.10.4 on CentOS 7.5.<br class="">
-) Both fsA & fsB have changelogs active.<br class="">
-) fsA has combined mgt/mdt on a single ZFS filesystem.<br class="">
-) fsB has a single mdt on a single ZFS filesystem.<br class="">
-) for fsA, I have no issues mounting any of the snapshots via lctl.<br class="">
-) for fsB, I can mount the most three recent snapshots, then encounter errors:<br class="">
<br class="">
[root@hpfs-fsl-mds0 ~]# lctl snapshot_mount -F eadc -n eadc_AutoSS-Mon<br class="">
mounted the snapshot eadc_AutoSS-Mon with fsname 3d40bbc<br class="">
[root@hpfs-fsl-mds0 ~]# lctl snapshot_umount -F eadc -n eadc_AutoSS-Mon<br class="">
[root@hpfs-fsl-mds0 ~]# lctl snapshot_mount -F eadc -n eadc_AutoSS-Sun<br class="">
mounted the snapshot eadc_AutoSS-Sun with fsname 584c07a<br class="">
[root@hpfs-fsl-mds0 ~]# lctl snapshot_umount -F eadc -n eadc_AutoSS-Sun<br class="">
[root@hpfs-fsl-mds0 ~]# lctl snapshot_mount -F eadc -n eadc_AutoSS-Sat<br class="">
mounted the snapshot eadc_AutoSS-Sat with fsname 4e646fe<br class="">
[root@hpfs-fsl-mds0 ~]# lctl snapshot_umount -F eadc -n eadc_AutoSS-Sat<br class="">
[root@hpfs-fsl-mds0 ~]# lctl snapshot_mount -F eadc -n eadc_AutoSS-Fri<br class="">
mount.lustre: mount metadata/meta-eadc@eadc_AutoSS-Fri at /mnt/eadc_AutoSS-Fri_MDT0000 failed: Read-only file system<br class="">
Can't mount the snapshot eadc_AutoSS-Fri: Read-only file system<br class="">
<br class="">
The relevant bits from dmesg are<br class="">
[1353434.417762] Lustre: 3d40bbc-MDT0000: set dev_rdonly on this device<br class="">
[1353434.417765] Lustre: Skipped 3 previous similar messages<br class="">
[1353434.649480] Lustre: 3d40bbc-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900<br class="">
[1353434.649484] Lustre: Skipped 3 previous similar messages<br class="">
[1353434.866228] Lustre: 3d40bbc-MDD0000: changelog on<br class="">
[1353434.866233] Lustre: Skipped 1 previous similar message<br class="">
[1353435.427744] Lustre: 3d40bbc-MDT0000: Connection restored to ...@tcp (at ...@tcp)<br class="">
[1353435.427747] Lustre: Skipped 23 previous similar messages<br class="">
[1353445.255899] Lustre: Failing over 3d40bbc-MDT0000<br class="">
[1353445.255903] Lustre: Skipped 3 previous similar messages<br class="">
[1353445.256150] LustreError: 11-0: 3d40bbc-OST0000-osc-MDT0000: operation ost_disconnect to node ...@tcp failed: rc = -107<br class="">
[1353445.257896] LustreError: Skipped 23 previous similar messages<br class="">
[1353445.353874] Lustre: server umount 3d40bbc-MDT0000 complete<br class="">
[1353445.353877] Lustre: Skipped 3 previous similar messages<br class="">
[1353475.302224] Lustre: 4e646fe-MDD0000: changelog on<br class="">
[1353475.302228] Lustre: Skipped 1 previous similar message<br class="">
[1353498.964016] LustreError: 25582:0:(osd_handler.c:341:osd_trans_create()) 36ca26b-MDT0000-osd: someone try to start transaction under readonly mode, should be disabled.<br class="">
[1353498.967260] LustreError: 25582:0:(osd_handler.c:341:osd_trans_create()) Skipped 1 previous similar message<br class="">
[1353498.968829] CPU: 6 PID: 25582 Comm: mount.lustre Kdump: loaded Tainted: P OE ------------ 3.10.0-862.6.3.el7.x86_64 #1<br class="">
[1353498.968830] Hardware name: Supermicro SYS-6027TR-D71FRF/X9DRT, BIOS 3.2a 08/04/2015<br class="">
[1353498.968832] Call Trace:<br class="">
[1353498.968841] [<ffffffffb5b0e80e>] dump_stack+0x19/0x1b<br class="">
[1353498.968851] [<ffffffffc0cbe5db>] osd_trans_create+0x38b/0x3d0 [osd_zfs]<br class="">
[1353498.968876] [<ffffffffc1116044>] llog_destroy+0x1f4/0x3f0 [obdclass]<br class="">
[1353498.968887] [<ffffffffc111f0f6>] llog_cat_reverse_process_cb+0x246/0x3f0 [obdclass]<br class="">
[1353498.968897] [<ffffffffc111a32c>] llog_reverse_process+0x38c/0xaa0 [obdclass]<br class="">
[1353498.968910] [<ffffffffc111eeb0>] ? llog_cat_process_cb+0x4e0/0x4e0 [obdclass]<br class="">
[1353498.968922] [<ffffffffc111af69>] llog_cat_reverse_process+0x179/0x270 [obdclass]<br class="">
[1353498.968932] [<ffffffffc1115585>] ? llog_init_handle+0xd5/0x9a0 [obdclass]<br class="">
[1353498.968943] [<ffffffffc1116e78>] ? llog_open_create+0x78/0x320 [obdclass]<br class="">
[1353498.968949] [<ffffffffc12e55f0>] ? mdd_root_get+0xf0/0xf0 [mdd]<br class="">
[1353498.968954] [<ffffffffc12ec7af>] mdd_prepare+0x13ff/0x1c70 [mdd]<br class="">
[1353498.968966] [<ffffffffc166b037>] mdt_prepare+0x57/0x3b0 [mdt]<br class="">
[1353498.968983] [<ffffffffc1183afd>] server_start_targets+0x234d/0x2bd0 [obdclass]<br class="">
[1353498.968999] [<ffffffffc1153500>] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass]<br class="">
[1353498.969012] [<ffffffffc118541d>] server_fill_super+0x109d/0x185a [obdclass]<br class="">
[1353498.969025] [<ffffffffc115cef8>] lustre_fill_super+0x328/0x950 [obdclass]<br class="">
[1353498.969038] [<ffffffffc115cbd0>] ? lustre_common_put_super+0x270/0x270 [obdclass]<br class="">
[1353498.969041] [<ffffffffb561f3bf>] mount_nodev+0x4f/0xb0<br class="">
[1353498.969053] [<ffffffffc1154f18>] lustre_mount+0x38/0x60 [obdclass]<br class="">
[1353498.969055] [<ffffffffb561ff3e>] mount_fs+0x3e/0x1b0<br class="">
[1353498.969060] [<ffffffffb563d4b7>] vfs_kern_mount+0x67/0x110<br class="">
[1353498.969062] [<ffffffffb563fadf>] do_mount+0x1ef/0xce0<br class="">
[1353498.969066] [<ffffffffb55f7c2c>] ? kmem_cache_alloc_trace+0x3c/0x200<br class="">
[1353498.969069] [<ffffffffb5640913>] SyS_mount+0x83/0xd0<br class="">
[1353498.969074] [<ffffffffb5b20795>] system_call_fastpath+0x1c/0x21<br class="">
[1353498.969079] LustreError: 25582:0:(llog_cat.c:1027:llog_cat_reverse_process_cb()) 36ca26b-MDD0000: fail to destroy empty log: rc = -30<br class="">
[1353498.970785] CPU: 6 PID: 25582 Comm: mount.lustre Kdump: loaded Tainted: P OE ------------ 3.10.0-862.6.3.el7.x86_64 #1<br class="">
[1353498.970786] Hardware name: Supermicro SYS-6027TR-D71FRF/X9DRT, BIOS 3.2a 08/04/2015<br class="">
[1353498.970787] Call Trace:<br class="">
[1353498.970790] [<ffffffffb5b0e80e>] dump_stack+0x19/0x1b<br class="">
[1353498.970795] [<ffffffffc0cbe5db>] osd_trans_create+0x38b/0x3d0 [osd_zfs]<br class="">
[1353498.970807] [<ffffffffc1117921>] llog_cancel_rec+0xc1/0x880 [obdclass]<br class="">
[1353498.970817] [<ffffffffc111e13b>] llog_cat_cleanup+0xdb/0x380 [obdclass]<br class="">
[1353498.970827] [<ffffffffc111f14d>] llog_cat_reverse_process_cb+0x29d/0x3f0 [obdclass]<br class="">
[1353498.970838] [<ffffffffc111a32c>] llog_reverse_process+0x38c/0xaa0 [obdclass]<br class="">
[1353498.970848] [<ffffffffc111eeb0>] ? llog_cat_process_cb+0x4e0/0x4e0 [obdclass]<br class="">
[1353498.970858] [<ffffffffc111af69>] llog_cat_reverse_process+0x179/0x270 [obdclass]<br class="">
[1353498.970868] [<ffffffffc1115585>] ? llog_init_handle+0xd5/0x9a0 [obdclass]<br class="">
[1353498.970878] [<ffffffffc1116e78>] ? llog_open_create+0x78/0x320 [obdclass]<br class="">
[1353498.970883] [<ffffffffc12e55f0>] ? mdd_root_get+0xf0/0xf0 [mdd]<br class="">
[1353498.970887] [<ffffffffc12ec7af>] mdd_prepare+0x13ff/0x1c70 [mdd]<br class="">
[1353498.970894] [<ffffffffc166b037>] mdt_prepare+0x57/0x3b0 [mdt]<br class="">
[1353498.970908] [<ffffffffc1183afd>] server_start_targets+0x234d/0x2bd0 [obdclass]<br class="">
[1353498.970924] [<ffffffffc1153500>] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass]<br class="">
[1353498.970938] [<ffffffffc118541d>] server_fill_super+0x109d/0x185a [obdclass]<br class="">
[1353498.970950] [<ffffffffc115cef8>] lustre_fill_super+0x328/0x950 [obdclass]<br class="">
[1353498.970962] [<ffffffffc115cbd0>] ? lustre_common_put_super+0x270/0x270 [obdclass]<br class="">
[1353498.970964] [<ffffffffb561f3bf>] mount_nodev+0x4f/0xb0<br class="">
[1353498.970976] [<ffffffffc1154f18>] lustre_mount+0x38/0x60 [obdclass]<br class="">
[1353498.970978] [<ffffffffb561ff3e>] mount_fs+0x3e/0x1b0<br class="">
[1353498.970980] [<ffffffffb563d4b7>] vfs_kern_mount+0x67/0x110<br class="">
[1353498.970982] [<ffffffffb563fadf>] do_mount+0x1ef/0xce0<br class="">
[1353498.970984] [<ffffffffb55f7c2c>] ? kmem_cache_alloc_trace+0x3c/0x200<br class="">
[1353498.970986] [<ffffffffb5640913>] SyS_mount+0x83/0xd0<br class="">
[1353498.970989] [<ffffffffb5b20795>] system_call_fastpath+0x1c/0x21<br class="">
[1353498.970996] LustreError: 25582:0:(mdd_device.c:354:mdd_changelog_llog_init()) 36ca26b-MDD0000: changelog init failed: rc = -30<br class="">
[1353498.972790] LustreError: 25582:0:(mdd_device.c:427:mdd_changelog_init()) 36ca26b-MDD0000: changelog setup during init failed: rc = -30<br class="">
[1353498.974525] LustreError: 25582:0:(mdd_device.c:1061:mdd_prepare()) 36ca26b-MDD0000: failed to initialize changelog: rc = -30<br class="">
[1353498.976229] LustreError: 25582:0:(obd_mount_server.c:1879:server_fill_super()) Unable to start targets: -30<br class="">
[1353499.072002] LustreError: 25582:0:(obd_mount.c:1582:lustre_fill_super()) Unable to mount (-30)<br class="">
<br class="">
<br class="">
I’m hoping those traces mean something to someone - any ideas?<br class="">
<br class="">
Thanks!<br class="">
<br class="">
--<br class="">
Benjamin S. Kirk<br class="">
<br class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br class="">
</blockquote>
<br class="">
Cheers, Andreas<br class="">
---<br class="">
Andreas Dilger<br class="">
CTO Whamcloud<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>