<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Please help! Got another panic today, while migrating
directories:</p>
<p><br>
</p>
<p><b><tt>[Tue Jul 23 16:04:18 2019] LustreError:
52142:0:(service.c:189:ptlrpc_save_lock()) ASSERTION(
rs->rs_nlocks < 8 ) failed:</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019] LustreError:
52142:0:(service.c:189:ptlrpc_save_lock()) LBUG</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019] Pid: 52142, comm:
mdt00_002 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Tue Apr 30
22:18:15 UTC 2019</tt></b><b><tt><br>
</tt></b><b><tt><br>
</tt></b><b><tt>Message from syslogd@cmal14lb27 at Jul 23
16:04:15 ...</tt></b><b><tt><br>
</tt></b><b><tt> kernel:[101545.117309] LustreError:
52142:0:(service.c:189:ptlrpc_save_lock()) ASSERTION(
rs->rs_nlocks < 8 ) failed:</tt></b><b><tt><br>
</tt></b><b><tt><br>
</tt></b><b><tt>Message from syslogd@cmal14lb27 at Jul 23
16:04:15 ...</tt></b><b><tt><br>
</tt></b><b><tt> kernel:[101545.119130] LustreError:
52142:0:(service.c:189:ptlrpc_save_lock()) LBUG</tt></b><b><tt><br>
</tt></b><b><tt><br>
</tt></b><b><tt>Message from syslogd@cmal14lb27 at Jul 23
16:04:15 ...</tt></b><b><tt><br>
</tt></b><b><tt> kernel:LustreError:
52142:0:(service.c:189:ptlrpc_save_lock()) ASSERTION(
rs->rs_nlocks < 8 ) failed:</tt></b><b><tt><br>
</tt></b><b><tt><br>
</tt></b><b><tt>Message from syslogd@cmal14lb27 at Jul 23
16:04:15 ...</tt></b><b><tt><br>
</tt></b><b><tt> kernel:LustreError:
52142:0:(service.c:189:ptlrpc_save_lock()) LBUG</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019] Call Trace:</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc0f717cc>] libcfs_call_trace+0x8c/0xc0
[libcfs]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc0f7187c>] lbug_with_loc+0x4c/0xa0 [libcfs]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc12f9c41>] ptlrpc_save_lock+0xc1/0xd0 [ptlrpc]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc1892b0b>] mdt_save_lock+0x20b/0x360 [mdt]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc1892cbc>] mdt_object_unlock+0x5c/0x3c0 [mdt]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc18aba52>]
mdt_reint_striped_unlock+0x1a2/0x2f0 [mdt]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc18abbc8>] mdt_migrate_object_unlock+0x28/0x60
[mdt]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc18b0544>] mdt_reint_migrate+0x934/0x1310
[mdt]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc18b0fa3>] mdt_reint_rec+0x83/0x210 [mdt]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc188f1b3>] mdt_reint_internal+0x6e3/0xaf0
[mdt]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc189a497>] mdt_reint+0x67/0x140 [mdt]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc1359e5a>] tgt_request_handle+0xaea/0x1580
[ptlrpc]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc12ff80b>]
ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffc130313c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffff9fec1c71>] kthread+0xd1/0xe0</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffa0575c37>] ret_from_fork_nospec_end+0x0/0x39</tt></b><b><tt><br>
</tt></b><b><tt>[Tue Jul 23 16:04:18 2019]
[<ffffffffffffffff>] 0xffffffffffffffff</tt></b><b><tt><br>
</tt></b><br>
</p>
<div class="moz-signature"> Regards,<br>
<br>
Jonny<br>
<br clear="all">
<div style="ignore-me" --="">
<div
style="margin:0px;color:rgba(0,0,0,0.87);font-family:Helvetica,Arial,sans-serif;font-size:14px;letter-spacing:normal"><br>
<table
style="margin:0px;border-collapse:collapse;vertical-align:top"
cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td colspan="2">
<hr></td>
</tr>
<tr>
<td style="padding:8px" valign="middle"> <img
src="https://s3.glbimg.com/v1/AUTH_0601abe00ef04d20a8b20254df29a44e/assinatura/Globocom.png"
alt="globo.com" title="Globo.com" width="56"
height="56"> </td>
<td>
<table cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="color:rgb(17,17,17);font-family:"
helvetica=""
neue",helvetica,arial,sans-serif;line-height:24px"="">
<b>João Carlos Mendes Luís</b> </td>
</tr>
<tr>
<td style="color:rgb(85,85,85);font-family:"
helvetica=""
neue",helvetica,arial,sans-serif;font-size:12px;line-height:16px"="">
<b>Senior DevOps Engineer</b> </td>
</tr>
<tr>
<td
style="padding-top:8px;color:rgb(85,85,85);font-family:"
helvetica=""
neue",helvetica,arial,sans-serif;font-size:12px;line-height:16px"="">
<div
style="line-height:20px;font-size:14px;letter-spacing:0px;margin:0px">
<a href="mailto:jonny@corp.globo.com">jonny@corp.globo.com</a><br>
+55-21-2483-6893<br>
+55-21-99218-1222<br>
</div>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
<p><br>
</p>
</div>
</div>
<div class="moz-cite-prefix">On 22/07/2019 23:35, Joao Carlos Mendes
Luis wrote:<br>
</div>
<blockquote type="cite"
cite="mid:3d1db39b-8bb5-015e-1ed4-51d4c1750f79@corp.globo.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div class="moz-cite-prefix">On 7/22/19 11:10 PM, Andreas Dilger
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:0FCD2EF2-A106-490F-A9A9-52142B7C5391@whamcloud.com">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
If you are trying to delete MDT0000 then that is definitely not
implemented yet...<br>
</blockquote>
<p><br>
</p>
<p>No, no, no...</p>
<p><br>
</p>
<p>This was my first idea, but then I understood that the root
directory is always on MDT0, so I had to migrate it to another
server (after having created two more, and crashed during
migration).</p>
<p>I will later try to migrate to another server, and then delete
MDT2. But first I need to finish this lfsck... :-(</p>
<p><br>
</p>
<p>These "NOT IMPLEMETED (sic)" messages are just from running
lfsck_start -A<br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:0FCD2EF2-A106-490F-A9A9-52142B7C5391@whamcloud.com"> <br>
<div id="AppleMailSignature" dir="ltr">Cheers, Andreas</div>
<div dir="ltr"><br>
On Jul 22, 2019, at 16:08, João Carlos Mendes Luís <<a
href="mailto:jonny@corp.globo.com" moz-do-not-send="true">jonny@corp.globo.com</a>>
wrote:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<p>Hi,</p>
<p> I'm running some lab tests with lustre 2.12.2 in
Oracle Linux Server release 7.6. Last test I did was
about migration and MDT splitting. I started with a
MGS+MDS node, and two OSS nodes, and one of the tests was
to create two more MDSs and migrate data between then,
until, after some time, I could delete the original MDS.
But something happened in the middle and the servers
panicked/rebooted.<br>
</p>
<p> I am now in what appears to be an lfsck bug. After
many other tests, I run lfsck_start, and after some time
get this message on the nodes:</p>
<p>MGS/MDS0:</p>
<p><b><tt>[Mon Jul 22 17:42:25 2019] LustreError:
24107:0:(osd_index.c:1872:osd_index_it_get()) NOT
IMPLEMETED YET (move to 0x2481000002000000)</tt></b></p>
<p>OSS1/MDS1</p>
<p><b><tt>[Mon Jul 22 17:40:29 2019] LustreError:
31558:0:(osd_index.c:1872:osd_index_it_get()) NOT
IMPLEMETED YET (move to 0xa41300c002000000)</tt></b></p>
<p>OST2/MDS2</p>
<p><tt><b>[Mon Jul 22 17:40:32 2019] LustreError:
8935:0:(osd_index.c:1872:osd_index_it_get()) NOT
IMPLEMETED YET (move to 0xa013000003000000)</b></tt></p>
<p><br>
</p>
<p> And for current lfsck status, I run <b><tt>lctl
get_param *.*.lfsck* | grep -E
'status|\.lfsck_lay|\.lfsck_name'</tt></b><br>
</p>
<p>MGS/MDS0:</p>
<p><b><tt>mdd.mirror01-MDT0000.lfsck_layout=</tt></b><b><tt><br>
</tt></b><b><tt>status: completed</tt></b><b><tt><br>
</tt></b><b><tt>mdd.mirror01-MDT0000.lfsck_namespace=</tt></b><b><tt><br>
</tt></b><b><tt>status: partial</tt></b></p>
<p>OSS1/MDS1</p>
<b><tt>mdd.mirror01-MDT0001.lfsck_layout=</tt></b><b><tt><br>
</tt></b><b><tt>status: completed</tt></b><b><tt><br>
</tt></b><b><tt>mdd.mirror01-MDT0001.lfsck_namespace=</tt></b><b><tt><br>
</tt></b><b><tt>status: partial</tt></b><b><tt><br>
</tt></b><b><tt>obdfilter.mirror01-OST0065.lfsck_layout=</tt></b><b><tt><br>
</tt></b><b><tt>status: completed</tt></b><br>
<br>
<p>OST2/MDS2</p>
<b><tt>mdd.mirror01-MDT0002.lfsck_layout=</tt></b><b><tt><br>
</tt></b><b><tt>status: completed</tt></b><b><tt><br>
</tt></b><b><tt>mdd.mirror01-MDT0002.lfsck_namespace=</tt></b><b><tt><br>
</tt></b><b><tt>status: partial</tt></b><b><tt><br>
</tt></b><b><tt>obdfilter.mirror01-OST0066.lfsck_layout=</tt></b><b><tt><br>
</tt></b><b><tt>status: completed</tt></b><br>
<p> Is this a known bug? How do I fix these "partial"
lsfck runs?<br>
</p>
<p> Thanks for any help,</p>
<div class="moz-signature"><br>
Jonny<br>
<br clear="all">
<div style="ignore-me" --="">
<div
style="margin:0px;color:rgba(0,0,0,0.87);font-family:Helvetica,Arial,sans-serif;font-size:14px;letter-spacing:normal"><br>
<table
style="margin:0px;border-collapse:collapse;vertical-align:top"
cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td colspan="2">
<hr> </td>
</tr>
<tr>
<td style="padding:8px" valign="middle"><img
src="https://s3.glbimg.com/v1/AUTH_0601abe00ef04d20a8b20254df29a44e/assinatura/Globocom.png"
alt="globo.com" title="Globo.com"
moz-do-not-send="true" width="56"
height="56"> </td>
<td>
<table cellspacing="0" cellpadding="0"
border="0">
<tbody>
<tr>
<td
style="color:rgb(17,17,17);font-family:"
helveticaneue?,helvetica,arial,sans-serif;line-height:24px?=""> <b>João
Carlos Mendes Luís</b> </td>
</tr>
<tr>
<td
style="color:rgb(85,85,85);font-family:"
helveticaneue?,helvetica,arial,sans-serif;font-size:12px;line-height:16px?="">
<b>Senior DevOps Engineer</b> </td>
</tr>
<tr>
<td
style="padding-top:8px;color:rgb(85,85,85);font-family:"
helveticaneue?,helvetica,arial,sans-serif;font-size:12px;line-height:16px?="">
<div
style="line-height:20px;font-size:14px;letter-spacing:0px;margin:0px"><a
href="mailto:jonny@corp.globo.com"
moz-do-not-send="true">jonny@corp.globo.com</a><br>
+55-21-2483-6893<br>
+55-21-99218-1222<br>
</div>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
<p><br>
</p>
</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr"><span>_______________________________________________</span><br>
<span>lustre-discuss mailing list</span><br>
<span><a href="mailto:lustre-discuss@lists.lustre.org"
moz-do-not-send="true">lustre-discuss@lists.lustre.org</a></span><br>
<span><a
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span><br>
</div>
</blockquote>
</blockquote>
<p><br>
</p>
<div class="moz-signature">
<p> Atenciosamente, <br>
</p>
<p> Jonny <br>
</p>
<p> -- <br>
João Carlos Mendes Luís <br>
Globo.COM - +55-21-2483-6893</p>
</div>
</blockquote>
</body>
</html>