<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>Please help!  Got another panic today, while migrating
      directories:</p>
    <p><br>
    </p>
    <p><b><tt>[Tue Jul 23 16:04:18 2019] LustreError:
          52142:0:(service.c:189:ptlrpc_save_lock()) ASSERTION(
          rs->rs_nlocks < 8 ) failed:</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] LustreError:
          52142:0:(service.c:189:ptlrpc_save_lock()) LBUG</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] Pid: 52142, comm:
          mdt00_002 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Tue Apr 30
          22:18:15 UTC 2019</tt></b><b><tt><br>
        </tt></b><b><tt><br>
        </tt></b><b><tt>Message from syslogd@cmal14lb27 at Jul 23
          16:04:15 ...</tt></b><b><tt><br>
        </tt></b><b><tt> kernel:[101545.117309] LustreError:
          52142:0:(service.c:189:ptlrpc_save_lock()) ASSERTION(
          rs->rs_nlocks < 8 ) failed:</tt></b><b><tt><br>
        </tt></b><b><tt><br>
        </tt></b><b><tt>Message from syslogd@cmal14lb27 at Jul 23
          16:04:15 ...</tt></b><b><tt><br>
        </tt></b><b><tt> kernel:[101545.119130] LustreError:
          52142:0:(service.c:189:ptlrpc_save_lock()) LBUG</tt></b><b><tt><br>
        </tt></b><b><tt><br>
        </tt></b><b><tt>Message from syslogd@cmal14lb27 at Jul 23
          16:04:15 ...</tt></b><b><tt><br>
        </tt></b><b><tt> kernel:LustreError:
          52142:0:(service.c:189:ptlrpc_save_lock()) ASSERTION(
          rs->rs_nlocks < 8 ) failed:</tt></b><b><tt><br>
        </tt></b><b><tt><br>
        </tt></b><b><tt>Message from syslogd@cmal14lb27 at Jul 23
          16:04:15 ...</tt></b><b><tt><br>
        </tt></b><b><tt> kernel:LustreError:
          52142:0:(service.c:189:ptlrpc_save_lock()) LBUG</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] Call Trace:</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc0f717cc>] libcfs_call_trace+0x8c/0xc0
          [libcfs]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc0f7187c>] lbug_with_loc+0x4c/0xa0 [libcfs]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc12f9c41>] ptlrpc_save_lock+0xc1/0xd0 [ptlrpc]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc1892b0b>] mdt_save_lock+0x20b/0x360 [mdt]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc1892cbc>] mdt_object_unlock+0x5c/0x3c0 [mdt]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc18aba52>]
          mdt_reint_striped_unlock+0x1a2/0x2f0 [mdt]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc18abbc8>] mdt_migrate_object_unlock+0x28/0x60
          [mdt]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc18b0544>] mdt_reint_migrate+0x934/0x1310
          [mdt]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc18b0fa3>] mdt_reint_rec+0x83/0x210 [mdt]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc188f1b3>] mdt_reint_internal+0x6e3/0xaf0
          [mdt]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc189a497>] mdt_reint+0x67/0x140 [mdt]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc1359e5a>] tgt_request_handle+0xaea/0x1580
          [ptlrpc]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc12ff80b>]
          ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffc130313c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffff9fec1c71>] kthread+0xd1/0xe0</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffa0575c37>] ret_from_fork_nospec_end+0x0/0x39</tt></b><b><tt><br>
        </tt></b><b><tt>[Tue Jul 23 16:04:18 2019] 
          [<ffffffffffffffff>] 0xffffffffffffffff</tt></b><b><tt><br>
        </tt></b><br>
    </p>
    <div class="moz-signature">    Regards,<br>
      <br>
              Jonny<br>
      <br clear="all">
      <div style="ignore-me" --="">
        <div
style="margin:0px;color:rgba(0,0,0,0.87);font-family:Helvetica,Arial,sans-serif;font-size:14px;letter-spacing:normal"><br>
          <table
            style="margin:0px;border-collapse:collapse;vertical-align:top"
            cellspacing="0" cellpadding="0" border="0">
            <tbody>
              <tr>
                <td colspan="2">
                  <hr></td>
              </tr>
              <tr>
                <td style="padding:8px" valign="middle"> <img
src="https://s3.glbimg.com/v1/AUTH_0601abe00ef04d20a8b20254df29a44e/assinatura/Globocom.png"
                    alt="globo.com" title="Globo.com" width="56"
                    height="56"> </td>
                <td>
                  <table cellspacing="0" cellpadding="0" border="0">
                    <tbody>
                      <tr>
                        <td style="color:rgb(17,17,17);font-family:"
                          helvetica=""
                          neue",helvetica,arial,sans-serif;line-height:24px"="">
                          <b>João Carlos Mendes Luís</b> </td>
                      </tr>
                      <tr>
                        <td style="color:rgb(85,85,85);font-family:"
                          helvetica=""
                          neue",helvetica,arial,sans-serif;font-size:12px;line-height:16px"="">
                          <b>Senior DevOps Engineer</b> </td>
                      </tr>
                      <tr>
                        <td
                          style="padding-top:8px;color:rgb(85,85,85);font-family:"
                          helvetica=""
                          neue",helvetica,arial,sans-serif;font-size:12px;line-height:16px"="">
                          <div
                            style="line-height:20px;font-size:14px;letter-spacing:0px;margin:0px">
                            <a href="mailto:jonny@corp.globo.com">jonny@corp.globo.com</a><br>
                            +55-21-2483-6893<br>
                            +55-21-99218-1222<br>
                          </div>
                        </td>
                      </tr>
                    </tbody>
                  </table>
                </td>
              </tr>
            </tbody>
          </table>
        </div>
        <p><br>
        </p>
      </div>
    </div>
    <div class="moz-cite-prefix">On 22/07/2019 23:35, Joao Carlos Mendes
      Luis wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:3d1db39b-8bb5-015e-1ed4-51d4c1750f79@corp.globo.com">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      <div class="moz-cite-prefix">On 7/22/19 11:10 PM, Andreas Dilger
        wrote:<br>
      </div>
      <blockquote type="cite"
        cite="mid:0FCD2EF2-A106-490F-A9A9-52142B7C5391@whamcloud.com">
        <meta http-equiv="Content-Type" content="text/html;
          charset=UTF-8">
        If you are trying to delete MDT0000 then that is definitely not
        implemented yet...<br>
      </blockquote>
      <p><br>
      </p>
      <p>No, no, no...</p>
      <p><br>
      </p>
      <p>This was my first idea, but then I understood that the root
        directory is always on MDT0, so I had to migrate it to another
        server (after having created two more, and crashed during
        migration).</p>
      <p>I will later try to migrate to another server, and then delete
        MDT2.  But first I need to finish this lfsck...   :-(</p>
      <p><br>
      </p>
      <p>These "NOT IMPLEMETED (sic)" messages are just from running
        lfsck_start -A<br>
      </p>
      <p><br>
      </p>
      <blockquote type="cite"
        cite="mid:0FCD2EF2-A106-490F-A9A9-52142B7C5391@whamcloud.com"> <br>
        <div id="AppleMailSignature" dir="ltr">Cheers, Andreas</div>
        <div dir="ltr"><br>
          On Jul 22, 2019, at 16:08, João Carlos Mendes Luís <<a
            href="mailto:jonny@corp.globo.com" moz-do-not-send="true">jonny@corp.globo.com</a>>
          wrote:<br>
          <br>
        </div>
        <blockquote type="cite">
          <div dir="ltr">
            <p>Hi,</p>
            <p>    I'm running some lab tests with lustre 2.12.2 in
              Oracle Linux Server release 7.6.  Last test I did was
              about migration and MDT splitting.  I started with a
              MGS+MDS node, and two OSS nodes, and one of the tests was
              to create two more MDSs and migrate data between then,
              until, after some time, I could delete the original MDS. 
              But something happened in the middle and the servers
              panicked/rebooted.<br>
            </p>
            <p>    I am now in what appears to be an lfsck bug.  After
              many other tests, I run lfsck_start, and after some time
              get this message on the nodes:</p>
            <p>MGS/MDS0:</p>
            <p><b><tt>[Mon Jul 22 17:42:25 2019] LustreError:
                  24107:0:(osd_index.c:1872:osd_index_it_get()) NOT
                  IMPLEMETED YET (move to 0x2481000002000000)</tt></b></p>
            <p>OSS1/MDS1</p>
            <p><b><tt>[Mon Jul 22 17:40:29 2019] LustreError:
                  31558:0:(osd_index.c:1872:osd_index_it_get()) NOT
                  IMPLEMETED YET (move to 0xa41300c002000000)</tt></b></p>
            <p>OST2/MDS2</p>
            <p><tt><b>[Mon Jul 22 17:40:32 2019] LustreError:
                  8935:0:(osd_index.c:1872:osd_index_it_get()) NOT
                  IMPLEMETED YET (move to 0xa013000003000000)</b></tt></p>
            <p><br>
            </p>
            <p>    And for current lfsck status, I run <b><tt>lctl
                  get_param *.*.lfsck* | grep -E
                  'status|\.lfsck_lay|\.lfsck_name'</tt></b><br>
            </p>
            <p>MGS/MDS0:</p>
            <p><b><tt>mdd.mirror01-MDT0000.lfsck_layout=</tt></b><b><tt><br>
                </tt></b><b><tt>status: completed</tt></b><b><tt><br>
                </tt></b><b><tt>mdd.mirror01-MDT0000.lfsck_namespace=</tt></b><b><tt><br>
                </tt></b><b><tt>status: partial</tt></b></p>
            <p>OSS1/MDS1</p>
            <b><tt>mdd.mirror01-MDT0001.lfsck_layout=</tt></b><b><tt><br>
              </tt></b><b><tt>status: completed</tt></b><b><tt><br>
              </tt></b><b><tt>mdd.mirror01-MDT0001.lfsck_namespace=</tt></b><b><tt><br>
              </tt></b><b><tt>status: partial</tt></b><b><tt><br>
              </tt></b><b><tt>obdfilter.mirror01-OST0065.lfsck_layout=</tt></b><b><tt><br>
              </tt></b><b><tt>status: completed</tt></b><br>
            <br>
            <p>OST2/MDS2</p>
            <b><tt>mdd.mirror01-MDT0002.lfsck_layout=</tt></b><b><tt><br>
              </tt></b><b><tt>status: completed</tt></b><b><tt><br>
              </tt></b><b><tt>mdd.mirror01-MDT0002.lfsck_namespace=</tt></b><b><tt><br>
              </tt></b><b><tt>status: partial</tt></b><b><tt><br>
              </tt></b><b><tt>obdfilter.mirror01-OST0066.lfsck_layout=</tt></b><b><tt><br>
              </tt></b><b><tt>status: completed</tt></b><br>
            <p>    Is this a known bug?  How do I fix these "partial"
              lsfck runs?<br>
            </p>
            <p>    Thanks for any help,</p>
            <div class="moz-signature"><br>
                      Jonny<br>
              <br clear="all">
              <div style="ignore-me" --="">
                <div
style="margin:0px;color:rgba(0,0,0,0.87);font-family:Helvetica,Arial,sans-serif;font-size:14px;letter-spacing:normal"><br>
                  <table
                    style="margin:0px;border-collapse:collapse;vertical-align:top"
                    cellspacing="0" cellpadding="0" border="0">
                    <tbody>
                      <tr>
                        <td colspan="2">
                          <hr> </td>
                      </tr>
                      <tr>
                        <td style="padding:8px" valign="middle"><img
src="https://s3.glbimg.com/v1/AUTH_0601abe00ef04d20a8b20254df29a44e/assinatura/Globocom.png"
                            alt="globo.com" title="Globo.com"
                            moz-do-not-send="true" width="56"
                            height="56"> </td>
                        <td>
                          <table cellspacing="0" cellpadding="0"
                            border="0">
                            <tbody>
                              <tr>
                                <td
                                  style="color:rgb(17,17,17);font-family:"
helveticaneue?,helvetica,arial,sans-serif;line-height:24px?=""> <b>João
                                    Carlos Mendes Luís</b> </td>
                              </tr>
                              <tr>
                                <td
                                  style="color:rgb(85,85,85);font-family:"
helveticaneue?,helvetica,arial,sans-serif;font-size:12px;line-height:16px?="">
                                  <b>Senior DevOps Engineer</b> </td>
                              </tr>
                              <tr>
                                <td
                                  style="padding-top:8px;color:rgb(85,85,85);font-family:"
helveticaneue?,helvetica,arial,sans-serif;font-size:12px;line-height:16px?="">
                                  <div
                                    style="line-height:20px;font-size:14px;letter-spacing:0px;margin:0px"><a
                                      href="mailto:jonny@corp.globo.com"
                                      moz-do-not-send="true">jonny@corp.globo.com</a><br>
                                    +55-21-2483-6893<br>
                                    +55-21-99218-1222<br>
                                  </div>
                                </td>
                              </tr>
                            </tbody>
                          </table>
                        </td>
                      </tr>
                    </tbody>
                  </table>
                </div>
                <p><br>
                </p>
              </div>
            </div>
          </div>
        </blockquote>
        <blockquote type="cite">
          <div dir="ltr"><span>_______________________________________________</span><br>
            <span>lustre-discuss mailing list</span><br>
            <span><a href="mailto:lustre-discuss@lists.lustre.org"
                moz-do-not-send="true">lustre-discuss@lists.lustre.org</a></span><br>
            <span><a
                href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
                moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a></span><br>
          </div>
        </blockquote>
      </blockquote>
      <p><br>
      </p>
      <div class="moz-signature">
        <p>    Atenciosamente, <br>
        </p>
        <p>        Jonny <br>
        </p>
        <p> -- <br>
          João Carlos Mendes Luís <br>
          Globo.COM - +55-21-2483-6893</p>
      </div>
    </blockquote>
  </body>
</html>