<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">Inline comments.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">Nasf<o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> lustre-discuss
[mailto:lustre-discuss-bounces@lists.lustre.org] <b>On Behalf Of </b></span><span style="font-size:11.0pt">代栋</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<b>Sent:</b> Thursday, May 3, 2018 1:22 AM<br>
<b>To:</b> Yong, Fan <fan.yong@intel.com><br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Is there a way to have faster lustre file system checker (lfsck)?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US">Many thanks. I will try to run namespace check again and see the results. BTW, is the rate control mechanism enabled by default? Can I disable it?<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:21.0pt"><b><i><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">[Nasf] The rate control is disabled by default. You can check current rate control via “lctl get_param mdd.*.lfsck_speed_limit”
on the MDT and/or “lctl get_param obdfilter.*.speed_limit” on the OSTs.</span></i></b><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US">I might be asking stupid questions. But, how MDT-object and OST-object pair each other during phase1 scanning? I mean whether OST sends objects metadata to MDT or
MDT sends objects metadata to OST for pairing? Could you point me to the source code, so that I can look for more details?<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:21.0pt"><b><i><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D">[Nasf] The LFSCK engine on the MDT will read all the (MDT) known OST-objects’ PFID EA from the OSTs, then verify
them. Please check lfsck_layout_assistant_handler_p1 for detail.</span></i></b><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US">thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US">- Dong<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US">On May 2, 2018, at 3:43 AM, Yong, Fan <<a href="mailto:fan.yong@intel.com">fan.yong@intel.com</a>> wrote:<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">Inline comments.<br>
<br>
--<br>
Cheers,<br>
Nasf<br>
<br style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><span lang="EN-US"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">-----Original Message-----<br>
From: </span><span style="font-size:13.5pt">代栋</span><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> [<a href="mailto:daidongly@gmail.com">mailto:daidongly@gmail.com</a>]<br>
Sent: Wednesday, May 2, 2018 4:06 PM<br>
To: Yong, Fan <<a href="mailto:fan.yong@intel.com">fan.yong@intel.com</a>><br>
Cc:<span class="apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
Subject: Re: [lustre-discuss] Is there a way to have faster lustre file system<br>
checker (lfsck)?<br>
<br>
Sorry, I misread “abnormal”. Anything I can check to help diagnose the<br>
slowness?<br>
<br>
Thanks,<br>
- Dong<br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">On May 2, 2018, at 2:53 AM,
</span><span style="font-size:13.5pt">代栋</span><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> <<a href="mailto:daidongly@gmail.com">daidongly@gmail.com</a>> wrote:<br>
<br>
Thanks very much for your reply.<br>
<br>
I used Lustre 2.9.0 and ran “lctl lfsck_start -M lustre-MDT0000 -A -t all -r” to<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">start LFSCK.<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
</blockquote>
</blockquote>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">You can try "lctl lfsck_start -M lustre-MDT0000 -A -t namespace -r" firstly to check the namespace LFSCK
speed. Since you has only one MDT, it should be quite faster. If yes, then check "lctl lfsck_start -M lustre-MDT0000 -A -t layout -r".<br>
<br style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><span lang="EN-US"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">Could you brief me more about the slowness? I mean scanning around 300K<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">inodes should not take that much time (80mins). These files were just created<br>
using a script after a fresh build of the lustre (no complex metadata operations<br>
at all).<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
</blockquote>
</blockquote>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">I do not know what caused such slowness. There may be many factors. Have you set some fail_loc? If not,
you may need to enable LFSCK debug on both the MDT and OST, then collect and analysis Lustre debug logs.<br>
<br>
<br style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><span lang="EN-US"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">Got it, so the 30-sec interval is just for checking the status of the MDT.<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">Another question is, for layout checking, does lfsck need to compare metadata<br>
stored in MDT (in LayoutEA) and metadata stored in OSTs (FID in LMA? not very<br>
sure) for orphan objects? When are these metadata gathered into one place for<br>
checking? I am asking this because previously I thought the periodically queries<br>
from OSTs to MDT are doing this job.<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
</blockquote>
</blockquote>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">In short, all the MDT-object OST-object pairs have been marked during the 1st stage scanning. So If there
are some OST-objects non-marked, then it may be orphans those will be handled during the 2nd phase scanning.<br>
<br>
<br style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><span lang="EN-US"><o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">Thanks,<br>
- Dong<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">On May 1, 2018, at 9:49 PM, Yong, Fan <<a href="mailto:fan.yong@intel.com">fan.yong@intel.com</a>> wrote:<br>
<br>
Inline comments.<br>
<br>
--<br>
Cheers,<br>
Nasf<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">-----Original Message-----<br>
From: lustre-discuss<br>
[<a href="mailto:lustre-discuss-bounces@lists.lustre.org">mailto:lustre-discuss-bounces@lists.lustre.org</a>] On Behalf Of
</span><span style="font-size:13.5pt">代栋</span><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
Sent: Wednesday, May 2, 2018 5:36 AM<br>
To: <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
Subject: [lustre-discuss] Is there a way to have faster lustre file<br>
system checker (lfsck)?<br>
<br>
Hi all,<br>
<br>
I am still new to Lustre, so please let me know if I should send<br>
this message to devel-list.<br>
<br>
This week, I tried to run LFSCK over a very small cluster<br>
configuration (1 mdt and<br>
3 osts). In this Lustre, I used about 300K inodes. It took me<br>
about 80 mins to finish a LFSCK run. And, more importantly, while I<br>
am running LFSCK, on both MDT and OSTS, the CPU utilization is 100%,<o:p></o:p></span></p>
</blockquote>
</blockquote>
</blockquote>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">taken by the lfsck thread.<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
Which version of Lustre and what is the LFSCK command line you used?<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">I understand that lfsck is operating in an online mode, so it is<br>
slow. But, I am wondering is there any way to accelerate this?<br>
Especially if I am allowed to run it offline, for example, during weekly<o:p></o:p></span></p>
</blockquote>
</blockquote>
</blockquote>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">maintenance.<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
Your slow is abnormal, not related with online. The LFSCK can NOT be run<o:p></o:p></span></p>
</blockquote>
</blockquote>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">under offline mode.<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
After checking the lfsck kernel logs, I noticed that in the phase2<br>
scanning on OSTs, there is an 30 seconds interval between querying<br>
the MDTs. I am wondering is there any reason to have this 30<br>
seconds interval, and will lfsck on OSTs be faster if we remove such 30<o:p></o:p></span></p>
</blockquote>
</blockquote>
</blockquote>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">seconds interval?<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
Normally, the master engine on the MDT will notify the LFSCK engine on the<o:p></o:p></span></p>
</blockquote>
</blockquote>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif">OST when the first phase done. But we can NOT guarantee that the LFSCK<br>
engine on the MDT always alive during the LFSCK (may because of some failure,<br>
or network trouble, or node crash, and so on), so in the 2nd phase scanning, if<br>
the LFSCK engine on the OST does not receive the notification from the MDT, it<br>
needs to query the LFSCK (on the MDT) status periodically. If the MDT finished<br>
the 1st phase scanning earlier than OST, then there will be no such query.<br>
Anyway, such query is NOT the reason of your slow LFSCK.<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US" style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
Thanks,<br>
- Dong<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></span></p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:21.0pt;mso-para-margin-left:1.75gd"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
</body>
</html>