<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Many thanks. I will try to run namespace check again and see the results. BTW, is the rate control mechanism enabled by default? Can I disable it?<div class=""><br class=""></div><div class="">I might be asking stupid questions. But, how MDT-object and OST-object pair each other during phase1 scanning? I mean whether OST sends objects metadata to MDT or MDT sends objects metadata to OST for pairing? Could you point me to the source code, so that I can look for more details?</div><div class=""><br class=""></div><div class="">thanks,</div><div class="">- Dong</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On May 2, 2018, at 3:43 AM, Yong, Fan <<a href="mailto:fan.yong@intel.com" class="">fan.yong@intel.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Inline comments.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">--</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Cheers,</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Nasf</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">-----Original Message-----<br class="">From: 代栋 [<a href="mailto:daidongly@gmail.com" class="">mailto:daidongly@gmail.com</a>]<br class="">Sent: Wednesday, May 2, 2018 4:06 PM<br class="">To: Yong, Fan <<a href="mailto:fan.yong@intel.com" class="">fan.yong@intel.com</a>><br class="">Cc:<span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">Subject: Re: [lustre-discuss] Is there a way to have faster lustre file system<br class="">checker (lfsck)?<br class=""><br class="">Sorry, I misread “abnormal”. Anything I can check to help diagnose the<br class="">slowness?<br class=""><br class="">Thanks,<br class="">- Dong<br class=""><br class=""><blockquote type="cite" class="">On May 2, 2018, at 2:53 AM, 代栋 <<a href="mailto:daidongly@gmail.com" class="">daidongly@gmail.com</a>> wrote:<br class=""><br class="">Thanks very much for your reply.<br class=""><br class="">I used Lustre 2.9.0 and ran “lctl lfsck_start -M lustre-MDT0000 -A -t all -r” to<br class=""></blockquote>start LFSCK.<br class=""><blockquote type="cite" class=""><br class=""></blockquote></blockquote><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">You can try "lctl lfsck_start -M lustre-MDT0000 -A -t namespace -r" firstly to check the namespace LFSCK speed. Since you has only one MDT, it should be quite faster. If yes, then check "lctl lfsck_start -M lustre-MDT0000 -A -t layout -r".</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" class="">Could you brief me more about the slowness? I mean scanning around 300K<br class=""></blockquote>inodes should not take that much time (80mins). These files were just created<br class="">using a script after a fresh build of the lustre (no complex metadata operations<br class="">at all).<br class=""><blockquote type="cite" class=""><br class=""></blockquote></blockquote><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">I do not know what caused such slowness. There may be many factors. Have you set some fail_loc? If not, you may need to enable LFSCK debug on both the MDT and OST, then collect and analysis Lustre debug logs.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" class="">Got it, so the 30-sec interval is just for checking the status of the MDT.<br class=""></blockquote>Another question is, for layout checking, does lfsck need to compare metadata<br class="">stored in MDT (in LayoutEA) and metadata stored in OSTs (FID in LMA? not very<br class="">sure) for orphan objects? When are these metadata gathered into one place for<br class="">checking? I am asking this because previously I thought the periodically queries<br class="">from OSTs to MDT are doing this job.<br class=""><blockquote type="cite" class=""><br class=""></blockquote></blockquote><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">In short, all the MDT-object OST-object pairs have been marked during the 1st stage scanning. So If there are some OST-objects non-marked, then it may be orphans those will be handled during the 2nd phase scanning.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" class="">Thanks,<br class="">- Dong<br class=""><br class=""><br class=""><blockquote type="cite" class="">On May 1, 2018, at 9:49 PM, Yong, Fan <<a href="mailto:fan.yong@intel.com" class="">fan.yong@intel.com</a>> wrote:<br class=""><br class="">Inline comments.<br class=""><br class="">--<br class="">Cheers,<br class="">Nasf<br class=""><br class=""><br class=""><blockquote type="cite" class="">-----Original Message-----<br class="">From: lustre-discuss<br class="">[<a href="mailto:lustre-discuss-bounces@lists.lustre.org" class="">mailto:lustre-discuss-bounces@lists.lustre.org</a>] On Behalf Of 代栋<br class="">Sent: Wednesday, May 2, 2018 5:36 AM<br class="">To: <a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">Subject: [lustre-discuss] Is there a way to have faster lustre file<br class="">system checker (lfsck)?<br class=""><br class="">Hi all,<br class=""><br class="">I am still new to Lustre, so please let me know if I should send<br class="">this message to devel-list.<br class=""><br class="">This week, I tried to run LFSCK over a very small cluster<br class="">configuration (1 mdt and<br class="">3 osts). In this Lustre, I used about 300K inodes. It took me<br class="">about 80 mins to finish a LFSCK run. And, more importantly, while I<br class="">am running LFSCK, on both MDT and OSTS, the CPU utilization is 100%,<br class=""></blockquote></blockquote></blockquote>taken by the lfsck thread.<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class="">Which version of Lustre and what is the LFSCK command line you used?<br class=""><br class=""><br class=""><blockquote type="cite" class="">I understand that lfsck is operating in an online mode, so it is<br class="">slow. But, I am wondering is there any way to accelerate this?<br class="">Especially if I am allowed to run it offline, for example, during weekly<br class=""></blockquote></blockquote></blockquote>maintenance.<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class="">Your slow is abnormal, not related with online. The LFSCK can NOT be run<br class=""></blockquote></blockquote>under offline mode.<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class=""><br class=""><blockquote type="cite" class=""><br class="">After checking the lfsck kernel logs, I noticed that in the phase2<br class="">scanning on OSTs, there is an 30 seconds interval between querying<br class="">the MDTs. I am wondering is there any reason to have this 30<br class="">seconds interval, and will lfsck on OSTs be faster if we remove such 30<br class=""></blockquote></blockquote></blockquote>seconds interval?<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class="">Normally, the master engine on the MDT will notify the LFSCK engine on the<br class=""></blockquote></blockquote>OST when the first phase done. But we can NOT guarantee that the LFSCK<br class="">engine on the MDT always alive during the LFSCK (may because of some failure,<br class="">or network trouble, or node crash, and so on), so in the 2nd phase scanning, if<br class="">the LFSCK engine on the OST does not receive the notification from the MDT, it<br class="">needs to query the LFSCK (on the MDT) status periodically. If the MDT finished<br class="">the 1st phase scanning earlier than OST, then there will be no such query.<br class="">Anyway, such query is NOT the reason of your slow LFSCK.<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class=""><br class=""><blockquote type="cite" class=""><br class="">Thanks,<br class="">- Dong<br class="">_______________________________________________<br class="">lustre-discuss mailing list<br class=""><a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a><br class="">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</blockquote></blockquote></blockquote></blockquote></div></blockquote></div><br class=""></div></body></html>