<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">> > Up to the point when you’ve experienced problems clearing records, had at any point Changelogs been consumed (read/cleared) in parallel?<o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">> Yes indeed!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">> There was parallel reading and clearing of the change logs. We had both robinhood as a daemon reading clearing and attempts to do the same manually for the same user at the same time.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal">Interesting… <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">>>> Could you please file a ticket in Jira about this, if it isn't already fixed.</span><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Bruno, if there is a ticket on this or if you create one, could you point it out to me?
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">:Andy <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Gibbins, Faye [mailto:Faye.Gibbins@cirrus.com] <br>
<b>Sent:</b> Friday, June 02, 2017 2:44<br>
<b>To:</b> Andy Moe <moea@cray.com>; lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> RE: [lustre-discuss] Changelog users failing to clear records in 2.8, can anyone help?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Hi Andy,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Yes indeed!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">There was parallel reading and clearing of the change logs. We had both robinhood as a daemon reading clearing and attempts to do the same manually for the same user at the same time.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">We simply didn’t know this would be an issues.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D">Faye<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Andy Moe [<a href="mailto:moea@cray.com">mailto:moea@cray.com</a>]
<br>
<b>Sent:</b> 01 June 2017 19:34<br>
<b>To:</b> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<b>Cc:</b> IT Software Systems - All <<a href="mailto:it_sw_sys_all@cirrus.com">it_sw_sys_all@cirrus.com</a>><br>
<b>Subject:</b> RE: [lustre-discuss] Changelog users failing to clear records in 2.8, can anyone help?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal">Faye,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Up to the point when you’ve experienced problems clearing records, had at any point Changelogs been consumed (read/cleared) in parallel? By you or anyone else? I’ve been doing a lot this in test (among other things), and I can somewhat
consistently corrupt the Changelog catalog and users file. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">========================<o:p></o:p></p>
<p class="MsoNormal">Andrew Moe<o:p></o:p></p>
<p class="MsoNormal">Storage Test Engineer<o:p></o:p></p>
<p class="MsoNormal">Cray, Inc.<o:p></o:p></p>
<p class="MsoNormal">Bloomington, MN USA<o:p></o:p></p>
<p class="MsoNormal">========================<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> lustre-discuss [<a href="mailto:lustre-discuss-bounces@lists.lustre.org">mailto:lustre-discuss-bounces@lists.lustre.org</a>]
<b>On Behalf Of </b>Colin Faber<br>
<b>Sent:</b> Thursday, June 01, 2017 12:38<br>
<b>To:</b> Dilger, Andreas <<a href="mailto:andreas.dilger@intel.com">andreas.dilger@intel.com</a>><br>
<b>Cc:</b> IT Software Systems - All <<a href="mailto:it_sw_sys_all@cirrus.com">it_sw_sys_all@cirrus.com</a>>;
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> Re: [lustre-discuss] Changelog users failing to clear records in 2.8, can anyone help?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">There seems to have been a few instances of this reported here on the list in the last few months, I don't recall the earlier versions of lustre, but we have also seen this in the wild for customer systems, so very likely a bug which results
in corruption of llog files.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-cf<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Jun 1, 2017 at 11:36 AM, Dilger, Andreas <<a href="mailto:andreas.dilger@intel.com" target="_blank">andreas.dilger@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal">On Jun 1, 2017, at 10:55, Faccini, Bruno <<a href="mailto:bruno.faccini@intel.com">bruno.faccini@intel.com</a>> wrote:<br>
><br>
> Hello,<br>
> According to the error msgs, looks like there is a corrupted plain-LLOG file for the ChangeLogs of MDT0. And unfortunately, neither e2fsck nor lfsck can help to recover in this case.<br>
<br>
Bruno,<br>
is this bug fixed in newer Lustre releases, or can something be done in the ChangeLog handling so that the ChangeLog can still be cleared in this case? I don't think we care if the record is invalid when it is being deleted... Could you please file a ticket
in Jira about this, if it isn't already fixed.<br>
<br>
Cheers, Andreas<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
> I think that to clear this situation you need to stop/umount this MDT and re-mount it as ldiskfs to move both changelog_users and changelog_catalog files to some alternate place/name (do not remove them!), umount ldiskfs, re-start/mount your MDT, re-run a
RBH full-scan, re-register a ChangeLog user.<br>
> Only side-effect doing so, can be the volume of orphan plain-LLOGs that will be kept consuming space on MDT. You should be able to identify them by running llog_reader tool over the saved/renamed old catalog file that will list you the references to all these
remaining plain-LLOGs, allowing you to find+remove them during a new ldiskfs-mount session.<br>
><br>
> Bruno.<br>
><br>
>> On Jun 1, 2017, at 4:09 PM, Gibbins, Faye <<a href="mailto:Faye.Gibbins@cirrus.com">Faye.Gibbins@cirrus.com</a>> wrote:<br>
>><br>
>> Hi,<br>
>><br>
>> We have 4 file systems on our lustre cluster. All have changelog users registered for robinhood to use.<br>
>><br>
>> We have discovered that a changelog user for one of the file systems is not catching up to its index. Manual runs of Robinhood fail to read any more records even though according to mdd/tools-MDT0000/changelog_users there are record to read!<br>
>><br>
>> Over time the change log had filled and the file system had become sluggish. Wiping the robinhood mysql and reinitializing robin hood with a full scan didn’t fix the issue and like I said above three other change logs from different file systems (on the
same MSG) are ok when used from the same robinhood instance.<br>
>><br>
>> What makes me think this is a lustre (and we are using 2.8 on ext4) problem is this (repeated) error we are getting in syslog:<br>
>><br>
>> [Wed May 31 14:06:59 2017] Lustre: 46400:0:(llog.c:530:llog_process_thread()) invalid length -420090294 in llog record for index 372672342/61708<br>
>> [Wed May 31 14:06:59 2017] LustreError: 46400:0:(mdd_device.c:261:llog_changelog_cancel()) tools-MDD0000: cancel idx 645 of catalog 0x7:10 rc=-22<br>
>><br>
>> Deregistering the user from the change log and starting with a new one has not changed the behaviour and we still can’t use this new user to track changes to the file system.<br>
>><br>
>> Can anyone offer any advice on how to resolve this issue in the changelog?<br>
>> If not can anyone confirm if taking the file system down for a e2fsck/lfsck will fix issues with the changelog? I’d settle for being able to clear the whole log and starting afresh if that’s possible?<br>
>><br>
>> Yours<br>
>> Faye Gibbins<br>
>> Snr SysAdmin, Unix Lead Architect<br>
>> Software Systems and Cloud Services<br>
>> Cirrus Logic | <a href="http://cirrus.com" target="_blank">cirrus.com</a> | <a href="tel:%2B44%20%280%29%20131%20272%207398">
+44 (0) 131 272 7398</a><br>
>><br>
>> <image001.png><br>
>><br>
>> This message and any attachments may contain privileged and confidential information that is intended solely for the person(s) to whom it is addressed. If you are not an intended recipient you must not: read; copy; distribute; discuss; take any action in
or make any reliance upon the contents of this message; nor open or read any attachment. If you have received this message in error, please notify us as soon as possible on the following telephone number and destroy this message including any attachments.
Thank you. Cirrus Logic International (UK) Ltd and Cirrus Logic International Semiconductor Ltd are companies registered in Scotland, with registered numbers SC089839 and SC495735 respectively. Our registered office is at 7B Nightingale Way, Quartermile, Edinburgh,
EH3 9EG, UK. Tel: <a href="tel:%2B44%20%280%29131%20272%207000">+44 (0)131 272 7000</a>. cirrus.com_______________________________________________<br>
>> lustre-discuss mailing list<br>
>> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
>> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwMGaQ&c=O3LcjD-V2Iepl5V0N1424A&r=uYX9IusTj04zoAQZpNJuM5SyJ7Npy_e1PVoo89nWUJ0&m=KB7BUUMX-CEDP0QEDKkP2ayfbxLj9o-1b_9y7BDM89Y&s=kVghLuYTnEEstWLiaPz93MKoU6b82UpmFIMos3MyC0o&e=" target="_blank">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
><br>
> ---------------------------------------------------------------------<br>
> Intel Corporation SAS (French simplified joint stock company)<br>
> Registered headquarters: "Les Montalets"- 2, rue de Paris,<br>
> 92196 Meudon Cedex, France<br>
> Registration Number: 302 456 199 R.C.S. NANTERRE<br>
> Capital: 4,572,000 Euros<br>
><br>
> This e-mail and any attachments may contain confidential material for<br>
> the sole use of the intended recipient(s). Any review or distribution<br>
> by others is strictly prohibited. If you are not the intended<br>
> recipient, please contact the sender and delete all copies.<br>
><br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwMGaQ&c=O3LcjD-V2Iepl5V0N1424A&r=uYX9IusTj04zoAQZpNJuM5SyJ7Npy_e1PVoo89nWUJ0&m=KB7BUUMX-CEDP0QEDKkP2ayfbxLj9o-1b_9y7BDM89Y&s=kVghLuYTnEEstWLiaPz93MKoU6b82UpmFIMos3MyC0o&e=" target="_blank">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">Cheers, Andreas<br>
--<br>
Andreas Dilger<br>
Lustre Principal Architect<br>
Intel Corporation<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=DwMGaQ&c=O3LcjD-V2Iepl5V0N1424A&r=uYX9IusTj04zoAQZpNJuM5SyJ7Npy_e1PVoo89nWUJ0&m=KB7BUUMX-CEDP0QEDKkP2ayfbxLj9o-1b_9y7BDM89Y&s=kVghLuYTnEEstWLiaPz93MKoU6b82UpmFIMos3MyC0o&e=" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>