<div dir="auto"><div dir="auto"><br></div><div>I am no sure if you hit the same bug as in our case: the llog was not cleared several times and filed the whole mdt space, but the upgrade from 2.8.x to 2.9 resolved the log clear problem.</div><div dir="auto"><br></div><div dir="auto"><br><br><div class="gmail_quote" dir="auto"><div dir="ltr">Am Fr., 25. Jan. 2019, 07:16 hat Colin Faber <<a href="mailto:cfaber@gmail.com">cfaber@gmail.com</a>> geschrieben:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Have you tried manually purging the changelog files and catalog then restarting by re-registering? Also, are you sure that _all_ consumers are requesting to clear the records?<br></div><br><div class="gmail_quote"><div dir="ltr" class="m_4834353573405065433gmail_attr">On Mon, Jan 7, 2019 at 11:40 AM <a href="mailto:nanava@luis.uni-hannover.de" target="_blank" rel="noreferrer">nanava@luis.uni-hannover.de</a> <<a href="mailto:nanava@luis.uni-hannover.de" target="_blank" rel="noreferrer">nanava@luis.uni-hannover.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
Any advice here what could be done to avoid storage overflow?. As only a few gb is left on mdt, we plan to resize the storage device, though this seems to be not a permanent solution ..<br>
<br>
Thank you.<br>
<br>
> <br>
> has this problem been resolved? We experience the same issue when we can't clear the changelog records and as a result MDT is gradually running out of space.<br>
> Will update to 2.10.x on MDS resolve the issue ?<br>
> <br>
> Thanks.<br>
> <br>
> Regards<br>
> -----<br>
> Gizo Nanava<br>
> Leibniz Universitaet IT Services<br>
> Leibniz Universitaet Hannover<br>
> Schlosswender Str. 5<br>
> D-30159 Hannover<br>
> Tel +49 511 762 7919085<br>
> <a href="http://www.luis.uni-hannover.de" rel="noreferrer noreferrer" target="_blank">http://www.luis.uni-hannover.de</a><br>
> <br>
> On Thursday, June 1, 2017 16:09 CEST, "Gibbins, Faye" <<a href="mailto:Faye.Gibbins@cirrus.com" target="_blank" rel="noreferrer">Faye.Gibbins@cirrus.com</a>> wrote: <br>
> <br>
> > Hi,<br>
> > <br>
> > We have 4 file systems on our lustre cluster. All have changelog users registered for robinhood to use.<br>
> > <br>
> > We have discovered that a changelog user for one of the file systems is not catching up to its index. Manual runs of Robinhood fail to read any more records even though according to mdd/tools-MDT0000/changelog_users there are record to read!<br>
> > <br>
> > Over time the change log had filled and the file system had become sluggish. Wiping the robinhood mysql and reinitializing robin hood with a full scan didn't fix the issue and like I said above three other change logs from different file systems (on the same MSG) are ok when used from the same robinhood instance.<br>
> > <br>
> > What makes me think this is a lustre (and we are using 2.8 on ext4) problem is this (repeated) error we are getting in syslog:<br>
> > <br>
> > [Wed May 31 14:06:59 2017] Lustre: 46400:0:(llog.c:530:llog_process_thread()) invalid length -420090294 in llog record for index 372672342/61708<br>
> > [Wed May 31 14:06:59 2017] LustreError: 46400:0:(mdd_device.c:261:llog_changelog_cancel()) tools-MDD0000: cancel idx 645 of catalog 0x7:10 rc=-22<br>
> > <br>
> > Deregistering the user from the change log and starting with a new one has not changed the behaviour and we still can't use this new user to track changes to the file system.<br>
> > <br>
> > Can anyone offer any advice on how to resolve this issue in the changelog?<br>
> > If not can anyone confirm if taking the file system down for a e2fsck/lfsck will fix issues with the changelog? I'd settle for being able to clear the whole log and starting afresh if that's possible?<br>
> > <br>
> > Yours<br>
> > Faye Gibbins<br>
> > Snr SysAdmin, Unix Lead Architect<br>
> > Software Systems and Cloud Services<br>
> > Cirrus Logic | <a href="http://cirrus.com" rel="noreferrer noreferrer" target="_blank">cirrus.com</a><<a href="http://www.cirrus.com/" rel="noreferrer noreferrer" target="_blank">http://www.cirrus.com/</a>> | +44 (0) 131 272 7398<br>
> > <br>
> > [cid:image002.png@01D2CF24.9A35B8F0]<br>
> > <br>
> > This message and any attachments may contain privileged and confidential information that is intended solely for the person(s) to whom it is addressed. If you are not an intended recipient you must not: read; copy; distribute; discuss; take any action in or make any reliance upon the contents of this message; nor open or read any attachment. If you have received this message in error, please notify us as soon as possible on the following telephone number and destroy this message including any attachments. Thank you. Cirrus Logic International (UK) Ltd and Cirrus Logic International Semiconductor Ltd are companies registered in Scotland, with registered numbers SC089839 and SC495735 respectively. Our registered office is at 7B Nightingale Way, Quartermile, Edinburgh, EH3 9EG, UK. Tel: +44 (0)131 272 7000. <a href="http://cirrus.com" rel="noreferrer noreferrer" target="_blank">cirrus.com</a><br>
> <br>
> <br>
> <br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br>
<br>
<br>
-- <br>
_______________________<br>
Dr. Gizo Nanava<br>
Leibniz Universitaet IT Services<br>
Leibniz Universitaet Hannover<br>
Schlosswender Str. 5<br>
D-30159 Hannover<br>
Tel +49 511 762 7919085<br>
<a href="http://www.luis.uni-hannover.de" rel="noreferrer noreferrer" target="_blank">http://www.luis.uni-hannover.de</a><br>
<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" rel="noreferrer">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div></div></div>