<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<br>
Julien,<br>
<br>
Could you share the details (LBUG plus full back trace, primarily) with the list? It would be good to know if it’s a known problem or not.<br>
<br>
Thanks!<br>
<br>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Julien Rey <julien.rey@univ-paris-diderot.fr><br>
<b>Sent:</b> Monday, December 17, 2018 3:40:56 AM<br>
<b>To:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] no more free slots in catalog</font>
<div> </div>
</div>
<div style="background-color:#FFFFFF">
<div class="x_moz-cite-prefix">Le 11/12/2018 15:47, <a class="x_moz-txt-link-abbreviated" href="mailto:quentin.bouget@cea.fr">
quentin.bouget@cea.fr</a> a écrit :<br>
</div>
<blockquote type="cite">
<div class="x_moz-cite-prefix">Le 11/12/2018 à 15:32, Julien Rey a écrit :<br>
</div>
<blockquote type="cite">
<div class="x_moz-cite-prefix">Le 11/12/2018 14:13, <a class="x_moz-txt-link-abbreviated" href="mailto:quentin.bouget@cea.fr">
quentin.bouget@cea.fr</a> a écrit :<br>
</div>
<blockquote type="cite">
<div class="x_moz-cite-prefix">Le 11/12/2018 à 10:28, Julien Rey a écrit :<br>
</div>
<blockquote type="cite">
<div class="x_moz-cite-prefix">Le 10/12/2018 13:33, <a class="x_moz-txt-link-abbreviated" href="mailto:quentin.bouget@cea.fr">
quentin.bouget@cea.fr</a> a écrit :<br>
</div>
<blockquote type="cite">
<div class="x_moz-cite-prefix">Le 10/12/2018 à 12:00, Julien Rey a écrit :<br>
</div>
<blockquote type="cite">Hello, <br>
<br>
We are running lustre 2.8.0-RC5--PRISTINE-2.6.32-573.12.1.el6_lustre.x86_64. <br>
<br>
Since thursday we are getting a "bad address" error when trying to write on the lustre volume.
<br>
<br>
Looking at the logs on the MDS, we are getting this kind of messages : <br>
<br>
Dec 10 06:26:18 localhost kernel: Lustre: 9593:0:(llog_cat.c:93:llog_cat_new_log()) lustre-MDD0000: there are no more free slots in catalog
<br>
Dec 10 06:26:18 localhost kernel: Lustre: 9593:0:(llog_cat.c:93:llog_cat_new_log()) Skipped 45157 previous similar messages
<br>
Dec 10 06:26:18 localhost kernel: LustreError: 9593:0:(mdd_dir.c:887:mdd_changelog_ns_store()) lustre-MDD0000: cannot store changelog record: type = 6, name = 'PEPFOLD-00016_bestene1-mc-SC-min-grompp.log', t = [0x20000a58f:0x858e:0x0], p = [0x20000a57d:0x17fd9:0x0]:
rc = -28 <br>
Dec 10 06:26:18 localhost kernel: LustreError: 9593:0:(mdd_dir.c:887:mdd_changelog_ns_store()) Skipped 45157 previous similar messages
<br>
<br>
<br>
I saw here that this issue was supposed to be solved in 2.8.0: <br>
<a class="x_moz-txt-link-freetext" href="https://jira.whamcloud.com/browse/LU-6556">https://jira.whamcloud.com/browse/LU-6556</a>
<br>
<br>
Could someone help us unlocking this situation ? <br>
<br>
Thanks. <br>
<br>
</blockquote>
<p>Hello,</p>
<p>The log messages don't point at a "bad address" issue but rather at a "no space left on device" one ("rc = -28" --> -ENOSPC).</p>
<p>You most likely have, at some point, registered a changelog user on your mds and that user is not consuming changelogs.</p>
<p>You can check this by running:</p>
<pre>[mds0]# lctl get_param mdd.*.changelog_users
mdd.lustre-MDT0000.changelog_users=
current index: 3
ID index
cl1 0
</pre>
<p>The most important thing to look for is the distance between "current index" and the index for "cl1", "cl2", ...<br>
I expect for at least one changelog user, that distance is 2^32 (the maximum number of changelog records).<br>
Note that changelog indexes wrap around (0, 1, 2, ..., 4294967295, 0, 1, ...).</p>
<p>If I am right, then you can either deregister the changelog user:</p>
<pre>[mds0]# lctl --device lustre-MDT0000 changelog_deregister cl1
</pre>
<p>or acknowledge the records:</p>
<pre>[client]# lfs changelog_clear lustre-MDT0000 cl1 0
</pre>
<p>(clearing with index 0 is a shortcut for "acknowledge every changelog records")</p>
<p>Both those options may take a while.</p>
<p>There is a third one that might yield faster result, but it is also much more dangerous to use (you might want to check with your support first) :<br>
</p>
<pre>[mds0]# umount /dev/mdt0
[mds0]# mount -t ldiskfs /dev/mdt0 /mnt/lustre-mdt0
[mds0]# rm /mnt/lustre-mdt0/changelog_catalog
[mds0]# rm /mnt/lustre-mdt0/changelog_users
[mds0]# umount /dev/mdt0
[mds0]# mount -t lustre /dev/mdt0 <...> # remount the mdt where it was
</pre>
<p><b>I cannot garantee this will not trash your filesystem. Use at your own risk.<br>
</b></p>
<p>---</p>
<p>In recent versions (2.12, maybe even 2.10), lustre comes with a builtin garbage collector for slow/inactive changelog users.</p>
<p>Regards,<br>
Quentin Bouget<br>
</p>
</blockquote>
<br>
Hello Quentin,<br>
<br>
Many thanks for your quick reply.<br>
<br>
This is what I got when I issued the command you suggested:<br>
<br>
<pre>[root@lustre-mds]# lctl get_param mdd.*.changelog_users</pre>
<pre>mdd.lustre-MDT0000.changelog_users=</pre>
<pre>current index: 4160462682</pre>
<pre>ID index</pre>
<pre>cl1 21020582</pre>
<br>
I then issued the following command:<br>
<pre>[root@lustre-mds]# lctl --device lustre-MDT0000 changelog_deregister cl1</pre>
<br>
It's been running for almost 20 hours now. Do you have an estimation of the time it could take ?<br>
</blockquote>
When you deregister a changelog user: every changelog record has to be invalidated (maybe this is batched, but I don't know enough about the on-disk structure to say).<br>
<br>
I do not recall ever waiting that long. Then again, I never personally deregistered a changelog users with that many pending changelog records.<br>
</blockquote>
<br>
The changelog_deregister command still hasn't finished yet. Is there any way to track the state of the purge of records ?<br>
</blockquote>
I believe there is an "llog_reader" implemented in the lustre sources, but I never really used it.<br>
<br>
<blockquote type="cite">
<blockquote type="cite">If you just want to make sure Lustre is doing something, you can have a look at your mdt0: invalidating changelog records should generate a high load of small random writes.<br>
If the device is idle, something is probably wrong.<br>
</blockquote>
<br>
Hard to tell. iostat doesn't show much I/O.<br>
<br>
<blockquote type="cite">Is your filesystem still unavailable?<br>
</blockquote>
<br>
The following command doesn't show any registered changelog user:<br>
<br>
cat /proc/fs/lustre/mdd/lustre-MDT0000/changelog_users<br>
<br>
I tried to mount the lustre volume on a client. I don't get the "Bad Address" error anymore.<br>
<br>
Best,<br>
</blockquote>
Yes, I don't think you need to wait for "changelog_deregister" to complete to start using your filesystem again.<br>
If there is not any changelog user registered on your system, changelog records are not emitted.<br>
<br>
Just try to wait until "changelog_deregister" completes before registering a new changelog user. =)<br>
<br>
<blockquote type="cite"><br>
<blockquote type="cite"><br>
<blockquote type="cite"><br>
Best,<br>
<pre class="x_moz-signature" cols="72">--
Julien REY
Plate-forme RPBS
Molécules Thérapeutiques In Silico (MTi)
Université Paris Diderot - Paris VII
tel : 01 57 27 83 95 </pre>
<br>
<fieldset class="x_mimeAttachmentHeader"></fieldset> <br>
<pre>_______________________________________________
lustre-discuss mailing list
<a class="x_moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a class="x_moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
Quentin<br>
</blockquote>
<br>
<br>
<pre class="x_moz-signature" cols="72">--
Julien REY
Plate-forme RPBS
Molécules Thérapeutiques In Silico (MTi)
Université Paris Diderot - Paris VII
tel : 01 57 27 83 95 </pre>
<br>
<fieldset class="x_mimeAttachmentHeader"></fieldset> <br>
<pre>_______________________________________________
lustre-discuss mailing list
<a class="x_moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a class="x_moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
<p><br>
Cheers,<br>
Quentin</p>
</blockquote>
<br>
The changelog_deregister command eventually ended with an LBUG after a few days. The mds rebooted automatically. But at least it's working fine now. :)<br>
<br>
Many thanks for your help.<br>
<br>
Cheers,<br>
<br>
<pre class="x_moz-signature" cols="72">--
Julien REY
Plate-forme RPBS
Molécules Thérapeutiques In Silico (MTi)
Université Paris Diderot - Paris VII
tel : 01 57 27 83 95 </pre>
</div>
</body>
</html>