<div dir="ltr"><div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><span class="im">By the way, are the llog files you mentioned virtual or real? if they are
real, where are they located? Need I clean them manually ?</span></blockquote><div>They are real, the location is O/1/...<br></div> lustre/utils/llog_reader ./changelog_catalog.dmp <br></div><div><div style="margin-left:40px">rec #1 type=1064553b len=64<br>Header size : 8192<br>Time : Mon Dec 7 15:44:37 2015<br>Number of records: 1<br>Target uuid : <br>-----------------------<br>#01 (064)ogen=0 name=0x8:1<br></div>...<br>I`ve dump and check file, location base at name from record.<br>debugfs: dump O/1/d8/8 plain.llog<br></div>lustre/utils/llog_reader ./plain.llog <br><div><div style="margin-left:40px">rec #1 type=10660000 len=96 offset 8192<br>Header size : 8192<br>Time : Mon Dec 7 15:46:40 2015<br>Number of records: 1<br>Target uuid : <br>-----------------------<br>#01 (096)changelog record id:0x0 cr_flags:0x1000 cr_type:CREAT(0x1)<br></div>Looks like O/1/ for llog files only.<br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 7, 2015 at 4:55 AM, wanglu <span dir="ltr"><<a href="mailto:wanglu@ihep.ac.cn" target="_blank">wanglu@ihep.ac.cn</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div style="MARGIN:10px">
<div>
<div style="FONT-FAMILY:Verdana">Hi Alexander, </div>
<div style="FONT-FAMILY:Verdana"> </div>
<div style="FONT-FAMILY:Verdana">Before I recieved this reply, I deregistered
the cl1 user. It took a very long time, and I am not sure if it successfully
finished or not since the server crashed once the next morning. </div><span class="">
<div style="FONT-FAMILY:Verdana">Then, I moved the old
changelog_catalog file, and created a zero changelog_user file instead.
</div>
<div style="FONT-FAMILY:Verdana">This is what I got from the old
changelog_catalog file. </div>
<div style="FONT-FAMILY:Verdana">
<div>
<div style="TEXT-INDENT:2em"># ls -l /tmp/changelog.dmp </div>
<div> -rw-r--r-- 1 root root 4153280 Dec 6 06:54
/tmp/changelog.dmp</div></div>
<div> # llog_reader changelog.dmp |grep "type=1064553b"
|wc -l </div></div>
</span><div style="FONT-FAMILY:Verdana"><span class="">
<div> 63432</div>
<div>This number is smaller than 64768, I am not sure if it is related to
the unfinished deregisteration or not. </div>
<div> </div>
<div>The first record number is 1, the last record number of is 64767. I
think there maybe some skipped record numbers: </div>
</span><div>
<div> # llog_reader changelog.dmp |grep "type=1064553b"
|head -n 1 </div>
<div> rec #1 type=1064553b len=64</div>
<div> # llog_reader changelog.dmp |grep "type=1064553b"
|tail -n 1 </div>
<div> rec #64767 type=1064553b len=64</div></div><span class="">
<div> # llog_reader changelog.dmp |grep "^rec" | grep -v
"type=1064553b" </div>
<div style="TEXT-INDENT:2em">return 0 lines. </div>
<div> </div>
</span><span class=""><div>By the way, are the llog files you mentioned virtual or real? if they are
real, where are they located? Need I clean them manually ?</div>
<div> </div>
<div>Thanks, </div>
<div>Lu,Wang</div></span></div></div><span class="">
<div style="BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0cm;PADDING-LEFT:0cm;PADDING-RIGHT:0cm;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<div style="PADDING-BOTTOM:8px;PADDING-LEFT:8px;PADDING-RIGHT:8px;FONT-FAMILY:tahoma;BACKGROUND:#efefef;COLOR:#000000;FONT-SIZE:12px;PADDING-TOP:8px">
<div><b>From:</b> <a href="mailto:alexander.boyko@seagate.com" target="_blank">Alexander
Boyko</a></div>
<div><b>Date:</b> 2015-12-04 21:36</div>
<div><b>To:</b> <a href="mailto:wanglu@ihep.ac.cn" target="_blank">wanglu</a>; <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss</a></div>
<div><b>Subject:</b> RE [lustre-discuss] No free catalog slots for log (
Lustre 2.5.3 & Robinhood 2.5.3 )</div></div></div>
</span><div>
<div style="BACKGROUND-COLOR:white">
<div dir="ltr">
<blockquote style="BORDER-LEFT:rgb(204,204,204) 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">Here are 4 questions which we cannot find answers in
LU-1586:<div><div class="h5"><br>1. According to Andres?s reply, there
should some unconsumed changelog files on our MDT, and these files have taken
all the space (file quotas?) Lustre gives to changelog. With Lustre 2.1, these
files are under OBJECTS directory and can be listed in ldiskfs mode. In our
case, with Lustre 2.5.3, there is no OBJECTS directory can be found. In this
case, how can we monitor the situation before the unconsumed changelogs takes
up all the disk space?<br></div></div></blockquote><div><div class="h5">
<div>The changelog base on one catalog file and a plain llog files. Catalog
stores limited number of records about 64768. A catalog record size is 64 byte.
Each record has information about plain llog file. A plain llog file stores
records about IO operation. A number of records at the plain llog file is about
64768 with different record size. So changelog could store 64768^2 IO operations
and it occupy filesystem space. The error "no free catalog slots" is happened
when changelog catalog doesn`t have a slot to store a record about new plain
lllog. All slots are filled or internal changelog markers became crazy and
internal logic don`t work.<br></div>
<div>To be closer to the root cause, you need to dump a changelog catalog and
check bitmap. Is there free slots? Something like<br><pre>debugfs -R "dump changelog_catalog changelog_catalog.dmp" /dev/md55 &&
used=`llog_reader changelog_catalog.dmp | grep "type=1064553b" | wc -l` <br></pre></div>
<blockquote style="BORDER-LEFT:rgb(204,204,204) 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">2. Why there are so many
unconsumed changelogs? Could it related to our frequent remount of MDT(
abort_recovery mode )?<br></blockquote>
<div>umount operation create half empty plain llog file. And changelog_clear
can`t remove it, if all slots is freed. Only new mount can remove that file. It
could be related or not.</div>
<div><br> </div>
<blockquote style="BORDER-LEFT:rgb(204,204,204) 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">3. When we remount the MDT, robinhood is still
running. Why robinhood can not consume those old changelogs after MDT service
is recovered?<br>4. Why there is a huge difference between current
index(4199610352 ) and cl1(49035933) index?<br><br>Thank you for your time and
help !<br><br>Wang,Lu<br clear="all"></blockquote><br>-- <br>
<div>
<div dir="ltr">Alexander Boyko
<div>Seagate</div><a href="http://www.seagate.com" target="_blank">www.seagate.com</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr">Alexander Boyko<div>Seagate</div>
<a href="http://www.seagate.com" target="_blank">www.seagate.com</a><br></div></div>
</div>