<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=UTF-8" http-equiv=Content-Type>
<STYLE>
BLOCKQUOTE {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px; MARGIN-LEFT: 2em
}
OL {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
UL {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
DIV.FoxDiv20151207095147638403 {
COLOR: #000000
}
P {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
BODY {
LINE-HEIGHT: 1.5; FONT-FAMILY: 微软雅黑; COLOR: #000000; FONT-SIZE: 10.5pt
}
</STYLE>
<META name=GENERATOR content="MSHTML 8.00.7601.19038"></HEAD>
<BODY style="MARGIN: 10px">
<DIV>
<DIV style="FONT-FAMILY: Verdana">Hi Alexander, </DIV>
<DIV style="FONT-FAMILY: Verdana"> </DIV>
<DIV style="FONT-FAMILY: Verdana">Before I recieved this reply, I deregistered
the cl1 user. It took a very long time, and I am not sure if it successfully
finished or not since the server crashed once the next morning. </DIV>
<DIV style="FONT-FAMILY: Verdana">Then, I moved the old
changelog_catalog file, and created a zero changelog_user file instead.
</DIV>
<DIV style="FONT-FAMILY: Verdana">This is what I got from the old
changelog_catalog file. </DIV>
<DIV style="FONT-FAMILY: Verdana">
<DIV>
<DIV style="TEXT-INDENT: 2em"># ls -l /tmp/changelog.dmp </DIV>
<DIV> -rw-r--r-- 1 root root 4153280 Dec 6 06:54
/tmp/changelog.dmp</DIV></DIV>
<DIV> # llog_reader changelog.dmp |grep "type=1064553b"
|wc -l </DIV></DIV>
<DIV style="FONT-FAMILY: Verdana">
<DIV> 63432</DIV>
<DIV>This number is smaller than 64768, I am not sure if it is related to
the unfinished deregisteration or not. </DIV>
<DIV> </DIV>
<DIV>The first record number is 1, the last record number of is 64767. I
think there maybe some skipped record numbers: </DIV>
<DIV>
<DIV> # llog_reader changelog.dmp |grep "type=1064553b"
|head -n 1 </DIV>
<DIV> rec #1 type=1064553b len=64</DIV>
<DIV> # llog_reader changelog.dmp |grep "type=1064553b"
|tail -n 1 </DIV>
<DIV> rec #64767 type=1064553b len=64</DIV></DIV>
<DIV> # llog_reader changelog.dmp |grep "^rec" | grep -v
"type=1064553b" </DIV>
<DIV style="TEXT-INDENT: 2em">return 0 lines. </DIV>
<DIV> </DIV>
<DIV>By the way, are the llog files you mentioned virtual or real? if they are
real, where are they located? Need I clean them manually ?</DIV>
<DIV> </DIV>
<DIV>Thanks, </DIV>
<DIV>Lu,Wang</DIV></DIV></DIV>
<DIV
style="BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0cm; PADDING-LEFT: 0cm; PADDING-RIGHT: 0cm; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<DIV
style="PADDING-BOTTOM: 8px; PADDING-LEFT: 8px; PADDING-RIGHT: 8px; FONT-FAMILY: tahoma; BACKGROUND: #efefef; COLOR: #000000; FONT-SIZE: 12px; PADDING-TOP: 8px">
<DIV><B>From:</B> <A href="mailto:alexander.boyko@seagate.com">Alexander
Boyko</A></DIV>
<DIV><B>Date:</B> 2015-12-04 21:36</DIV>
<DIV><B>To:</B> <A href="mailto:wanglu@ihep.ac.cn">wanglu</A>; <A
href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss</A></DIV>
<DIV><B>Subject:</B> RE [lustre-discuss] No free catalog slots for log (
Lustre 2.5.3 & Robinhood 2.5.3 )</DIV></DIV></DIV>
<DIV>
<DIV style="BACKGROUND-COLOR: white" class=FoxDiv20151207095147638403>
<DIV dir=ltr>
<BLOCKQUOTE
style="BORDER-LEFT: rgb(204,204,204) 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex"
class=gmail_quote>Here are 4 questions which we cannot find answers in
LU-1586:<BR>1. According to Andres?s reply, there
should some unconsumed changelog files on our MDT, and these files have taken
all the space (file quotas?) Lustre gives to changelog. With Lustre 2.1, these
files are under OBJECTS directory and can be listed in ldiskfs mode. In our
case, with Lustre 2.5.3, there is no OBJECTS directory can be found. In this
case, how can we monitor the situation before the unconsumed changelogs takes
up all the disk space?<BR></BLOCKQUOTE>
<DIV>The changelog base on one catalog file and a plain llog files. Catalog
stores limited number of records about 64768. A catalog record size is 64 byte.
Each record has information about plain llog file. A plain llog file stores
records about IO operation. A number of records at the plain llog file is about
64768 with different record size. So changelog could store 64768^2 IO operations
and it occupy filesystem space. The error "no free catalog slots" is happened
when changelog catalog doesn`t have a slot to store a record about new plain
lllog. All slots are filled or internal changelog markers became crazy and
internal logic don`t work.<BR></DIV>
<DIV>To be closer to the root cause, you need to dump a changelog catalog and
check bitmap. Is there free slots? Something like<BR><PRE>debugfs -R "dump changelog_catalog changelog_catalog.dmp" /dev/md55 &&
used=`llog_reader changelog_catalog.dmp | grep "type=1064553b" | wc -l` <BR></PRE></DIV>
<BLOCKQUOTE
style="BORDER-LEFT: rgb(204,204,204) 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex"
class=gmail_quote>2. Why there are so many
unconsumed changelogs? Could it related to our frequent remount of MDT(
abort_recovery mode )?<BR></BLOCKQUOTE>
<DIV>umount operation create half empty plain llog file. And changelog_clear
can`t remove it, if all slots is freed. Only new mount can remove that file. It
could be related or not.</DIV>
<DIV><BR> </DIV>
<BLOCKQUOTE
style="BORDER-LEFT: rgb(204,204,204) 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex"
class=gmail_quote>3. When we remount the MDT, robinhood is still
running. Why robinhood can not consume those old changelogs after MDT service
is recovered?<BR>4. Why there is a huge difference between current
index(4199610352 ) and cl1(49035933) index?<BR><BR>Thank you for your time and
help !<BR><BR>Wang,Lu<BR clear=all></BLOCKQUOTE><BR>-- <BR>
<DIV class=gmail_signature>
<DIV dir=ltr>Alexander Boyko
<DIV>Seagate</DIV><A href="http://www.seagate.com"
target=_blank>www.seagate.com</A><BR></DIV></DIV></DIV></DIV></DIV></BODY></HTML>