[lustre-discuss] Changelog record cleanup in /O/1/d*

[Cory Spitz]

Craig, FWIW, this sounds a lot like https://jira.hpdd.intel.com/browse/LU-5038, which was addressed in 2.7.0.
-Cory

--


From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of "Prescott,Craig P" <prescott at rc.ufl.edu>
Date: Monday, December 5, 2016 at 3:02 PM
To: "lustre-discuss at lists.lustre.org" <lustre-discuss at lists.lustre.org>
Subject: [lustre-discuss] Changelog record cleanup in /O/1/d*




We were running 2.5.3.90 with changelogs enabled earlier this summer.  We ran into a catalog corruption issue (LU-6556) - we decided to deregister our changelog users, move the CONFIGS/changelog_{catalog,users} files out of the way, and carry on until we had an opportunity to upgrade.  We did not remove anything from /O/1/d* at that time (though we probably should have).



We've observed that mounting our MDT can take several-to-many minutes - I can see with iostat that the MDT is very busy with reads while it is being mounted.  I suspect that those stale files in /O/1/d* are the reason (there are lots of them), as they are processed by the OSP sync at MDT startup.   I looked with debugfs at the /O/1/d* directories - there are 1000s of files and their timestamps are consistent with when we were using changelogs.  I dumped a few randomly selected ones and checked with llog_reader that the types of records they contain are CHANGELOG_REC (type=10660000).



At the least, I think we should to remove the files in /O/1/d* that contain CHANGELOG_REC entries.  Can I just delete every file in /O/1/d*, or do I need to be careful and only remove the CHANGELOG_REC entries?



The reason I ask is that I do see a handful of files that are not changelog-related in these directories - their timestamps are newer and their record type as reported by llog_reader is not CHANGELOG_REC or CHANGELOG_USER.  There are only a small number of such files, though.



Thanks,

Craig Prescott
University of Florida Research Computing
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20161205/ef297ab1/attachment.htm>