[lustre-discuss] Removing stale files

[Andreas Dilger]

On May 31, 2022, at 13:01, William D. Colburn <wcolburn at nrao.edu<mailto:wcolburn at nrao.edu>> wrote:

We had a filesystem corruption back in February, and we've been trying
to salvage things since then.  I've spent the past month slowly draining
the corrupt OST, and over the weekend it finally finished.  An lfs find
on the filesystem says that thhow ere are no files stored on that OST.  The
OST is 100% full, and if I mount it as an ldiskfs I can see a little
over five millions files in O/*/*.

How did you drain the OST?  Was the OST totally deactivated, or "max_create_count=0"?
If it was deactivated, then this will prevent OST objects from being destroyed when the
MDT inode is deleted.

Most of them have numbers as names, and some of them are named LAST_ID.

This is normal.  The number is the object ID.  You can check the OST objects with
ll_decode_filter_fid (when mounted as type ldiskfs) to report the parent MDT FID
that the object belongs/belonged to.  Then "lfs fid2path" can be used to check if the
file still exists and/or if the OST object is still part of the layout (which it should not be).

All of the numbered files seem to be user data, with owners, and real data in them
(based on ls and the find command)

I would like to clean out this OST and readd it to lustre, but I'm
unsure of how to best approach this.  I see several options:

OPTION ONE: run lfsck against the entire filesystem with the full and
previously corrupt OST mounted.

This is unnecessary, since these objects are not used.   With the "--orphan"
option it will link the OST objects into $mount/.lustre/lost+found where they
could be deleted, essentially the same as #4 below, but it is easier to just
delete the objects directly if you know they are not needed.

OPTION TWO: run lfsck against only the corrupt OST in the hopes that
cleans up all of the orphans on that OST.

This won't help, since the OST LFSCK would not attach the object into the
visible namespace, so it won't really change the situation.

OPTION THREE: mounted as ldiskfs remove O/*/[1234567890]*[1234567890]
and then remount the file system.

This would be one option.  Note that with DNE the OST object names will be
in hex, so the above regexp would not catch all objects.

OPTION FOUR: newfs the bad OST and readd it losing the old index.

This would also work, if you use the "--replace" option when formatting.

We tried option one once before, and it killed cluster jobs because it
made files unreadable while they were in use.  Option two might avoid
that since it would not be affecting existing files.  Option three
sounds like it will work based on my limited knowledge of how lustre
works, and would probably be the most expedient method.  Option four is
annoying because it leaves a hole in the lustre that is upsetting to our
OCD tendencies.

Any and all advice is appreciated here.  Thank you.

--Schlake
 Sysadmin IV, NRAO
 Work: 575-835-7281 (BACK IN THE OFFICE!)
 Cell: 575-517-5668 (out of work hours)
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220609/ffbbd273/attachment-0001.htm>