[lustre-discuss] Need advice for fixing a file system

[William D. Colburn]

The topic is just too broad for a single subject line, sorry, and has no
TL;DNR summary.

Our lustre servers are running lustre-2.10.8 under RHEL 7.6.  We have 12
OSSes, for a 2P filesystem.  I know it will be advised, but we are not
planning to upgrade the lustre version until 2024.

Our story starts about a month ago.  A human error caused an OSS to be
powered off while operational.  When it came back up one of the four
OSTs reported an error, and wouldn't mount.  We ran an e2fsck on that one
filesystem and it all seemed happy for about a month.

We are noticing now that a small handful of files are hanging on the
OSS.  We noticed three, and the syslogs tell us six.

cat /var/log/messages /var/log/messages.? | grep "error destroying
object"| awk ' { print $12 } ' | sort -nr | uniq -c
    881 [0x1000e0000:0x6190fc5:0x0]:
   1069 [0x1000e0000:0x6187f4d:0x0]:
    649 [0x1000e0000:0x61876fe:0x0]:
    267 [0x1000e0000:0x6187310:0x0]:
    651 [0x1000e0000:0x6187014:0x0]:
    710 [0x1000e0000:0x6186a7c:0x0]:
wcolburn at aocoss04<~>$

The actual errors look like this on the OSS:

[2397666.284621] LustreError: 401793:0:(osd_compat.c:609:osd_obj_update_entry()) aoclst03-OST000e: the FID [0x100000000:0x5d332f8:0x0] is used by two objects: 1777938/3255960136 1750082/3255960136
[2397679.626771] Lustre: aoclst03-OST000e: trigger OI scrub by RPC for the [0x1000e0000:0x5fc978c:0x0] with flags 0x4a, rc = 0


I tried to investigate those files, following information here:

  http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2018-May/015516.html
  http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2018-May/015521.html
  https://www.eofs.eu/_media/events/elw11/08_johann_lombardi_hands_on_lustre_2.x.pdf

My goal was to find the human-readable paths to the affected files.  I
did the debugfs successfully and acquired a fid for one of the files.
To turn a fid into path I tried "lfs fid2path" but when I typed that
command from a lustre client the MDS crashed hard (no response on the
monitor and the capslock key no longer turned on and off).  I don't have
the output of the debugs saved in my terminal window, and I'm little
afraid to run that command again to reacquire the output.

We want to try and solve this problem on Saturday.  Our best idea right
now is to umount the four filesystems on that OSS and run lfsck on each
of them.  A couple of coworkers have said that we can run lfsck while
lustre is alive and mounted.  The fact that I crashed the MDS from an
"lfs fid2path" makes me hesitant to try that though.

I think I've hit all the important bits you need.  What advice can
anyone offer on fixing this problem?  Specifically does the mailing
list advise running lfsck on each of the four filesystems (and mounted
or unmounted if it does) or should we be trying something else instead?


--Schlake
  Sysadmin IV, NRAO
  Work: 575-835-7281 (BACK IN THE OFFICE!)
  Cell: 575-517-5668 (out of work hours)