[lustre-discuss] MDT inode corruption problem

Tommi T tommi_t77 at yahoo.com
Thu Feb 11 00:11:08 PST 2016


Hi,

During the service break e2fsck revealed that one user had a corrupted directory structure on MDT. I've tried to run e2fsck -f multiple times but it cannot fix this problem. Mounting MDT as a lustre OI-scrub will start and after some time MDS will hit LBUG. I think next step could be just remove problematic directory with debugfs 
clri <inode> but do you have any better suggestions? (fs was formatted with lustre 2.1.3 tools, upgraded to 2.5.x later)

e2fsck 1.42.13.wc4 (28-Nov-2015)
MMP interval is 10 seconds and total wait time is 42 seconds. Please wait...
wrk-MDT0000: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Inodes that were part of a corrupted orphan linked list found.  Fix<y>? yes
Inode 397458656 was part of the orphaned inode list.  FIXED.
Inode 397458656, i_blocks is 3118989520, should be 0.  Fix<y>? yes
Inode 397458688 has illegal block(s).  Clear<y>? yes
Illegal block #2 (2730544606) in inode 397458688.  CLEARED.
Illegal block #4 (2631036428) in inode 397458688.  CLEARED.
Illegal block #5 (1163698720) in inode 397458688.  CLEARED.
Illegal block #6 (4018192778) in inode 397458688.  CLEARED.
Illegal block #8 (2730544607) in inode 397458688.  CLEARED.
Illegal indirect block (3417817744) in inode 397458688.  CLEARED.
Inode 397458688, i_size is 1414676816, should be 49152.  Fix<y>? yes
Inode 397458688, i_blocks is 1, should be 48.  Fix<y>? yes
Running additional passes to resolve blocks claimed by more than one inode...
Pass 1B: Rescanning for multiply-claimed blocks
Multiply-claimed block(s) in inode 397458688: 34 1 1 128 128
Pass 1C: Scanning directories for inodes with multiply-claimed blocks
Pass 1D: Reconciling multiply-claimed blocks
(There are 1 inodes containing multiply-claimed blocks.)
File ... (inode #397458688, mod time Mon Jun  7 11:03:47 1971) 
has 5 multiply-claimed block(s), shared with 1 file(s):
<filesystem metadata>
Clone multiply-claimed blocks<y>? yes
Pass 2: Checking directory structure
Directory inode 397458688, block #0, offset 0: directory corrupted
Salvage<y>? yes
Missing '.' in directory inode 397458688.
Fix<y>? yes
Inode 397458688 is badly corrupt (badness value = 10).  Clear<y>? yes
Missing '..' in directory inode 397458688.
Fix<y>? yes
Setting filetype for entry '..' in ??? (397458688) to 2.
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Failed to optimize directory ??? (397458688): EXT2 directory corrupted
Pass 4: Checking reference counts
Inode 2 ref count is 12, should be 13.  Fix<y>? yes
Inode 397458656 (...) has invalid mode (0151571).
Clear<y>? yes
Pass 5: Checking group summary information
Free blocks count wrong for group #0 (2100, counted=2095).
Fix<y>? yes
Free blocks count wrong for group #4878 (15003, counted=15002).
Fix<y>? yes
Free blocks count wrong (441277870, counted=440655796).
Fix<y>? yes
Free inodes count wrong for group #12129 (149, counted=147).
Fix<y>? yes
Directories count wrong for group #12129 (2655, counted=2656).
Fix<y>? yes
Free inodes count wrong (530993943, counted=530993800).
Fix<y>? yes
[QUOTA WARNING] Usage inconsistent for ID 0:actual (16773439488, 22078) != expected (14225448960, 21937)
[QUOTA WARNING] Missing quota entry ID 8265
[QUOTA WARNING] Missing quota entry ID 59094
Update quota info for quota type 0<y>? yes
[QUOTA WARNING] Usage inconsistent for ID 0:actual (16773435392, 30288) != expected (14225444864, 30147)
[QUOTA WARNING] Missing quota entry ID 16008
[QUOTA WARNING] Missing quota entry ID 52735
Update quota info for quota type 1<y>? yes
wrk-MDT0000: ***** FILE SYSTEM WAS MODIFIED *****
wrk-MDT0000: 811183480/1342177280 files (0.1% non-contiguous), 230432844/671088640 blocks



debugfs 1.42.13.wc4 (28-Nov-2015)
/dev/mapper/lustre-MDT_wrk: catastrophic mode - not reading inode or group bitmaps
Inode: 397458688   Type: directory    Mode:  0105   Flags: 0x0
Generation: 0    Version: 0x00000000
User:  8265   Group: 52735   Size: 49152
File ACL: 0    Directory ACL: 0
Links: 0   Blockcount: 48
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x0000005c -- Thu Jan  1 02:01:32 1970
atime: 0x00010000 -- Thu Jan  1 20:12:16 1970
mtime: 0x02b0ae73 -- Mon Jun  7 11:03:47 1971
dtime: 0x56bb5e7a -- Wed Feb 10 17:59:54 2016
Size of extra inode fields: 0
BLOCKS:
(0):13820, (3):13821, (7):79926036, (9):13822, (10):13834, (11):13856
TOTAL: 6


More information about the lustre-discuss mailing list