[Lustre-discuss] e2fsck claims OST is clean, then dirty

Thu Jan 21 08:42:49 PST 2010

I had problems with this OST a few weeks ago. It's journal became slow and
eventually we experienced an OST failure and file corruption.
I've since taken the entire FS offline and e2fsck'd every OST and the MDT. I
found a few bad inode count errors but nothing really serious looking like
when we lost data.

Now the same OST is acting up again. I'm seeing CPU soft lockups and
messages like this in the lustre debug logs.:
=========
00010000:00000400:5:1264089753.654698:0:11495:0:(ldlm_lib.c:541:target_handle_reconnect())
scratch-OST000e: 31dd4e5b-99c7-677d-5301-e1ba53c1e56b reconnecting
00010000:00000400:5:1264089753.654708:0:11495:0:(ldlm_lib.c:835:target_handle_connect())
scratch-OST000e: refuse reconnection from
31dd4e5b-99c7-677d-5301-e1ba53c1e56b at 10.2.1.121@o2ib2 to 0
xffff810304230c00; still busy with 4 active RPCs
00010000:00020000:5:1264089753.654715:0:11495:0:(ldlm_lib.c:1863:target_send_reply_msg())
@@@ processing error (-16)  req at ffff810229675c00 x1324238487685594/t0
o8->31dd4e5b-99c7-677d-5301-e
1ba53c1e56b at NET_0x500020a020179_UUID:0/0 lens 368/264 e 0 to 0 dl 1264089853
ref 1 fl Interpret:/0/0 rc -16/0
=========

=========
[root at oss-0-0 ~]# dumpe2fs -h /dev/dsk/ost24 | grep state
dumpe2fs 1.41.6.sun1 (30-May-2009)
Filesystem state:         clean
=========

=========
[root at oss-0-0 ~]# e2fsck -n /dev/dsk/ost24  | tee
/state/partition1/e2fsck-n_ost24_`date '+%m.%d.%y-%H:%M:%S'`.log
e2fsck 1.41.6.sun1 (30-May-2009)
device /dev/sdy mounted by lustre per
/proc/fs/lustre/obdfilter/scratch-OST000e/mntdev
Warning!  /dev/dsk/ost24 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem
check.
scratch-OSTffff: clean, 3219531/183009280 files, 60844652/732023488 blocks
=========

I ran that again and it produced a ton of errors. Could it be because the
OST is active? (Log attached)

Then I ran it again and it e2fsck reported that the OST was clean.

Another odd note: tune2fs is reporting the wrong lustre UUID for the OST. It
should be scratch-OST000e

=========
[root at oss-0-0 ~]# tunefs.lustre --dryrun /dev/dsk/ost24 | grep Target | uniq
Target:     scratch-OST000e
=========

I don't see any errors on the storage systems.

Any ideas?
Thanks
Erik Froese
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100121/5b823e73/attachment.htm>
-------------- next part --------------
[root at oss-0-0 ~]# e2fsck -n /dev/dsk/ost20
e2fsck 1.41.6.sun1 (30-May-2009)
device /dev/sdw mounted by lustre per /proc/fs/lustre/obdfilter/scratch-OST0002/mntdev
Warning!  /dev/dsk/ost20 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
scratch-OST0002: clean, 2800421/183009280 files, 62858912/732023488 blocks

[root at oss-0-0 ~]# e2fsck -n /dev/dsk/ost20
e2fsck 1.41.6.sun1 (30-May-2009)
device /dev/sdw mounted by lustre per /proc/fs/lustre/obdfilter/scratch-OST0002/mntdev
Warning!  /dev/dsk/ost20 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
scratch-OST0002 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry '11503074' in /O/0/d2 (8642570) has deleted/unused inode 168035577.  Clear? no

Entry '11503040' in /O/0/d0 (8642568) has deleted/unused inode 168020980.  Clear? no

Entry '11503060' in /O/0/d20 (8642588) has deleted/unused inode 167657988.  Clear? no

Entry '11503043' in /O/0/d3 (8642571) has deleted/unused inode 168043523.  Clear? no

Entry '11502823' in /O/0/d7 (8642575) references inode 167559141 found in group 20453's unused inodes area.
Fix? no

Entry '11502823' in /O/0/d7 (8642575) has deleted/unused inode 167559141.  Clear? no

Entry '11503058' in /O/0/d18 (8642586) has deleted/unused inode 167642410.  Clear? no

Entry '11503029' in /O/0/d21 (8642589) has deleted/unused inode 167666096.  Clear? no

Entry '11502902' in /O/0/d22 (8642590) references inode 167419897 found in group 20436's unused inodes area.
Fix? no

Entry '11502902' in /O/0/d22 (8642590) has deleted/unused inode 167419897.  Clear? no

Entry '11503041' in /O/0/d1 (8642569) has deleted/unused inode 168029214.  Clear? no

Entry '11503015' in /O/0/d7 (8642575) references inode 167559144 found in group 20453's unused inodes area.
Fix? no

Entry '11503015' in /O/0/d7 (8642575) has deleted/unused inode 167559144.  Clear? no

Entry '11503036' in /O/0/d28 (8642596) has deleted/unused inode 167467012.  Clear? no

Entry '11502838' in /O/0/d22 (8642590) references inode 167419895 found in group 20436's unused inodes area.
Fix? no

Entry '11502838' in /O/0/d22 (8642590) has deleted/unused inode 167419895.  Clear? no

Entry '11502806' in /O/0/d22 (8642590) references inode 167419894 found in group 20436's unused inodes area.
Fix? no

Entry '11502806' in /O/0/d22 (8642590) has deleted/unused inode 167419894.  Clear? no

Entry '11502742' in /O/0/d22 (8642590) references inode 167419892 found in group 20436's unused inodes area.
Fix? no

Entry '11502742' in /O/0/d22 (8642590) has deleted/unused inode 167419892.  Clear? no

Entry '11503026' in /O/0/d18 (8642586) has deleted/unused inode 167642407.  Clear? no

Entry '11502919' in /O/0/d7 (8642575) references inode 167559143 found in group 20453's unused inodes area.
Fix? no

Entry '11502919' in /O/0/d7 (8642575) has deleted/unused inode 167559143.  Clear? no

Entry '11503023' in /O/0/d15 (8642583) has deleted/unused inode 167621104.  Clear? no

Entry '11502791' in /O/0/d7 (8642575) references inode 167559140 found in group 20453's unused inodes area.
Fix? no

Entry '11502791' in /O/0/d7 (8642575) has deleted/unused inode 167559140.  Clear? no

Entry '11503045' in /O/0/d5 (8642573) has deleted/unused inode 168059270.  Clear? no

Entry '11503020' in /O/0/d12 (8642580) has deleted/unused inode 167597793.  Clear? no

Entry '11503037' in /O/0/d29 (8642597) has deleted/unused inode 167475171.  Clear? no

Entry '11503044' in /O/0/d4 (8642572) has deleted/unused inode 168051033.  Clear? no

Entry '11503047' in /O/0/d7 (8642575) references inode 167559145 found in group 20453's unused inodes area.
Fix? no

Entry '11503047' in /O/0/d7 (8642575) has deleted/unused inode 167559145.  Clear? no

Entry '11502710' in /O/0/d22 (8642590) references inode 167419891 found in group 20436's unused inodes area.
Fix? no

Entry '11502710' in /O/0/d22 (8642590) has deleted/unused inode 167419891.  Clear? no

Entry '11503030' in /O/0/d22 (8642590) references inode 167419899 found in group 20436's unused inodes area.
Fix? no

Entry '11503030' in /O/0/d22 (8642590) has deleted/unused inode 167419899.  Clear? no

Entry '11503011' in /O/0/d3 (8642571) has deleted/unused inode 168043522.  Clear? no

Entry '11502759' in /O/0/d7 (8642575) references inode 167559139 found in group 20453's unused inodes area.
Fix? no

Entry '11502759' in /O/0/d7 (8642575) has deleted/unused inode 167559139.  Clear? no

Entry '11502887' in /O/0/d7 (8642575) references inode 167559142 found in group 20453's unused inodes area.
Fix? no

Entry '11502887' in /O/0/d7 (8642575) has deleted/unused inode 167559142.  Clear? no

Entry '11503012' in /O/0/d4 (8642572) has deleted/unused inode 168051032.  Clear? no

Entry '11503038' in /O/0/d30 (8642598) has deleted/unused inode 167482951.  Clear? no

Entry '11502774' in /O/0/d22 (8642590) references inode 167419893 found in group 20436's unused inodes area.
Fix? no

Entry '11502774' in /O/0/d22 (8642590) has deleted/unused inode 167419893.  Clear? no

Entry '11502678' in /O/0/d22 (8642590) references inode 167419890 found in group 20436's unused inodes area.
Fix? no

Entry '11502678' in /O/0/d22 (8642590) has deleted/unused inode 167419890.  Clear? no

Entry '11503062' in /O/0/d22 (8642590) references inode 167419896 found in group 20436's unused inodes area.
Fix? no

Entry '11503062' in /O/0/d22 (8642590) has deleted/unused inode 167419896.  Clear? no

Entry '11503039' in /O/0/d31 (8642599) has deleted/unused inode 167491010.  Clear? no

Entry '11503059' in /O/0/d19 (8642587) has deleted/unused inode 167650585.  Clear? no

Entry '11503019' in /O/0/d11 (8642579) has deleted/unused inode 167589926.  Clear? no

Entry '11503014' in /O/0/d6 (8642574) has deleted/unused inode 167550783.  Clear? no

Entry '11503027' in /O/0/d19 (8642587) has deleted/unused inode 167650581.  Clear? no

Entry '11503024' in /O/0/d16 (8642584) has deleted/unused inode 167627697.  Clear? no

Entry '11503046' in /O/0/d6 (8642574) has deleted/unused inode 167550784.  Clear? no

Entry '11502438' in /O/0/d6 (8642574) has deleted/unused inode 167056904.  Clear? no

Entry '11503042' in /O/0/d2 (8642570) has deleted/unused inode 168035569.  Clear? no

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Unattached zero-length inode 167024510.  Clear? no

Unattached inode 167024510
Connect to /lost+found? no

Unattached zero-length inode 167054393.  Clear? no

Unattached inode 167054393
Connect to /lost+found? no

Unattached zero-length inode 167065382.  Clear? no

Unattached inode 167065382
Connect to /lost+found? no

Unattached zero-length inode 167098292.  Clear? no

Unattached inode 167098292
Connect to /lost+found? no

Unattached zero-length inode 167105974.  Clear? no

Unattached inode 167105974
Connect to /lost+found? no

Unattached zero-length inode 167131057.  Clear? no

Unattached inode 167131057
Connect to /lost+found? no

Unattached zero-length inode 167177214.  Clear? no

Unattached inode 167177214
Connect to /lost+found? no

Unattached zero-length inode 167245519.  Clear? no

Unattached inode 167245519
Connect to /lost+found? no

Unattached inode 167317242
Connect to /lost+found? no

Unattached zero-length inode 167435638.  Clear? no

Unattached inode 167435638
Connect to /lost+found? no

Unattached zero-length inode 167443845.  Clear? no

Unattached inode 167443845
Connect to /lost+found? no

Unattached zero-length inode 167443846.  Clear? no

Unattached inode 167443846
Connect to /lost+found? no

Unattached zero-length inode 167443847.  Clear? no

Unattached inode 167443847
Connect to /lost+found? no

Unattached zero-length inode 167449677.  Clear? no

Unattached inode 167449677
Connect to /lost+found? no

Unattached zero-length inode 167450803.  Clear? no

Unattached inode 167450803
Connect to /lost+found? no

Unattached zero-length inode 167457939.  Clear? no

Unattached inode 167457939
Connect to /lost+found? no

Unattached zero-length inode 167458040.  Clear? no

Unattached inode 167458040
Connect to /lost+found? no

Unattached zero-length inode 167458500.  Clear? no

Unattached inode 167458500
Connect to /lost+found? no

Unattached zero-length inode 167465834.  Clear? no

Unattached inode 167465834
Connect to /lost+found? no

Unattached zero-length inode 167467021.  Clear? no

Unattached inode 167467021
Connect to /lost+found? no

Unattached zero-length inode 167473951.  Clear? no

Unattached inode 167473951
Connect to /lost+found? no

Unattached zero-length inode 167475170.  Clear? no

Unattached inode 167475170
Connect to /lost+found? no

Unattached zero-length inode 167481775.  Clear? no

Unattached inode 167481775
Connect to /lost+found? no

Unattached zero-length inode 167481895.  Clear? no

Unattached inode 167481895
Connect to /lost+found? no

Unattached zero-length inode 167482950.  Clear? no

Unattached inode 167482950
Connect to /lost+found? no

Unattached zero-length inode 167489932.  Clear? no

Unattached inode 167489932
Connect to /lost+found? no

Unattached zero-length inode 167490010.  Clear? no

Unattached inode 167490010
Connect to /lost+found? no

Unattached zero-length inode 167490093.  Clear? no

Unattached inode 167490093
Connect to /lost+found? no

Unattached zero-length inode 167533943.  Clear? no

Unattached inode 167533943
Connect to /lost+found? no

Unattached zero-length inode 167550782.  Clear? no

Unattached inode 167550782
Connect to /lost+found? no

Unattached zero-length inode 167557940.  Clear? no

Unattached inode 167557940
Connect to /lost+found? no

Unattached zero-length inode 167566347.  Clear? no

Unattached inode 167566347
Connect to /lost+found? no

Unattached zero-length inode 167574528.  Clear? no

Unattached inode 167574528
Connect to /lost+found? no

Unattached zero-length inode 167580757.  Clear? no

Unattached inode 167580757
Connect to /lost+found? no

Unattached zero-length inode 167588968.  Clear? no

Unattached inode 167588968
Connect to /lost+found? no

Unattached inode 167596806
Connect to /lost+found? no

Unattached zero-length inode 167596827.  Clear? no

Unattached inode 167596827
Connect to /lost+found? no

Unattached zero-length inode 167604944.  Clear? no

Unattached inode 167604944
Connect to /lost+found? no

Unattached zero-length inode 167622065.  Clear? no

Unattached inode 167622065
Connect to /lost+found? no

Unattached zero-length inode 167634957.  Clear? no

Unattached inode 167634957
Connect to /lost+found? no

Unattached zero-length inode 167635985.  Clear? no

Unattached inode 167635985
Connect to /lost+found? no

Unattached zero-length inode 167641186.  Clear? no

Unattached inode 167641186
Connect to /lost+found? no

Unattached zero-length inode 167642412.  Clear? no

Unattached inode 167642412
Connect to /lost+found? no

Unattached zero-length inode 167650588.  Clear? no

Unattached inode 167650588
Connect to /lost+found? no

Unattached zero-length inode 167657990.  Clear? no

Unattached inode 167657990
Connect to /lost+found? no

Unattached zero-length inode 167657995.  Clear? no

Unattached inode 167657995
Connect to /lost+found? no

Unattached zero-length inode 167666098.  Clear? no

Unattached inode 167666098
Connect to /lost+found? no

Unattached zero-length inode 167682073.  Clear? no

Unattached inode 167682073
Connect to /lost+found? no

Unattached zero-length inode 168019942.  Clear? no

Unattached inode 168019942
Connect to /lost+found? no

Unattached zero-length inode 168020979.  Clear? no

Unattached inode 168020979
Connect to /lost+found? no

Unattached zero-length inode 168042566.  Clear? no

Unattached inode 168042566
Connect to /lost+found? no

Unattached zero-length inode 168043521.  Clear? no

Unattached inode 168043521
Connect to /lost+found? no

Unattached zero-length inode 168051022.  Clear? no

Unattached inode 168051022
Connect to /lost+found? no

Pass 5: Checking group summary information
Block bitmap differences:  -(10586341--10586342) -10588153 -(10589685--10589687) -(10595314--10595318) -(10618313--10618346) -10618879 -10623999 -(10626555--10626558) +(10630059--10630127) +(10638300--10638303) -(10638304--10638305) -(10644300--10644478) +(24983806--24983815) +(86245353--86245355) +(86450167--86450169) -(86535213--86535235) +(86542318--86542320) +(86543346--86543349) +(86543351--86543354) -(86778413--86778435) -(86793459--86793598) +(86796196--86796201) -(86806965--86807009) +86836244 +(86840826--86840829) +(86852053--86852056) +(86852090--86852093) +(86857148--86857149) +(86860791--86860793) +(86862834--86862838) +(86862843--86862846) +(86865501--86865569) -(86865570--86865638) +(206146032--206146036) +(206151670--206151675) +(206162924--206162932) +(206261186--206261189) +(206268388--206268391) +(206280561--206280562) +(206280698--206280703) +(206283748--206283750) +(206289792--206289796) +(206289870--206289871) +(206292414--206292418) +(206292466--206292467) +(206292471--206292476) +(206292972--206292975) +(206297580--206297585) +(206335479--206335483) +(206339579--206339582) +(206345211--206345212) +(206352888--206352893) +(206362093--206362100) +(206385920--206386175)Group 6336 block(s) in use but group is marked BLOCK_UNINIT
Fix? no

 +(206616832--206617087)Group 6336 block(s) in use but group is marked BLOCK_UNINIT
Fix? no