[Lustre-discuss] OST crashed after slow journal messages

[Erik Froese]

Hey everyone,
I had an OST crash (actually it made the entire OSS unresponsive to the
point where I had to shoot it).
There were messages in /var/log/messages complaining about slow journal
performance (we have separate OSTs and journal disks).

Dec 28 20:29:02 oss-0-0 kernel: LustreError:
12638:0:(filter_io_26.c:742:filter_commitrw_write()) scratch-OST000e: slow
direct_io 85s
Dec 28 20:29:02 oss-0-0 kernel: LustreError:
12638:0:(filter_io_26.c:742:filter_commitrw_write()) Skipped 58 previous
similar messages
Dec 28 21:50:13 oss-0-0 kernel: LustreError:
16499:0:(lustre_fsfilt.h:320:fsfilt_commit_wait()) scratch-OST000e: slow
journal start 51s
Dec 28 21:50:13 oss-0-0 kernel: LustreError:
16499:0:(lustre_fsfilt.h:320:fsfilt_commit_wait()) Skipped 66 previous
similar messages

Lustre and e2fsprogs versions:

[root at oss-0-0 ~]# rpm -q kernel-lustre
kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1
[root at oss-0-0 ~]# rpm -q e2fsprogs
e2fsprogs-1.41.6.sun1-0redhat


Then there's this interesting message:
Dec 29 14:11:32 oss-0-0 kernel: LDISKFS-fs error (device sdz):
ldiskfs_lookup: unlinked inode 5384166 in dir #145170469
Dec 29 14:11:32 oss-0-0 kernel: Remounting filesystem read-only

Followed by a whole lot of these:
Dec 29 14:11:32 oss-0-0 kernel: LustreError:
18559:0:(fsfilt-ldiskfs.c:336:fsfilt_ldiskfs_start()) error starting handle
for op 8 (106 credits): rc -30
Dec 29 14:11:33 oss-0-0 kernel: LustreError:
12648:0:(fsfilt-ldiskfs.c:336:fsfilt_ldiskfs_start()) error starting handle
for op 8 (106 credits): rc -30
Dec 29 14:11:34 oss-0-0 kernel: LustreError:
16430:0:(fsfilt-ldiskfs.c:336:fsfilt_ldiskfs_start()) error starting handle
for op 8 (106 credits): rc -30
Dec 29 14:11:35 oss-0-0 kernel: LustreError:
16446:0:(fsfilt-ldiskfs.c:470:fsfilt_ldiskfs_brw_start()) can't get handle
for 125 credits: rc = -30
Dec 29 14:11:35 oss-0-0 kernel: LustreError:
16446:0:(filter_io_26.c:684:filter_commitrw_write()) error starting
transaction: rc = -30
Dec 29 14:11:35 oss-0-0 kernel: LustreError:
12666:0:(filter_io_26.c:684:filter_commitrw_write()) error starting
transaction: rc = -30
Dec 29 14:11:36 oss-0-0 kernel: LustreError:
16417:0:(fsfilt-ldiskfs.c:470:fsfilt_ldiskfs_brw_start()) can't get handle
for 471 credits: rc = -30
Dec 29 14:11:36 oss-0-0 kernel: LustreError:
16417:0:(fsfilt-ldiskfs.c:470:fsfilt_ldiskfs_brw_start()) Skipped 1 previous
similar message
Dec 29 14:11:36 oss-0-0 kernel: LustreError:
16417:0:(filter_io_26.c:684:filter_commitrw_write()) error starting
transaction: rc = -30

Whenever I try to mount the ost (known as /dev/dsk/ost24) I get the
following messages:
Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs error (device sdz):
ldiskfs_check_descriptors: Checksum for group 16303 failed (64812!=44)
Dec 29 19:25:35 oss-0-0 kernel: LDISKFS-fs: group descriptors corrupted!
Dec 29 19:25:35 oss-0-0 kernel: LustreError:
15067:0:(obd_mount.c:1278:server_kernel_mount()) premount /dev/dsk/ost24:0x0
ldiskfs failed: -22, ldiskfs2 failed: -19.  Is the ldi
skfs module available?
Dec 29 19:25:35 oss-0-0 kernel: LustreError:
15067:0:(obd_mount.c:1592:server_fill_super()) Unable to mount device
/dev/dsk/ost24: -22
Dec 29 19:25:35 oss-0-0 kernel: LustreError:
15067:0:(obd_mount.c:1997:lustre_fill_super()) Unable to mount  (-22)
Dec 29 19:25:35 oss-0-0 Filesystem[15020]: ERROR: Couldn't mount filesystem
/dev/dsk/ost24 on /mnt/scratch/ost24
Dec 29 19:25:35 oss-0-0 Filesystem[15009]: ERROR:  Generic error

So it looks like the "group descriptors" are corrupted. I'm not sure what
those are but e2fsck -n sure enough complains about them. So I tried running
it for real.

I ran e2fsck -j /dev/$JOURNAL -v -fy -C 0 /dev/$DEVICE.

The first time I ran to what looked like completion. It printed a summary
and all but then didn't exit. I sent it a kill but that didn't stop it. So I
let it run and went back to sleep for 3 hours. When I woke up the process
was gone but I still get the same error messages.

I found this discussion
http://lists.lustre.org/pipermail/lustre-discuss/2009-March/009885.html
and tried the tune2fs command followed by the e2fsck but it hasn't exited
yet (its a 2.7 TB OST)

The LUN comes from a Sun STK 6140/CSM200 device which isn't reporting any
warning, events, or errors.

I deactivated the OST with lctl but it still shows up as active on the
clients. Also lfs find /scratch -O scratch-OST000e_UUID HANGS!

Are we screwed here? Is there a way to run lfs find with the OST disabled?
Shouldn't that just be a metadata operation?

Thanks
Erik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20091230/9929ee13/attachment.htm>