[Lustre-discuss] big problem: read-only fs

[Papp Tamás]

hi All,

This morning we see on some client, it cannot connect to one of our node.

I run fsck on the node, and remounted it. Fsck found a lot of errors.


After this I see this on the logs again:

Sep 16 18:16:08 node1 kernel: LustreError: 
2538:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for 
resource 80132: rc -2
Sep 16 18:16:08 node1 kernel: LustreError: 
2538:0:(ldlm_resource.c:719:ldlm_resource_add()) Skipped 15 previous 
similar messages
Sep 16 18:27:15 node1 kernel: LustreError: 
2487:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for 
resource 232490: rc -2
Sep 16 18:27:15 node1 kernel: LustreError: 
2487:0:(ldlm_resource.c:719:ldlm_resource_add()) Skipped 16 previous 
similar messages
Sep 16 18:30:08 node1 kernel: LDISKFS-fs error (device sdb1): 
ldiskfs_ext_find_extent: bad header in inode #58262056: invalid magic - 
magic 0, entries 0, max 0(0), depth 0(0)
Sep 16 18:30:08 node1 kernel: Remounting filesystem read-only
Sep 16 18:30:16 node1 kernel: Lustre: Skipped 1 previous similar message
Sep 16 18:30:16 node1 kernel: LustreError: 
3986:0:(fsfilt-ldiskfs.c:1318:fsfilt_ldiskfs_write_record()) can't start 
transaction for 37 blocks (128 bytes)
Sep 16 18:30:16 node1 kernel: LustreError: 
3986:0:(fsfilt-ldiskfs.c:1318:fsfilt_ldiskfs_write_record()) Skipped 53 
previous similar messages
Sep 16 18:30:16 node1 kernel: LustreError: 
3986:0:(filter.c:360:filter_client_free()) zeroing out client 
2bee00d4-c421-4c8e-bf27-8dd131e0bc55 at idx 51 (14720) in last_rcvd rc -30
Sep 16 18:34:32 node1 kernel: LustreError: 
2426:0:(fsfilt-ldiskfs.c:281:fsfilt_ldiskfs_start()) error starting 
handle for op 8 (71 credits): rc -30
Sep 16 18:34:32 node1 kernel: LustreError: 
2426:0:(fsfilt-ldiskfs.c:281:fsfilt_ldiskfs_start()) Skipped 36 previous 
similar messages
Sep 16 18:37:16 node1 kernel: LustreError: 
2416:0:(ldlm_resource.c:719:ldlm_resource_add()) lvbo_init failed for 
resource 212576: rc -2
Sep 16 18:37:16 node1 kernel: LustreError: 
2416:0:(ldlm_resource.c:719:ldlm_resource_add()) Skipped 15 previous 
similar messages
Sep 16 18:43:20 node1 kernel: LustreError: 
2341:0:(fsfilt-ldiskfs.c:281:fsfilt_ldiskfs_start()) error starting 
handle for op 8 (71 credits): rc -30
Sep 16 18:43:20 node1 kernel: LustreError: 
2341:0:(filter.c:273:filter_client_add()) unable to start transaction: 
rc -30
Sep 16 18:43:20 node1 kernel: LustreError: 
2341:0:(filter.c:273:filter_client_add()) Skipped 36 previous similar 
messages
Sep 16 18:43:20 node1 kernel: LustreError: 
2341:0:(filter.c:294:filter_client_add()) error writing last_rcvd client 
idx 34: rc -30
Sep 16 18:43:20 node1 kernel: LustreError: 
2341:0:(filter.c:294:filter_client_add()) Skipped 36 previous similar 
messages
Sep 16 18:43:20 node1 kernel: LustreError: 
2341:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error 
(-30)  req at ffff81000276de00 x8/t0 o8-><?>@<?>:-1 lens 240/144 ref 0 fl 
Interpret:/0/0 rc -30/0
Sep 16 18:43:21 node1 kernel: LustreError: 
2341:0:(ldlm_lib.c:1442:target_send_reply_msg()) Skipped 5 previous 
similar messages
Sep 16 18:44:10 node1 kernel: LustreError: 
2399:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error 
(-30)  req at ffff81005f5d0400 x38/t0 o8-><?>@<?>:-1 lens 240/144 ref 0 fl 
Interpret:/0/0 rc -30/0
Sep 16 18:44:10 node1 kernel: LustreError: 
2399:0:(ldlm_lib.c:1442:target_send_reply_msg()) Skipped 1 previous 
similar message
Sep 16 18:44:35 node1 kernel: LustreError: 
2341:0:(filter.c:273:filter_client_add()) unable to start transaction: 
rc -30
Sep 16 18:44:35 node1 kernel: LustreError: 
2341:0:(filter.c:273:filter_client_add()) Skipped 2 previous similar 
messages
Sep 16 18:44:35 node1 kernel: LustreError: 
2341:0:(filter.c:294:filter_client_add()) error writing last_rcvd client 
idx 51: rc -30
Sep 16 18:44:35 node1 kernel: LustreError: 
2341:0:(filter.c:294:filter_client_add()) Skipped 2 previous similar 
messages

errno -2 was right after fsck, it's OK.

But why does -30 is here? I hoped, it will disappear after fsck, but I 
see again. What could cause this problem? How can I solve it?

Thank you,

tamas