Mar 14 16:44:05 cn02 kernel: LustreError: 3040:0:(statahead.c:129:ll_sai_entry_set()) ASSERTION(entry->se_stat == SA_ENTRY_UNSTATED) failed
Mar 14 16:44:05 cn02 kernel: LustreError: 3040:0:(tracefile.c:431:libcfs_assertion_failed()) LBUG
Mar 14 16:44:05 cn02 kernel: Lustre: 3040:0:(linux-debug.c:168:libcfs_debug_dumpstack()) showing stack for process 3040
Mar 14 16:44:05 cn02 kernel: ptlrpcd       R  running task       0  3040      1          3041  3038 (L-TLB)
Mar 14 16:44:05 cn02 kernel:  ffff810204a84608 ffff810204a84760 ffff810204a84600 ffff810228ab7b40
Mar 14 16:44:05 cn02 kernel:  ffffffffffffffff ffff81021e219ee0 ffffffff8855736a 0000000000007864
Mar 14 16:44:05 cn02 kernel:  ffff81022c8e7988 ffffffff00000000 ffff81022f874820 ffffffffffffffff
Mar 14 16:44:05 cn02 kernel: Call Trace:
Mar 14 16:44:05 cn02 kernel:  [<ffffffff8855736a>] :ptlrpc:ptlrpc_check_set+0x9aa/0xb60
Mar 14 16:44:05 cn02 kernel:  [<ffffffff80048a05>] try_to_del_timer_sync+0x51/0x5a
Mar 14 16:44:05 cn02 kernel:  [<ffffffff88583c96>] :ptlrpc:ptlrpcd_check+0x176/0x2a0
Mar 14 16:44:05 cn02 kernel:  [<ffffffff80092968>] process_timeout+0x0/0x5
Mar 14 16:44:05 cn02 kernel:  [<ffffffff885842f6>] :ptlrpc:ptlrpcd+0x1e6/0x26f
Mar 14 16:44:05 cn02 kernel:  [<ffffffff800867b0>] default_wake_function+0x0/0xe
Mar 14 16:44:05 cn02 kernel:  [<ffffffff800867b0>] default_wake_function+0x0/0xe
Mar 14 16:44:05 cn02 kernel:  [<ffffffff8005bc25>] child_rip+0xa/0x11
Mar 14 16:44:05 cn02 kernel:  [<ffffffff88584110>] :ptlrpc:ptlrpcd+0x0/0x26f
Mar 14 16:44:05 cn02 kernel:  [<ffffffff8005bc1b>] child_rip+0x0/0x11
Mar 14 16:44:06 cn02 kernel: 
Mar 14 16:44:06 cn02 kernel: LustreError: dumping log to /tmp/lustre-log.1205509445.3040
Mar 14 16:44:15 cn02 kernel: BUG: soft lockup detected on CPU#1!
Mar 14 16:44:15 cn02 kernel: 
Mar 14 16:44:15 cn02 kernel: Call Trace:
Mar 14 16:44:15 cn02 kernel:  <IRQ>  [<ffffffff800b26bc>] softlockup_tick+0xdb/0xed
Mar 14 16:44:15 cn02 kernel:  [<ffffffff80092f52>] update_process_times+0x42/0x68
Mar 14 16:44:15 cn02 kernel:  [<ffffffff800738e4>] smp_local_timer_interrupt+0x23/0x47
Mar 14 16:44:15 cn02 kernel:  [<ffffffff80073fa6>] smp_apic_timer_interrupt+0x41/0x47
Mar 14 16:44:15 cn02 kernel:  [<ffffffff8005b902>] apic_timer_interrupt+0x66/0x6c
Mar 14 16:44:15 cn02 kernel:  <EOI>  [<ffffffff800622d4>] .text.lock.spinlock+0x2/0x30
Mar 14 16:44:15 cn02 kernel:  [<ffffffff886d3f4e>] :lustre:ll_stop_statahead+0x3e/0x270
Mar 14 16:44:15 cn02 kernel:  [<ffffffff8001a36e>] cp_new_stat+0xe5/0xfd
Mar 14 16:44:15 cn02 kernel:  [<ffffffff8869e509>] :lustre:ll_file_release+0x119/0x2b0
Mar 14 16:44:15 cn02 kernel:  [<ffffffff80012239>] __fput+0x9e/0x186
Mar 14 16:44:15 cn02 kernel:  [<ffffffff8002357f>] filp_close+0x5c/0x64
Mar 14 16:44:15 cn02 kernel:  [<ffffffff8001d52f>] sys_close+0x88/0xa2
Mar 14 16:44:15 cn02 kernel:  [<ffffffff8005af01>] tracesys+0xd1/0xdc
Mar 14 16:44:15 cn02 kernel: 
Mar 14 16:44:15 cn02 kernel: BUG: soft lockup detected on CPU#2!
Mar 14 16:44:15 cn02 kernel: 
Mar 14 16:44:15 cn02 kernel: Call Trace:
Mar 14 16:44:15 cn02 kernel:  <IRQ>  [<ffffffff800b26bc>] softlockup_tick+0xdb/0xed
Mar 14 16:44:15 cn02 kernel:  [<ffffffff80092f52>] update_process_times+0x42/0x68
Mar 14 16:44:15 cn02 kernel:  [<ffffffff800738e4>] smp_local_timer_interrupt+0x23/0x47
Mar 14 16:44:15 cn02 kernel:  [<ffffffff80073fa6>] smp_apic_timer_interrupt+0x41/0x47
Mar 14 16:44:15 cn02 kernel:  [<ffffffff8005b902>] apic_timer_interrupt+0x66/0x6c
Mar 14 16:44:15 cn02 kernel:  <EOI>  [<ffffffff800622d4>] .text.lock.spinlock+0x2/0x30
Mar 14 16:44:15 cn02 kernel:  [<ffffffff886d3f4e>] :lustre:ll_stop_statahead+0x3e/0x270
Mar 14 16:44:15 cn02 kernel:  [<ffffffff8001a36e>] cp_new_stat+0xe5/0xfd
Mar 14 16:44:15 cn02 kernel:  [<ffffffff8869e509>] :lustre:ll_file_release+0x119/0x2b0
Mar 14 16:44:15 cn02 kernel:  [<ffffffff80012239>] __fput+0x9e/0x186
Mar 14 16:44:15 cn02 kernel:  [<ffffffff8002357f>] filp_close+0x5c/0x64
Mar 14 16:44:15 cn02 kernel:  [<ffffffff8001d52f>] sys_close+0x88/0xa2
Mar 14 16:44:15 cn02 kernel:  [<ffffffff8005af01>] tracesys+0xd1/0xdc
Mar 14 16:44:15 cn02 kernel: 
Mar 14 16:44:15 cn02 kernel: BUG: soft lockup detected on CPU#0!
Mar 14 16:44:15 cn02 kernel: 
Mar 14 16:44:15 cn02 kernel: Call Trace:
Mar 14 16:44:15 cn02 kernel:  <IRQ>  [<ffffffff800b26bc>] softlockup_tick+0xdb/0xed
Mar 14 16:44:15 cn02 kernel:  [<ffffffff80092f52>] update_process_times+0x42/0x68
Mar 14 16:44:16 cn02 kernel:  [<ffffffff800738e4>] smp_local_timer_interrupt+0x23/0x47
Mar 14 16:44:16 cn02 kernel:  [<ffffffff80073fa6>] smp_apic_timer_interrupt+0x41/0x47
Mar 14 16:44:16 cn02 kernel:  [<ffffffff8005b902>] apic_timer_interrupt+0x66/0x6c
Mar 14 16:44:16 cn02 kernel:  <EOI>  [<ffffffff800622d4>] .text.lock.spinlock+0x2/0x30
Mar 14 16:44:16 cn02 kernel:  [<ffffffff886d3f4e>] :lustre:ll_stop_statahead+0x3e/0x270
Mar 14 16:44:16 cn02 kernel:  [<ffffffff8001a36e>] cp_new_stat+0xe5/0xfd
Mar 14 16:44:16 cn02 kernel:  [<ffffffff8869e509>] :lustre:ll_file_release+0x119/0x2b0
Mar 14 16:44:16 cn02 kernel:  [<ffffffff80012239>] __fput+0x9e/0x186
Mar 14 16:44:16 cn02 kernel:  [<ffffffff8002357f>] filp_close+0x5c/0x64
Mar 14 16:44:16 cn02 kernel:  [<ffffffff8001d52f>] sys_close+0x88/0xa2
Mar 14 16:44:16 cn02 kernel:  [<ffffffff8005af01>] tracesys+0xd1/0xdc
Mar 14 16:44:16 cn02 kernel: 
Mar 14 16:49:05 cn02 kernel: LustreError: 11055:0:(client.c:975:ptlrpc_expire_one_request()) @@@ timeout (sent at 1205509445, 300s ago)  req@ffff8101d5b86800 x109103715/t0 o101->lustre-MDT0000_UUID@10.128.15.41@tcp:12 lens 488/816 ref 2 fl Rpc:P/0/0 rc 0/-22
Mar 14 16:49:05 cn02 kernel: LustreError: 11055:0:(client.c:975:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Mar 14 16:49:05 cn02 kernel: Lustre: lustre-MDT0000-mdc-ffff81021da30400: Connection to service lustre-MDT0000 via nid 10.128.15.41@tcp was lost; in progress operations using this service will wait for recovery to complete.
