[lustre-discuss] Filesystem started crashing recently

Steve Barnet barnet at icecube.wisc.edu
Wed Jan 23 11:48:29 PST 2019 

Hi all,

    Since early last summer, we have been running a 2.10.4 filesystem
pretty much without incident. Then about 2 weeks ago, it started
crashing for no immediately obvious reason. There are no indications
of hardware related problems in the logs, and the work loads have
not changed significantly as far as we can tell.

    I can't rule out hardware or system performance problems, but if
that is the case, there are no obvious pointers as to what those
would be. We had one workload that seemed to trigger the problem
(a couple dozen jobs running du on parts of the filesystem), but
that had been running for months, and even after we killed that
we had a couple crashes.

    Since the first crash (on 7 January) we have experience
these crashes sporadically. Sometimes days between crashes,
other times, hours.

    The symptoms are the filesystem becoming unresponsive, and a
load spike on the MDS and one OSS (we have 8x OSS). The OSS
affected seems to be somewhat random. In the system logs, we see
hung_task timeouts and stack traces, followed shortly by lustre-log
dumps. The only real commonality I have seen is that on the MDS,
the first hung task is in jbd2_journal_commit_transaction.

    To recover the filesystems, I have done e2fsck on the MDT,
and any affected OSTs. They have come back cleanly every time.

    I have attached snippets of the log files at the time of
the most recent crash.

A high level summary of our system:

MDS (1x) & OSS (8x)
    OS: CentOS Linux release 7.6.1810 (Core)
    kernel: 3.10.0-862.2.3.el7_lustre.x86_64
    Lustre: 2.10.4 (ldiskfs)

Clients: a mix, but predominantly CentOS 7.x running 2.10.4

Any insights would be greatly appreciated. There are lots of
logs, so if they would be helpful, I can certainly make them
available. In particular, that first lustre-log is pretty big,
so I just grabbed the lines in closest proximity to the crash.

Also, if there's a way to get more debugging level
information from lustre, I'm happy to try that as well.

And I realize this is all at a very high level, so I'll be
happy to provide any additional info needed to help me figure
this out.

Thanks much for taking the time!

Best,

---Steve



-------------- next part --------------
Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io00_002:152121 blocked for more than 120 seconds.
Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io00_002 D ffff914c50f4eeb0     0 152121      2 0x00000000
Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace:
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed4ade>] ? account_entity_dequeue+0xae/0xd0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task jbd2/dm-10-8:152218 blocked for more than 120 seconds.
Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:02 lfs6-oss-4 kernel: jbd2/dm-10-8    D ffff914c44ac2f70     0 152218      2 0x00000000
Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace:
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01fb2ac>] jbd2_journal_commit_transaction+0x23c/0x19b0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed1f75>] ? sched_clock_cpu+0x85/0xc0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4e2959e>] ? __switch_to+0xce/0x580
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551294c>] ? __schedule+0x41c/0xa20
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ea376e>] ? try_to_del_timer_sync+0x5e/0x90
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0201ab9>] kjournald2+0xc9/0x260 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc02019f0>] ? commit_timeout+0x10/0x10 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec9667>] ? finish_task_switch+0x57/0x170
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io01_006:193341 blocked for more than 120 seconds.
Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io01_006 D ffff914c509bbf40     0 193341      2 0x00000000
Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace:
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io01_007:193344 blocked for more than 120 seconds.
Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io01_007 D ffff914b61bdeeb0     0 193344      2 0x00000000
Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace:
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed4ade>] ? account_entity_dequeue+0xae/0xd0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io01_009:193346 blocked for more than 120 seconds.
Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io01_009 D ffff914b61bdaf70     0 193346      2 0x00000000
Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace:
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed4ade>] ? account_entity_dequeue+0xae/0xd0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff5239>] ? ___slab_alloc+0x209/0x4f0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_out00_00:205734 blocked for more than 120 seconds.
Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_out00_00 D ffff916453cabf40     0 205734      2 0x00000000
Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace:
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5514865>] rwsem_down_write_failed+0x225/0x3a0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb515aa77>] call_rwsem_down_write_failed+0x17/0x30
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb55120ed>] down_write+0x2d/0x3d
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d30a8c>] osd_write_lock+0x5c/0xe0 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0aa2119>] out_tx_attr_set_exec+0x69/0x3f0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9c591>] out_tx_end+0xe1/0x5c0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0aa06d3>] out_handle+0x1453/0x1bc0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a30f72>] ? lustre_msg_get_opc+0x22/0xf0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a94c69>] ? tgt_request_preprocess.isra.26+0x299/0x790 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io00_009:261208 blocked for more than 120 seconds.
Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io00_009 D ffff9148bb720000     0 261208      2 0x00000000
Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace:
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff5239>] ? ___slab_alloc+0x209/0x4f0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io00_011:261718 blocked for more than 120 seconds.
Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io00_011 D ffff9148bb723f40     0 261718      2 0x00000000
Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace:
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io01_017:410044 blocked for more than 120 seconds.
Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io01_017 D ffff9142da8a0fd0     0 410044      2 0x00000000
Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace:
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed4ade>] ? account_entity_dequeue+0xae/0xd0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io01_020:3987 blocked for more than 120 seconds.
Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io01_020 D ffff914ec23e8fd0     0  3987      2 0x00000000
Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace:
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed4ade>] ? account_entity_dequeue+0xae/0xd0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
-------------- next part --------------
Jan 18 04:26:32 lfs6-mds-1 kernel: LNet: Service thread pid 63647 was inactive for 200.55s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jan 18 04:26:32 lfs6-mds-1 kernel: LNet: 63574:0:(linux-debug.c:185:libcfs_call_trace()) can't show stack: kernel doesn't export show_task
Jan 18 04:26:32 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807192.63647
Jan 18 04:26:37 lfs6-mds-1 kernel: LNet: Service thread pid 40323 was inactive for 200.38s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jan 18 04:26:37 lfs6-mds-1 kernel: LNet: 63574:0:(linux-debug.c:185:libcfs_call_trace()) can't show stack: kernel doesn't export show_task
Jan 18 04:26:37 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807197.40323
Jan 18 04:26:37 lfs6-mds-1 kernel: LNet: Service thread pid 113300 was inactive for 200.42s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jan 18 04:26:37 lfs6-mds-1 kernel: LNet: Service thread pid 434500 was inactive for 200.35s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jan 18 04:26:37 lfs6-mds-1 kernel: LNet: Skipped 118 previous similar messages
Jan 18 04:26:38 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807198.434571
Jan 18 04:26:38 lfs6-mds-1 kernel: LNet: Service thread pid 434632 was inactive for 200.36s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jan 18 04:26:38 lfs6-mds-1 kernel: LNet: Skipped 99 previous similar messages
Jan 18 04:26:39 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807199.434732
Jan 18 04:26:40 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807200.434844
Jan 18 04:26:40 lfs6-mds-1 kernel: LNet: Service thread pid 63654 was inactive for 200.27s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jan 18 04:26:40 lfs6-mds-1 kernel: LNet: Skipped 35 previous similar messages
Jan 18 04:26:41 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807201.103920
Jan 18 04:26:42 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807202.434796
Jan 18 04:26:43 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807203.434917
Jan 18 04:26:44 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807204.434919
Jan 18 04:26:44 lfs6-mds-1 kernel: LNet: Service thread pid 434927 was inactive for 200.35s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jan 18 04:26:44 lfs6-mds-1 kernel: LNet: Skipped 30 previous similar messages
Jan 18 04:26:45 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807205.434934
Jan 18 04:26:46 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807206.434944
Jan 18 04:26:47 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807207.434945
Jan 18 04:26:48 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807208.434948
Jan 18 04:26:49 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807209.434960
Jan 18 04:26:50 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807210.63656
Jan 18 04:26:51 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807211.434967
Jan 18 04:26:52 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807212.434998
Jan 18 04:26:52 lfs6-mds-1 kernel: LNet: Service thread pid 434999 was inactive for 200.35s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jan 18 04:26:52 lfs6-mds-1 kernel: LNet: Skipped 36 previous similar messages
Jan 18 04:26:53 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807213.434992
Jan 18 04:26:54 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807214.242587
Jan 18 04:26:55 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807215.434986
Jan 18 04:26:56 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807216.435002
Jan 18 04:26:57 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807217.435020
Jan 18 04:26:58 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807218.435021
Jan 18 04:26:59 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807219.435013
Jan 18 04:27:00 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807220.435036
Jan 18 04:27:01 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807221.435026
Jan 18 04:27:02 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807222.435007
Jan 18 04:27:03 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807223.435041
Jan 18 04:27:04 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807224.434715
Jan 18 04:27:05 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807225.435050
Jan 18 04:27:06 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807226.435055
Jan 18 04:27:07 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807227.435062
Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task jbd2/dm-3-8:63269 blocked for more than 120 seconds.
Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:08 lfs6-mds-1 kernel: jbd2/dm-3-8     D ffff9d7e5877eeb0     0 63269      2 0x00000000
Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace:
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc04102ac>] jbd2_journal_commit_transaction+0x23c/0x19b0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896d4ade>] ? account_entity_dequeue+0xae/0xd0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896d865c>] ? dequeue_entity+0x11c/0x5e0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896d1f75>] ? sched_clock_cpu+0x85/0xc0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff8962959e>] ? __switch_to+0xce/0x580
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1294c>] ? __schedule+0x41c/0xa20
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896a376e>] ? try_to_del_timer_sync+0x5e/0x90
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0416ab9>] kjournald2+0xc9/0x260 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc04169f0>] ? commit_timeout+0x10/0x10 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt00_000:63633 blocked for more than 120 seconds.
Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:08 lfs6-mds-1 kernel: mdt00_000       D ffff9d7e3ae18fd0     0 63633      2 0x00000000
Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace:
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt00_001:63634 blocked for more than 120 seconds.
Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:08 lfs6-mds-1 kernel: mdt00_001       D ffff9d7e3ae19fa0     0 63634      2 0x00000000
Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace:
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee80>] ? wake_up_state+0x20/0x20
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt00_002:63635 blocked for more than 120 seconds.
Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:08 lfs6-mds-1 kernel: mdt00_002       D ffff9d7e3ae1af70     0 63635      2 0x00000000
Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace:
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt01_000:63636 blocked for more than 120 seconds.
Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:08 lfs6-mds-1 kernel: mdt01_000       D ffff9d7e3ae1bf40     0 63636      2 0x00000000
Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace:
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt01_001:63637 blocked for more than 120 seconds.
Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:08 lfs6-mds-1 kernel: mdt01_001       D ffff9d7e3ae1cf10     0 63637      2 0x00000000
Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace:
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt01_002:63638 blocked for more than 120 seconds.
Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:08 lfs6-mds-1 kernel: mdt01_002       D ffff9d7e3ae1dee0     0 63638      2 0x00000000
Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace:
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt02_000:63639 blocked for more than 120 seconds.
Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:08 lfs6-mds-1 kernel: mdt02_000       D ffff9d7e3ae1eeb0     0 63639      2 0x00000000
Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace:
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt02_001:63640 blocked for more than 120 seconds.
Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:08 lfs6-mds-1 kernel: mdt02_001       D ffff9d7df65f0000     0 63640      2 0x00000000
Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace:
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt02_002:63641 blocked for more than 120 seconds.
Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 18 04:27:08 lfs6-mds-1 kernel: mdt02_002       D ffff9d7df65f0fd0     0 63641      2 0x00000000
Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace:
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc]
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21
Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:27:09 lfs6-mds-1 kernel: LNet: Service thread pid 435054 was inactive for 200.69s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jan 18 04:27:09 lfs6-mds-1 kernel: LNet: Skipped 34 previous similar messages
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lustre-log.1547807192.63647-snippet.gz
Type: application/x-gzip
Size: 4201 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20190123/62781c5f/attachment-0001.bin>

More information about the lustre-discuss mailing list