[lustre-discuss] lustre mounts stuck in llog_process_or_fork

[Nehring, Shane R [LAS]]

Hello all,

We recently rebuilt our filesystem on 2.15.0 due to issues we were seeing on our
legacy volume. The mds and oss-es are rhel 8.6 The mdt is zfs as are all the
osts. We built from b2_15 to get the rhel 8.6 compat commits.

It's been working pretty well for the past week or so until we had a client
crash earlier today. Said client wasn't able to access the mount when it came
back so we decided to reboot the mds.

The mounts for the mdts hung in llog_process_or_fork for about 25 minutes or so
before actually beginning the mounting process and starting recovery. Similarly
in two clients we had with connectivity issues with the filesystem and rebooted
we see them stuck in llog_process_or_fork

stack traces:
client:
[<0>] llog_process_or_fork+0x43c/0x560 [obdclass]
[<0>] llog_process+0x10/0x20 [obdclass]
[<0>] class_config_parse_llog+0x1e9/0x3e0 [obdclass]
[<0>] mgc_process_cfg_log+0x709/0xd80 [mgc]
[<0>] mgc_process_log+0x6c3/0x800 [mgc]
[<0>] mgc_process_config+0xb53/0xe60 [mgc]
[<0>] lustre_process_log+0x5fa/0xad0 [obdclass]
[<0>] ll_fill_super+0x739/0x1190 [lustre]
[<0>] lustre_fill_super+0xf4/0x4a0 [lustre]
[<0>] mount_nodev+0x48/0xa0
[<0>] legacy_get_tree+0x27/0x40
[<0>] vfs_get_tree+0x25/0xb0
[<0>] do_mount+0x2e2/0x950
[<0>] ksys_mount+0xb6/0xd0
[<0>] __x64_sys_mount+0x21/0x30
[<0>] do_syscall_64+0x5b/0x1a0
[<0>] entry_SYSCALL_64_after_hwframe+0x65/0xca

server:
[<0>] llog_process_or_fork+0x43c/0x560 [obdclass]
[<0>] llog_process+0x10/0x20 [obdclass]
[<0>] class_config_parse_llog+0x1e9/0x3e0 [obdclass]
[<0>] mgc_process_cfg_log+0x709/0xd80 [mgc]
[<0>] mgc_process_log+0x6c3/0x800 [mgc]
[<0>] mgc_process_config+0xb53/0xe60 [mgc]
[<0>] lustre_process_log+0x5fa/0xad0 [obdclass]
[<0>] ll_fill_super+0x739/0x1190 [lustre]
[<0>] lustre_fill_super+0xf4/0x4a0 [lustre]
[<0>] mount_nodev+0x48/0xa0
[<0>] legacy_get_tree+0x27/0x40
[<0>] vfs_get_tree+0x25/0xb0
[<0>] do_mount+0x2e2/0x950
[<0>] ksys_mount+0xb6/0xd0
[<0>] __x64_sys_mount+0x21/0x30
[<0>] do_syscall_64+0x5b/0x1a0
[<0>] entry_SYSCALL_64_after_hwframe+0x65/0xca

any recommendations or insight would be appreciated

Thanks

Shane
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5680 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220706/21860fb8/attachment.bin>