<div dir="ltr">Greetings,<div><br></div><div>We have experienced some type of issue that is causing both of our MDS servers to only be able to mount the mdt device in read only mode.  Here are some of the error messages we are seeing in the log files below.   We lost our Lustre expert a while back and we are not sure how to proceed to troubleshoot this issue.   Can anybody provide us guidance on how to proceed?</div><div><br></div><div>Thanks,</div><div><br></div><div>Mike</div><div><br></div><div><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: INFO: task mount.lustre:4123 blocked for more than 120 seconds.</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: mount.lustre    D ffff9f27a3bc5230     0  4123      1 0x00000086</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: Call Trace:</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbb585da9>] schedule+0x29/0x70</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbb5838b1>] schedule_timeout+0x221/0x2d0</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbaf6b8e5>] ? tracing_is_on+0x15/0x30</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbaf6f5bd>] ? tracing_record_cmdline+0x1d/0x120</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbaf77d9b>] ? probe_sched_wakeup+0x2b/0xa0</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbaed7d15>] ? ttwu_do_wakeup+0xb5/0xe0</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbb58615d>] wait_for_completion+0xfd/0x140</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbaedb990>] ? wake_up_state+0x20/0x20</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0f529a4>] llog_process_or_fork+0x244/0x450 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0f52bc4>] llog_process+0x14/0x20 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0f85d05>] class_config_parse_llog+0x125/0x350 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0a69fc0>] mgc_process_cfg_log+0x790/0xc40 [mgc]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0a6d4cc>] mgc_process_log+0x3dc/0x8f0 [mgc]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0a6e15f>] ? config_recover_log_add+0x13f/0x280 [mgc]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0f8df40>] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0a6eb2b>] mgc_process_config+0x88b/0x13f0 [mgc]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0f91b58>] lustre_process_log+0x2d8/0xad0 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0e5a177>] ? libcfs_debug_msg+0x57/0x80 [libcfs]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0f7c8b9>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0fc08f4>] server_start_targets+0x13a4/0x2a20 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0f94bb0>] ? lustre_start_mgc+0x260/0x2510 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0f8df40>] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0fc303c>] server_fill_super+0x10cc/0x1890 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0f97a08>] lustre_fill_super+0x468/0x960 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0f975a0>] ? lustre_common_put_super+0x270/0x270 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbb0510cf>] mount_nodev+0x4f/0xb0</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffc0f8f9a8>] lustre_mount+0x38/0x60 [obdclass]</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbb051c4e>] mount_fs+0x3e/0x1b0</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbb0707a7>] vfs_kern_mount+0x67/0x110</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbb072edf>] do_mount+0x1ef/0xd00</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbb049d7a>] ? __check_object_size+0x1ca/0x250</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbb0288ec>] ? kmem_cache_alloc_trace+0x3c/0x200</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbb073d33>] SyS_mount+0x83/0xd0</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:12:14 hyd-mds1 kernel: [<ffffffffbb592ed2>] system_call_fastpath+0x25/0x2a</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:13:14 hyd-mds1 kernel: LNet: 4458:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 172.16.100.4@o2ib: 9 seconds</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:13:14 hyd-mds1 kernel: LNet: 4458:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 239 previous similar messages</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:14:14 hyd-mds1 kernel: INFO: task mount.lustre:4123 blocked for more than 120 seconds.</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:14:14 hyd-mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Jun 20 15:14:14 hyd-mds1 kernel: mount.lustre    D ffff9f27a3bc5230     0  4123      1 0x00000086</span></p></div><div><br></div><div>dumpe2fs seems to show that the file systems are clean i.e.</div><div><br></div><div><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">dumpe2fs 1.45.6.wc1 (20-Mar-2020)</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Filesystem volume name:   hydra-MDT0000</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Last mounted on:          /</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Filesystem UUID:          3ae09231-7f2a-43b3-a4ee-7f36080b5a66</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Filesystem magic number:  0xEF53</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Filesystem revision #:    1 (dynamic)</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Filesystem features:      has_journal ext_attr resize_inode dir_index filetype mmp flex_bg dirdata sparse_super large_file huge_file uninit_bg dir_nlink quota</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Filesystem flags:         signed_directory_hash </span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Default mount options:    user_xattr acl</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Filesystem state:         clean</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Errors behavior:          Continue</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Filesystem OS type:       Linux</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Inode count:              2247671504</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Block count:              1404931944</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Reserved block count:     70246597</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Free blocks:              807627552</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Free inodes:              2100036536</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">First block:              0</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Block size:               4096</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Fragment size:            4096</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Reserved GDT blocks:      1024</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Blocks per group:         20472</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Fragments per group:      20472</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Inodes per group:         32752</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Inode blocks per group:   8188</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Flex block group size:    16</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Filesystem created:       Thu Aug  8 14:21:01 2019</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Last mount time:          Tue Jun 20 15:19:03 2023</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Last write time:          Wed Jun 21 10:43:51 2023</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Mount count:              38</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Maximum mount count:      -1</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Last checked:             Thu Aug  8 14:21:01 2019</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Check interval:           0 (<none>)</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Lifetime writes:          219 TB</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Reserved blocks uid:      0 (user root)</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Reserved blocks gid:      0 (group root)</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">First inode:              11</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Inode size:<span style="white-space-collapse: preserve;">    </span>          1024</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Required extra isize:     32</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Desired extra isize:      32</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Journal inode:            8</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Default directory hash:   half_md4</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Directory Hash Seed:      2e518531-82d9-4652-9acd-9cf9ca09c399</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Journal backup:           inode blocks</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">MMP block number:         1851467</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">MMP update interval:      5</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">User quota inode:         3</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Group quota inode:        4</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Journal features:         journal_incompat_revoke</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Journal size:             4096M</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Journal length:           1048576</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Journal sequence:         0x0a280713</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">Journal start:            0</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">MMP_block:</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">    mmp_magic: 0x4d4d50</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">    mmp_check_interval: 6</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">    mmp_sequence: 0xff4d4d50</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">    mmp_update_date: Wed Jun 21 10:43:51 2023</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">    mmp_update_time: 1687358631</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">    mmp_node_name: <a href="http://hyd-mds1.uncc.edu/" target="_blank">hyd-mds1.uncc.edu</a></span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(16,8,182)"><span style="font-variant-ligatures:no-common-ligatures">    mmp_device_name: dm-0</span></p></div></div>