[lustre-discuss] question mark when listing file after the upgrade

[Jane Liu]

Hello,

I'm writing to ask for your help on one issue we observed after a major 
upgrade of a large Lustre system from RHEL7 + 2.12.9 to RHEL8 + 2.15.2. 
Basically we preserved MDT disk (VDisk on a VM) and also all OST disk 
(JBOD) in RHEL7 and then reinstalled RHEL8 OS and then attached those 
preserved disks to RHEL8 OS. However, I met an issue after the OS 
upgrade and lustre installation.

I believe the issue is related to metadata.

The old MDS was a virtual machine, and the MDT vdisk was preserved 
during the upgrade. When a new VM was created with the same hostname and 
IP, the preserved MDT vdisk was attached to it. Everything seemed fine 
initially. However, after the client mount was completed, the file 
listing displayed question marks, as shown below:

[root at experimds01 ~]# mount -t lustre 11.22.33.44 at tcp:/experi01 
/mntlustre/
[root at experimds01 ~]# cd /mntlustre/
[root at experimds01 mntlustre]# ls -l
ls: cannot access 'experipro': No such file or directory
ls: cannot access 'admin': No such file or directory
ls: cannot access 'test4': No such file or directory
ls: cannot access 'test3': No such file or directory
total 0
d????????? ? ? ? ?            ? admin
d????????? ? ? ? ?            ? experipro
-????????? ? ? ? ?            ? test3
-????????? ? ? ? ?            ? test4

I shut down the MDT and ran "e2fsck -p 
/dev/mapper/experimds01-experimds01". It reported "primary superblock 
features different from
  backup, check forced."

[root at experimds01 ~]# e2fsck -p /dev/mapper/experimds01-experimds01
experi01-MDT0000 primary superblock features different from backup, 
check forced.
experi01-MDT0000: 9493348/429444224 files (0.5% non-contiguous), 
109369520/268428864 blocks

Running e2fsck again showed that the filesystem was clean.
[root at experimds01 /]# e2fsck -p /dev/mapper/experimds01-experimds01
experi01-MDT0000: clean, 9493378/429444224 files, 109369610/268428864 
blocks

However, the issue persisted. The file listing continued to display 
question marks.

Do you have any idea what could be causing this problem and how to fix 
it? By the way, I have an e2image backup of the MDT from the
RHEL7 system just in case we need fix it using the backup. Also, after 
the upgrade, the command "lfs df" shows that all OSTs and MDT
  are fine.

Thank you in advance for your assistance.

Best regards,
Jane