[lustre-discuss] question mark when listing file after the upgrade

[Andreas Dilger]

This looks like https://jira.whamcloud.com/browse/LU-16655 causing problems after the upgrade from 2.12.x to 2.15.[012] breaking the Object Index files.

A patch for this has already been landed to b2_15 and will be included in 2.15.3. If you've hit this issue, then you need to backup/delete the OI files (off of Lustre) and run OI Scrub to rebuild them.

I believe the OI Scrub/rebuild is described in the Lustre Manual.

Cheers, Andreas

On May 3, 2023, at 09:30, Colin Faber via lustre-discuss <lustre-discuss at lists.lustre.org> wrote:


Hi,

What does your client log indicate? (dmesg / syslog)

On Wed, May 3, 2023, 7:32 AM Jane Liu via lustre-discuss <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>> wrote:
Hello,

I'm writing to ask for your help on one issue we observed after a major
upgrade of a large Lustre system from RHEL7 + 2.12.9 to RHEL8 + 2.15.2.
Basically we preserved MDT disk (VDisk on a VM) and also all OST disk
(JBOD) in RHEL7 and then reinstalled RHEL8 OS and then attached those
preserved disks to RHEL8 OS. However, I met an issue after the OS
upgrade and lustre installation.

I believe the issue is related to metadata.

The old MDS was a virtual machine, and the MDT vdisk was preserved
during the upgrade. When a new VM was created with the same hostname and
IP, the preserved MDT vdisk was attached to it. Everything seemed fine
initially. However, after the client mount was completed, the file
listing displayed question marks, as shown below:

[root at experimds01 ~]# mount -t lustre 11.22.33.44 at tcp:/experi01
/mntlustre/
[root at experimds01 ~]# cd /mntlustre/
[root at experimds01 mntlustre]# ls -l
ls: cannot access 'experipro': No such file or directory
ls: cannot access 'admin': No such file or directory
ls: cannot access 'test4': No such file or directory
ls: cannot access 'test3': No such file or directory
total 0
d????????? ? ? ? ?            ? admin
d????????? ? ? ? ?            ? experipro
-????????? ? ? ? ?            ? test3
-????????? ? ? ? ?            ? test4

I shut down the MDT and ran "e2fsck -p
/dev/mapper/experimds01-experimds01". It reported "primary superblock
features different from
  backup, check forced."

[root at experimds01 ~]# e2fsck -p /dev/mapper/experimds01-experimds01
experi01-MDT0000 primary superblock features different from backup,
check forced.
experi01-MDT0000: 9493348/429444224 files (0.5% non-contiguous),
109369520/268428864 blocks

Running e2fsck again showed that the filesystem was clean.
[root at experimds01 /]# e2fsck -p /dev/mapper/experimds01-experimds01
experi01-MDT0000: clean, 9493378/429444224 files, 109369610/268428864
blocks

However, the issue persisted. The file listing continued to display
question marks.

Do you have any idea what could be causing this problem and how to fix
it? By the way, I have an e2image backup of the MDT from the
RHEL7 system just in case we need fix it using the backup. Also, after
the upgrade, the command "lfs df" shows that all OSTs and MDT
  are fine.

Thank you in advance for your assistance.

Best regards,
Jane
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20230503/3e715bbe/attachment-0001.htm>