[lustre-discuss] question mark when listing file after the upgrade

[Jane Liu]

Hi Andreas,

Thank you so much for the information provided. Without it, we might 
have struggled to find a solution for an extended period.

I attempted to rebuild the OI using LFSCK, but experienced hang-ups and 
Lustre crashes while executing "lctl lfsck_start." Later, we were able 
to address the OI issue by using the "-o resetoi" option during MDT 
mounting:

mount -o resetoi /mdt00

After resolving OI issue, I was then able to run LFSCK to conduct a 
comprehensive scan of the MDT.

Thanks again,
Jane



On 2023-05-03 14:52, Andreas Dilger wrote:
> This looks like https://jira.whamcloud.com/browse/LU-16655 [2] causing
> problems after the upgrade from 2.12.x to 2.15.[012] breaking the
> Object Index files.
> 
> A patch for this has already been landed to b2_15 and will be included
> in 2.15.3. If you've hit this issue, then you need to backup/delete
> the OI files (off of Lustre) and run OI Scrub to rebuild them.
> 
> I believe the OI Scrub/rebuild is described in the Lustre Manual.
> 
> Cheers, Andreas
> 
>> On May 3, 2023, at 09:30, Colin Faber via lustre-discuss
>> <lustre-discuss at lists.lustre.org> wrote:
> 
>> 
>> Hi,
>> 
>> What does your client log indicate? (dmesg / syslog)
>> 
>> On Wed, May 3, 2023, 7:32 AM Jane Liu via lustre-discuss
>> <lustre-discuss at lists.lustre.org> wrote:
>> 
>>> Hello,
>>> 
>>> I'm writing to ask for your help on one issue we observed after a
>>> major
>>> upgrade of a large Lustre system from RHEL7 + 2.12.9 to RHEL8 +
>>> 2.15.2.
>>> Basically we preserved MDT disk (VDisk on a VM) and also all OST
>>> disk
>>> (JBOD) in RHEL7 and then reinstalled RHEL8 OS and then attached
>>> those
>>> preserved disks to RHEL8 OS. However, I met an issue after the OS
>>> upgrade and lustre installation.
>>> 
>>> I believe the issue is related to metadata.
>>> 
>>> The old MDS was a virtual machine, and the MDT vdisk was preserved
>>> 
>>> during the upgrade. When a new VM was created with the same
>>> hostname and
>>> IP, the preserved MDT vdisk was attached to it. Everything seemed
>>> fine
>>> initially. However, after the client mount was completed, the file
>>> 
>>> listing displayed question marks, as shown below:
>>> 
>>> [root at experimds01 ~]# mount -t lustre 11.22.33.44 at tcp:/experi01
>>> /mntlustre/
>>> [root at experimds01 ~]# cd /mntlustre/
>>> [root at experimds01 mntlustre]# ls -l
>>> ls: cannot access 'experipro': No such file or directory
>>> ls: cannot access 'admin': No such file or directory
>>> ls: cannot access 'test4': No such file or directory
>>> ls: cannot access 'test3': No such file or directory
>>> total 0
>>> d????????? ? ? ? ?            ? admin
>>> d????????? ? ? ? ?            ? experipro
>>> -????????? ? ? ? ?            ? test3
>>> -????????? ? ? ? ?            ? test4
>>> 
>>> I shut down the MDT and ran "e2fsck -p
>>> /dev/mapper/experimds01-experimds01". It reported "primary
>>> superblock
>>> features different from
>>> backup, check forced."
>>> 
>>> [root at experimds01 ~]# e2fsck -p
>>> /dev/mapper/experimds01-experimds01
>>> experi01-MDT0000 primary superblock features different from
>>> backup,
>>> check forced.
>>> experi01-MDT0000: 9493348/429444224 files (0.5% non-contiguous),
>>> 109369520/268428864 blocks
>>> 
>>> Running e2fsck again showed that the filesystem was clean.
>>> [root at experimds01 /]# e2fsck -p
>>> /dev/mapper/experimds01-experimds01
>>> experi01-MDT0000: clean, 9493378/429444224 files,
>>> 109369610/268428864
>>> blocks
>>> 
>>> However, the issue persisted. The file listing continued to
>>> display
>>> question marks.
>>> 
>>> Do you have any idea what could be causing this problem and how to
>>> fix
>>> it? By the way, I have an e2image backup of the MDT from the
>>> RHEL7 system just in case we need fix it using the backup. Also,
>>> after
>>> the upgrade, the command "lfs df" shows that all OSTs and MDT
>>> are fine.
>>> 
>>> Thank you in advance for your assistance.
>>> 
>>> Best regards,
>>> Jane
>>> _______________________________________________
>>> lustre-discuss mailing list
>>> lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org [1]
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org [1]
> 
> 
> Links:
> ------
> [1] 
> https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!P4SdNyxKAPE!EjGkARsk0Yp3SMoPZG3ShFH-7yqJP8eiiu01CbPIWhkz9haU5QrbeCbYwZNFupImfJ2pe5WFfJT-Ci0S-Qw0nsps$
> [2] 
> https://urldefense.com/v3/__https://jira.whamcloud.com/browse/LU-16655__;!!P4SdNyxKAPE!EjGkARsk0Yp3SMoPZG3ShFH-7yqJP8eiiu01CbPIWhkz9haU5QrbeCbYwZNFupImfJ2pe5WFfJT-Ci0S-fnmcNE6$