[lustre-discuss] recovery MDT ".." directory entries (LU-5626)

[Dilger, Andreas]

On 2015/11/04, 02:42, "lustre-discuss on behalf of Martin Hecht"
<lustre-discuss-bounces at lists.lustre.org on behalf of hecht at hlrs.de> wrote:

>On 11/04/2015 03:23 AM, Patrick Farrell wrote:
>> PAF: Remember, the specific conditions are pretty tight.  Created under
>>1.8, not empty (if it's empty, the .. dentry is not misplaced when
>>moved) but also non-htree, then moved with dirdata enabled, and then
>>grown to this larger size.  How many existing (small) directories do you
>>move and then add a bunch of files to?  It's a pretty rare operation.
>>We only hit it at Martin's site because of an automated tool they have
>>to re-arrange user/job directories.
>Well, not only because of the tool. Especially, because when the
>directories have been moved by the tool, no files are added anymore.
>However, our mechanism gives a reason to the users to move their data
>from time to time (that's not the intention of the mechanism, but that's
>how some users react).
>
>But I'm not quite sure anymore if moving the directories is really a
>precondition to run into LU-5626.
>We have run the background lfsck which adds the FID to the existing
>dentries. This might be an important detail, because in our case a
>second '..' entry containing the FID was presumably created by lfsck (in
>the wrong place), and not by moving the directory. To my current
>understanding the user then only has to add some files to trigger the
>LBUG.
>A subsequent e2fsck will not only find this particular directory but all
>other small directories with a '..' entry in the wrong place. When
>e2fsck tries to fix these directories, some entries are overwritten by
>the FID and these files are then moved to lost+found.

Note that newer versions of LFSCK namespace checking (2.6 or 2.7, don't
recall offhand) will be able to return such entries from lost+found back
into the proper parent directory in the namespace, assuming they were
created under 2.x.  Lustre stores an extra "link" xattr on each inode with
the filename and parent directory FID for each link to the file (up to the
available xattr space for each inode), so in case of directory corruption
it would be possible to rebuild the directory structure just from the
"link" xattrs on each file.

In the meantime, I attached a script to LU-5626 that could be used to
re-link files from lost+found into the right directory and filename based
on the output from e2fsck.  It is a bit rough (needs manual editing of
pathnames), but may be useful if someone has hit this problem.

Cheers, Andreas

>If one of these first entries happens to be a small subdirectory, I
>believe there is a chance to run into the same issue again, when you
>move everything back to the original location after the e2fsck and
>someone starts adding files in these subdirectories.
>
>However, the preconditions are still quite narrow: small directories,
>not empty, created without fid, then converted by lfsck (or
>alternatively moved to a different place which would also create the
>second '..' entry). To trigger the LBUG files need to be added to one of
>these directories and for a second occurrence of the LBUG the same
>conditions must hold for another subdirectory which must have been at
>the very beginning of the directory.
>
>Martin
>
>
>


Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division