[lustre-discuss] 1.8 client on 3.13.0 kernel

[Martin Hecht]

Hi,

A backup is always a good idea if feasible. It gives you at least the
chance to go back and start over again. However, a backup of the MDT
alone wouldn't help much, because as soon as you put the file system
online and users start to work on their files, also the content of the
OSTs will change. Restoring the MDT backup would cause the MDT being out
of sync with the OSTs.
You notice some bugs immediately during the upgrade (e.g. the one with
the CATALOGS file which prevents you from starting the MDT again), but
some others (e.g. quota bugs or the one about the FID) pop up a few
hours or days after you have started production again, and then you have
to make a decision. Even if you have a full backup, it's always a
trade-off if you decide to give it a try to fix the problems based on
the state where you are or if you go back and restore the backup.
But even then, you have to put some measures in place which ensure that
you won't run into the same problem again. In the worst case it's
reinstalling the servers with the lustre version you have used before. A
full backup at least gives you this fallback for the worst case
scenario. It can also be useful for offline analysis in case you have to
investigate what's going wrong.
In the particular case with the FID in the directory a file level backup
of the MDT wouldn't have been of that much help, because you also have
to backup the extended attributes. There is a section in the lustre
manual how to do this. However, these structures must be converted (at
least if you want to make use of the fid_in_dirent feature). If I'm not
mistaken, the structures were ok right after the upgrade and the
subsequent lfsck run, but the ldiskfs backend contained a bug which
caused things to be overwritten, when users started to move files
somewhere else. Lustre 2.4.3 is marked as affected in LU-5626.

regards,
Martin

On 09/11/2015 04:14 PM, Patrick Farrell wrote:
> Having an MDT backup might perhaps have allowed recovery and trying an improved upgrade process and/or upgrading to a version with the fixes in it.  It's not a bad idea if practical.  (And yes, the changes are MDT specific.)
>
> By the way, the fid-in-dirent bug that Martin described is fixed in the most recent 2.5 from Intel, but I don't think it's fixed in 2.4?  Unsure.
> But I'd recommend targeting 2.5 as the destination version for an upgrade.
> ________________________________________
> From: lustre-discuss [lustre-discuss-bounces at lists.lustre.org] on behalf of Chris Hunter [chris.hunter at yale.edu]
> Sent: Friday, September 11, 2015 8:02 AM
> To: lustre-discuss at lists.lustre.org
> Subject: Re: [lustre-discuss] 1.8 client on 3.13.0 kernel
>
> Hi
> I believe FID & dirdata feature changes would only affect the MDT during
> a lustre upgrade. In hindsight/retrospective do you think a file-level
> backup/restore of the MDT would have avoided some of these issues ?
>
> thanks
> chris hunter
>
>> On 9/10/15 11:17 AM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
>>> Lewis,
>>>
>>> I did an upgrade from Lustre 1.8.6 to 2.4.3 on our servers, and for the most part things went pretty good.  I?ll chime in on a couple of Martin?s points and mention a few other things.
>>>
>>>> On Sep 10, 2015, at 9:30 AM, Martin Hecht <hecht at hlrs.de> wrote:
>>>>
>>>> In any case the file systems should be clean before starting the
>>>> upgrade, so I would recommend to run e2fsck on all targets and repair
>>>> them before starting the upgrade. We did so, but unfortunately our
>>>> e2fsprogs were not really up to date and after our lustre upgrade a lot
>>>> of fixes for e2fsprogs were committed to whamclouds e2fsprogs git. So,
>>>> probably some errors on the file systems were still present, but
>>>> unnoticed when we did the upgrade.
>>> This is a very important point.  While I didn?t run e2fsck before the upgrade (but maybe I should have), I made sure to install the latest e2fsprogs.
>>>
>>>> Lustre 2 introduces the FID (which is something like an inode number,
>>>> where lustre 1.8 used the inode number of the underlying ldiskfs, but
>>>> with the possibility to have several MDTs in one file system a
>>>> replacement was needed). The FID is stored in the inode, but it can also
>>>> be activated that the FIDs are stored in the directory node, which makes
>>>> lookups faster, especially when there are many files in a directory.
>>>> However, there were bugs in the code that takes care about adding the
>>>> FID to the directory entry when the file system is converted from 1.8 to
>>>> 2.x. So, I would recommend to use a version in which these bug are
>>>> solved. We went to 2.4.1 that time. By default this fid_in_dirent
>>>> feature is not automatically enabled, however, this is the only point
>>>> where a performance boost may be expected... so we took the risk to
>>>> enable this... and ran into some bugs.
>>> Enabling fid_in_dirent prevents you from backing out of the upgrade.  In theory, if you upgraded to Lustre 2.x without enabling fid_in_dirent, you could always revert back to Lustre 1.8.  We tried this on a test system, and the downgrade seemed to work.  However, this was a small scale test and I have never tried it on a production file system.  But if you want to minimize possible complications, you could always leave this disabled for a while after the updgrade, and then if things are going well, enable it later on.
>>>
>>>> LU-4504 quota out of sync: turn off quota, run e2fsck, turn it on again
>>>> - I believe that's something which must be done anyhow quite often,
>>>> because there is no quotacheck anymore. It's run in the background when
>>>> enabling quotas, but file systems have to be unmounted for this.
>>> We didn?t exactly hit this bug, but I will mention that we have had a couple of instance where e2fsck complained about problems on an OST, and it turned out that we had to disable and re-enable quotas on the OST to correct the issue.
>>>
>>>> LU-4743: We had to remove the CATALOGS file on another file system
>>>> (otherwise the MDT wouldn't mount)
>>> We hit this problem.
>>>
>>> Someone I know had to do a Lustre upgrade, and they suggested that I apply a patch for LU-4708 (which I did).  But if you upgrade to Lustre 2.5.2 or later, that patch should already be included.
>>>
>>> My only other advice is to test as much as possible prior to the upgrade.  If you have a little test hardware, install the same Lustre 1.8 version you are currently running in production and then try upgrading that to the new Lustre version.  I think preparation is the key.  I think I spent about 2 months reading about upgrade procedures, talking with others who have upgraded, reading JIRA bug reports, and running tests on hardware.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2252 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20150914/b7c01118/attachment.bin>