<div dir="ltr">OK. Let me provide more details about the upgrade from 2.12.X to 2.15.X.<div><br></div><div>We have several production clusters. Due to different requirements and hardware, each has slightly different Lustre configurations. Our Linux distribution is Debian. Sometimes due to hardware spec, we need to use different Linux kernel versions. So we always have to compile Kernel (vanilla Linux kernel), MLNX_OFED, e2fsprog / ZFS, and Lustre. Here I temporarily skip the compilation (that involved a lot of details), just mention the upgrade procedure for two different cases.</div><div><br></div><div>1. Two of our clusters have Lustre-2.12.6, with MDT with ZFS backend, and OST with ZFS backend, where ZFS is 0.7.13. From the compatibility list mentioned in documentation, the upgrade involves both ZFS upgrade to 2.0.7 and Lustre upgrade to 2.15.X. It was very smooth and successful. The procedure is:</div><div> 1) Compile and install new versions of Linux kernel, MLNX_OFED, ZFS, and Lustre. Depending on the hardware, some servers we use Linux kernel 4.19.X + MLNX_OFED-4.6, or Linux kernel 4.19.X + MLNX_OFED-4.9, or Linux kernel 5.4.X + MLNX_OFED-4.9. They are all compatible to Lustre-2.15.X and ZFS-2.0.7.</div><div> 2) After installation of the new software, start up ZFS first:</div><div>```</div><div>modprobe zfs</div><div>zpool import <lustre_pool_name></div><div>zpool status <lustre_pool_name></div><div>```</div><div> There will be messages about ZFS pool needs to be upgraded to enable new features. But after upgrade, the ZFS pool will not be compatible to the old ZFS version anymore. The command to upgrade ZFS pool is (for all the MDT and OST pools):</div><div>```</div><div>zpool upgrade <lustre_pool_name></div><div>zpool status <lustre_pool_name></div><div>```</div><div> Checking zpool status again, the warning messages were disappeared.</div><div> 3) I checked the Lustre Operation Manual again, and saw that in the recent version it does not need to run tunefs.lustre --writeconf command. Sorry that it was my fault. But in our case, we run this command whenever doing Lustre upgrade. Note that to run it, we have to run it in all the Lustre MDT and OST pools. It cleared out the logs and will regenerate it when mounting Lustre.</div><div><br></div><div>2. One of our clusters have Lustre-2.12.6, with MDT with ldiskfs backend, and OST with ZFS backend, where ZFS is 0.7.13. We have to upgrade it to e2fsprogs-1.47.0, ZFS-2.0.7 and Lustre-2.15.3. The upgrade of OST part is exactly the same as the above, so I don't repeat. The major challenge is MDT with ldiskfs. What I have done is</div><div> 1) After installing all the new version of the software, running tunefs.lustre --writeconf (for all MDT and OST). Probably this is the wrong step for the upgrade to Lustre-2.15.X.</div><div> 2) According to Lustre Operation manual chapter 17, to upgrade Lustre from 2.13.0 and before to 2.15.X, we should run</div><div><pre class="gmail-screen" style="background-color:rgb(224,224,224);padding:5px;color:rgb(0,0,0)">tune2fs -O ea_inode /dev/<em class="gmail-replaceable"><code>mdtdev</code></em></pre></div><div>After that, as I have posted, we encounter problem of mounting MDT. Then we cure this problem by following the section 18 of Lustre Operation manual.<br></div><div><br></div><div>My personal suggestions are:</div><div>1. In the future, to do major revision upgrade for our production run systems (say, 2.12.X to 2.15.X, or 2.15.X to 2.16 or later), I will develop a small testing system, installing exactly the same software of the production run, and test the upgrade, to make sure that every steps are correct. We did this for upgrading Lustre with ZFS backend. But this time due to time pressure, we skip this step for upgrading Lustre with ldiskfs backend. I think no matter what situation, it is still worth to do this step in order to avoid any risks.</div><div><br></div><div>2. Currently compiling Lustre with ldiskfs backend is still a nightmare. The ldiskfs code is not a self-contained, stand along code. It actually copied codes from the kernel ext4 code, did a lot of patches, and then did the compilation, on the fly. So we have to be very careful to select the Linux kernel, to select a compatible one for both our hardware and Lustre version. The ZFS backend is much more cleaner. It is a stand along and self-contained code. We don't need to do patches on the fly. So I would like to suggest the Lustre developer to consider to make the ldiskfs to be a stand along and self-contained code in the future release. That will bring us a lot of convenient.</div><div><br></div><div>Hope that the above experiences could be useful to our community.</div><div><br></div><div>ps. Lustre Operation Manual: <a href="https://doc.lustre.org/lustre_manual.xhtml#Upgrading_2.x">https://doc.lustre.org/lustre_manual.xhtml#Upgrading_2.x</a></div><div><br></div><div>Best Regards,</div><div><br></div><div>T.H.Hsieh</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Audet, Martin <<a href="mailto:Martin.Audet@cnrc-nrc.gc.ca">Martin.Audet@cnrc-nrc.gc.ca</a>> 於 2023年9月27日 週三 上午3:44寫道：<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-8366546904526686493">
<div dir="ltr">
<div id="m_-181371690363826012divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p>Hello all,</p>
<p><br>
</p>
<p>I would appreciate if the community would give more attention to this issue because upgrading from 2.12.x to 2.15.x, two LTS versions, is something that we can expect many cluster admin will try to do in the next few months...</p>
<p><br>
</p>
<p>We ourselves plan to upgrade a small Lustre (production) system from 2.12.9 to 2.15.3 in the next couple of weeks...</p>
<p></p>
<div><br>
</div>
<div><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">After seeing problems reports like this we start feeling a bit
nervous...</span><br>
</div>
<p></p>
<p><br>
</p>
<p>The documentation for doing this major update appears to me as not very specific...</p>
<p><br>
</p>
<p>In this document for example, <a href="https://doc.lustre.org/lustre_manual.xhtml#upgradinglustre" id="m_-181371690363826012LPlnk112373" style="font-size:12pt" target="_blank">https://doc.lustre.org/lustre_manual.xhtml#upgradinglustre</a> , the update process appears not
so difficult and there is no mention of using "<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">tunefs.lustre --writeconf"
for this kind of update.</span></p>
<p><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px"><br>
</span></p>
<p><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">Or am I missing something ?</span></p>
<p><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px"><br>
</span></p>
<p><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">Thanks in advance for providing more tips for this kind of update.</span></p>
<p><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px"><br>
</span></p>
<p><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">Martin Audet</span></p>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block;width:98%">
<div id="m_-181371690363826012divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> lustre-discuss <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of Tung-Han Hsieh via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a>><br>
<b>Sent:</b> September 23, 2023 2:20 PM<br>
<b>To:</b> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> [lustre-discuss] Cannot mount MDT after upgrading from Lustre 2.12.6 to 2.15.3</font>
<div> </div>
</div>
<div>
<p style="text-align:left"><span style="color:rgb(0,0,0)"><strong>***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC.</strong></span></p>
<div>
<div dir="ltr">Dear All,
<div><br>
</div>
<div>Today we tried to upgrade Lustre file system from version 2.12.6 to 2.15.3. But after the work, we cannot mount MDT successfully. Our MDT is ldiskfs backend. The procedure of upgrade is</div>
<div><br>
</div>
<div>1. Install the new version of e2fsprogs-1.47.0</div>
<div>2. Install Lustre-2.15.3</div>
<div>3. After reboot, run: tunefs.lustre --writeconf /dev/md0</div>
<div><br>
</div>
<div>Then when mounting MDT, we got the error message in dmesg:</div>
<div><br>
</div>
<div>===========================================================</div>
<div>[11662.434724] LDISKFS-fs (md0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc<br>
[11662.584593] Lustre: 3440:0:(scrub.c:189:scrub_file_load()) chome-MDT0000: reset scrub OI count for format change (LU-16655)<br>
[11666.036253] Lustre: MGS: Logs for fs chome were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0<br>
[11666.523144] Lustre: chome-MDT0000: Imperative Recovery not enabled, recovery window 300-900<br>
[11666.594098] LustreError: 3440:0:(mdd_device.c:1355:mdd_prepare()) chome-MDD0000: get default LMV of root failed: rc = -2<br>
[11666.594291] LustreError: 3440:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2<br>
[11666.594951] Lustre: Failing over chome-MDT0000<br>
[11672.868438] Lustre: 3440:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1695492248/real 1695492248] req@000000005dfd9b53 x1777852464760768/t0(0) o251->MGC192.168.32.240@o2ib@0@lo:26/25 lens 224/224 e 0
to 1 dl 1695492254 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:''<br>
[11672.925905] Lustre: server umount chome-MDT0000 complete<br>
[11672.926036] LustreError: 3440:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -2<br>
[11872.893970] LDISKFS-fs (md0): mounted filesystem with ordered data mode. Opts: (null)<br>
</div>
<div>============================================================</div>
<div><br>
</div>
<div>Could anyone help to solve this problem ? Sorry that it is really urgent.</div>
<div><br>
</div>
<div>Thank you very much.</div>
<div><br>
</div>
<div>T.H.Hsieh</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div>