<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Scott,</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">This sounds great. Slower, but safer. You might want to integrate the pool suggestion Jongwoo made in the other recent thread in order to control allocations to your new OSTs (assuming you're trying to stay live during
most of this).</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">- Patrick</p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Scott Wood <woodystrash@hotmail.com><br>
<b>Sent:</b> Thursday, February 28, 2019 6:15:54 PM<br>
<b>To:</b> Patrick Farrell; Jongwoo Han<br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Draining and replacing OSTs with larger volumes</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
My Thanks to both Jongwoo and Patrick for your responses.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Great advice to do a practice run in a virtual environment but I'm lucky enough to have a physical one. <span style="font-family:Calibri,Helvetica,sans-serif; background-color:rgb(255,255,255); display:inline!important">I have a testbed that has the same versions
of all software but with iscsi targets as the OSTs, rather than physical arrays, and not so many OSTs (8 in the testbed and 60 in production) I do use it for test upgrades and fully intend to do a dry run there.</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Jongwoo, to address your point, yes the rolling migration is forced, as we only have two new arrays, and 10 existing arrays which we can upgrade the drives in. You asked about OST sizes. OSTs are 29TB, six per array, two arrays per OSS pair, 5 OSS pairs.
I also expect the migrate-replace-migrate-replace to be painfully slow, but with the hardware at hand, it's the only option. I was figuring they may take a few weeks to drain each pair of arrays. As for the rolling upgrade, based on yours and Patrick's responses,
we'll skip that to keep things cleaner.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Taking your points in to consideration, the amended plan will be:</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
1) Deploy a new HA pair of OSSs with arrays populated with OSTs that are twice the size of our current ones, but stick with the existing v2.10.3</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
2) Remove the 12 OSTs that are connected to my oldest HA pair of OSSs as described in 14.9.3, using 12 parallel migrate processes across 12 clients</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
3) Repopulate those arrays with the larger drives and make new 12 OSTs from scratch, with fresh indices, and bring them online</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
4) Repeat steps 2 and 3 for the four remaining original HA pairs of OSSs</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
5) Take a break and let the dust settle</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
6) At a later date, have a scheduled outage and upgrade from 2.10.3 to whatever the current maintenance release is</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Again, you feedback is appreciated.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Cheers</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Scott</div>
<div id="x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Patrick Farrell <pfarrell@whamcloud.com><br>
<b>Sent:</b> Thursday, 28 February 2019 11:06 PM<br>
<b>To:</b> Jongwoo Han; Scott Wood<br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Draining and replacing OSTs with larger volumes</font>
<div> </div>
</div>
<div>Scott,<br>
<br>
I’d like to strongly second all of Jongwoo’s advice, particularly that about adding new OSTs rather than replacing existing ones, if possible. That procedure is so much simpler and involves a lot less messing around “under the hood”. It takes you from a complex
procedure with many steps to, essentially, copying a bunch of data around while your file system remains up, and adding and removing a few OSTs at either end.<br>
<br>
It would also be non-destructive for your existing data. One of the scary things about the original proposed process is that if something goes wrong partway through, the original data is already gone (or at least very hard to get).<br>
<br>
Regards,<br>
- Patrick
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Jongwoo Han <jongwoohan@gmail.com><br>
<b>Sent:</b> Thursday, February 28, 2019 5:36:54 AM<br>
<b>To:</b> Scott Wood<br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Draining and replacing OSTs with larger volumes</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="x_x_x_gmail_quote">
<div dir="ltr" class="x_x_x_gmail_attr">On Thu, Feb 28, 2019 at 11:09 AM Scott Wood <<a href="mailto:woodystrash@hotmail.com">woodystrash@hotmail.com</a>> wrote:<br>
</div>
<blockquote class="x_x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi folks,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Big upgrade process in the works and I had some questions. Our current infrastructure has 5 HA pairs of OSSs and arrays with an HA pair of management and metadata servers who also share an array, all running lustre 2.10.3. Pretty standard stuff. Our upgrade
plan is as follows:</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
1) Deploy a new HA pair of OSSs with arrays populated with OSTs that are twice the size of our originals.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
2) Follow the process in section 14.9 of the lustre docs to drain all OSTs in one of existing the HA pairs' arrays</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
3) Repopulate the first old pair of deactivated and drained arrays with new larger drives</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
4) Upgrade the offline OSSs from 2.10.3 to 2.10.latest?</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
5) Return them to service</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
6) Repeat steps 2-4 for the other 4 old HA pairs of OSSs and OSTs</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I'd expect this would be doable without downtime as we'd only be taking arrays offline that have no objects on them, and we've added new arrays and OSSs before with no issues. I have a few questions before we begin the process:</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
1) My interpretation of the docs is that we OK to install them with 2.10.6 (or 2.10.7, if it's out), as rolling upgrades withing X.Y are supported. Is that correct?</div>
</div>
</blockquote>
<div><br>
</div>
<div>In theory, rolling upgrade should work, but generally recommended upgrade procedure is to stop filesystem and unmount all MDS and OSS, upgrade package and bring them up. This will prevent human errors during repeated per-server upgrade. </div>
<div>When it is done correctly, It will take not more than 2 hours.</div>
<div> </div>
<blockquote class="x_x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
2) Until the whole process is complete, we'll have imbalanced OSTs. I know that's not ideal, but is it all that big an issue </div>
</div>
</blockquote>
<div><br>
</div>
<div>Rolling upgrade will cause imbalance, but after long run, the files will be assigned will be evenly distributed. No need to worry about it on one-shot upgrade scenario. </div>
<div> </div>
<blockquote class="x_x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
3) When draining the OSTs of files, section 14.9.3, point 2.a. states that the lfs find |lfs migrate can take multiple OSTs as args, but I thought it would be better to run one instance of that per OST and distribute them across multiple clients . Is that
reasonable (and faster)?</div>
</div>
</blockquote>
<div><br>
</div>
<div>Parallel redistribute is generally faster than one-by-one. If the MDT can endure scanning load, run multiple migrate processes each for against one OST</div>
<blockquote class="x_x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
4) When the drives are replaced with bigger ones, can the original OST configuration files be restored to them as described in Docs section 14.9.5, or due the the size mismatch, will that be bad?</div>
</div>
</blockquote>
<div><br>
</div>
<div>Since this process will treat objects as files, the configurations should go as same.</div>
<div> </div>
<blockquote class="x_x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
5) What questions should I be asking that I haven't thought of?</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I do not know the size of OSTs to deal with, but I think migrate(empty)-replace-migrate-replace is really painful process as it will take long time. If circumtances allow, I suggest add all new OST arrays to OSS with new OST nums, migrate OST objects,
deactivate and remove old OSTs. </div>
<div> </div>
<blockquote class="x_x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
If that all goes well, and we did upgrade the OSSs to a newer 2.10.x, we'd follow it up with a migration of the MGT and MDT to one of the management servers, upgrade the other, fail them back, upgrade the second, and rebalance the MDT and MGT services back
across the two. We'd expect the usual pause in services as those migrate but other than that, fingers crossed, should all be good. Are we missing anything?</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>If this plan is forced, rolling migrate and upgrade should be planned carefully. It will be better to set up correct procedure checklist by practicing on a virtual environment with identical versions. </div>
<div> </div>
<blockquote class="x_x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Cheers</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Scott</div>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_x_x_gmail_signature"><font face="'courier new', monospace">Jongwoo Han</font>
<div><font face="'courier new', monospace">+82-505-227-6108</font></div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>