<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">The classical way to put a limit on recovery is to use the recovery_time_soft and recovery_time_hard mount options.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">See the mount.lustre options: <a href="https://doc.lustre.org/lustre_manual.xhtml#idm139974521647280">https://doc.lustre.org/lustre_manual.xhtml#idm139974521647280</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="1" cellpadding="0">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">recovery_time_soft=<i>timeout</i><o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">Allows timeout seconds for clients to reconnect for recovery after a server crash. This timeout is incrementally extended if it is about to expire and the server is still handling new connections from recoverable clients.<o:p></o:p></p>
<p class="MsoNormal">The default soft recovery timeout is 3 times the value of the Lustre timeout parameter (see <a href="https://doc.lustre.org/lustre_manual.xhtml#section_c24_nt5_dl" title="39.5.2. Setting Static Timeouts">Section 39.5.2, “Setting Static Timeouts”</a>). The default Lustre timeout is 100 seconds, which would make the soft recovery timeout default to 300 seconds (5 minutes). The soft recovery timeout is set at mount time and will not change if the Lustre timeout is changed after mount time.<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">recovery_time_hard=<i>timeout</i><o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">The server is allowed to incrementally extend its timeout up to a hard maximum of <i>timeout</i> seconds.<o:p></o:p></p>
<p class="MsoNormal">The default hard recovery timeout is 9 times the value of the Lustre timeout parameter (see <a href="https://doc.lustre.org/lustre_manual.xhtml#section_c24_nt5_dl" title="39.5.2. Setting Static Timeouts">Section 39.5.2, “Setting Static Timeouts”</a>). The default Lustre timeout is 100 seconds, which would make the hard recovery timeout default to 900 seconds (15 minutes). The hard recovery timeout is set at mount time and will not change if the Lustre timeout is changed after mount time.<o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">> Alternatively, any way to manually end the recovery window would be appreciated<o:p></o:p></p>
<p class="MsoNormal">Yes, I think that the abort_recovery option to lctl is allows you to do this at runtime. See:
<a href="https://doc.lustre.org/lustre_manual.xhtml#idm139974522276784">https://doc.lustre.org/lustre_manual.xhtml#idm139974522276784</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="1" cellpadding="0">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p><code><span style="font-size:10.0pt">abort_recovery</span></code><o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p>Aborts the recovery process on a re-starting MDT or OST.<o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Cory<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 8/6/22, 9:23 PM, "lustre-discuss" <lustre-discuss-bounces@lists.lustre.org> wrote:</p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">The maximum amount of time that recovery will run is controlled by "at_max". The default is 600s (10 mins), but on my 2-client home cluster (with a relatively light load) the recovery is usually finished in 10s
or less. <br>
<br>
You can reduce the timeout based on what is your typical time. Note that the recovery should finish sooner even without changing at_max if all of the clients are available (as they should be).
<br>
<br>
Cheers, Andreas<br>
<br>
> On Aug 4, 2022, at 22:37, Christian Kuntz <c.kuntz@opendrives.com> wrote:<br>
> <br>
> <br>
> Hello all,<br>
> <br>
> I'm wondering if there is any way to tune the maximum amount of time that lustre will use for a recovery window in the event that imperative recovery fails due to the failover of an MGS. On MGS failover, we appear to hit a default timeout of around 6 minutes
that seems to be unavoidable. We're at a scale of less than 10 total nodes, so it seems that this timeout could safely be made much shorter.<br>
> <br>
> I understand that I'm approaching an unsafe/risky situation and asking for it to be made more unsafe, but we'd like to get start time in the event of a total cluster failure as fast as possible (within reason, of course). Alternatively, any way to manually
end the recovery window would be appreciated.<br>
> <br>
> Cheers, and thanks for your attention,<br>
> Christian Kuntz<br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> lustre-discuss@lists.lustre.org<br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</p>
</div>
</div>
</body>
</html>