[lustre-discuss] Changing default recovery window time settings

[Spitz, Cory James]

The classical way to put a limit on recovery is to use the recovery_time_soft and recovery_time_hard mount options.

See the mount.lustre options: https://doc.lustre.org/lustre_manual.xhtml#idm139974521647280

recovery_time_soft=timeout
Allows timeout seconds for clients to reconnect for recovery after a server crash. This timeout is incrementally extended if it is about to expire and the server is still handling new connections from recoverable clients.
The default soft recovery timeout is 3 times the value of the Lustre timeout parameter (see Section 39.5.2, “Setting Static Timeouts”<https://doc.lustre.org/lustre_manual.xhtml#section_c24_nt5_dl>). The default Lustre timeout is 100 seconds, which would make the soft recovery timeout default to 300 seconds (5 minutes). The soft recovery timeout is set at mount time and will not change if the Lustre timeout is changed after mount time.
recovery_time_hard=timeout
The server is allowed to incrementally extend its timeout up to a hard maximum of timeout seconds.
The default hard recovery timeout is 9 times the value of the Lustre timeout parameter (see Section 39.5.2, “Setting Static Timeouts”<https://doc.lustre.org/lustre_manual.xhtml#section_c24_nt5_dl>). The default Lustre timeout is 100 seconds, which would make the hard recovery timeout default to 900 seconds (15 minutes). The hard recovery timeout is set at mount time and will not change if the Lustre timeout is changed after mount time.


> Alternatively, any way to manually end the recovery window would be appreciated
Yes, I think that the abort_recovery option to lctl is allows you to do this at runtime.  See: https://doc.lustre.org/lustre_manual.xhtml#idm139974522276784


abort_recovery

Aborts the recovery process on a re-starting MDT or OST.

-Cory


On 8/6/22, 9:23 PM, "lustre-discuss" <lustre-discuss-bounces at lists.lustre.org> wrote:

The maximum amount of time that recovery will run is controlled by "at_max".  The default is 600s (10 mins), but on my 2-client home cluster (with a relatively light load) the recovery is usually finished in 10s or less.

You can reduce the timeout based on what is your typical time. Note that the recovery should finish sooner even without changing at_max if all of the clients are available (as they should be).

Cheers, Andreas

> On Aug 4, 2022, at 22:37, Christian Kuntz <c.kuntz at opendrives.com> wrote:
>
> 
> Hello all,
>
> I'm wondering if there is any way to tune the maximum amount of time that lustre will use for a recovery window in the event that imperative recovery fails due to the failover of an MGS. On MGS failover, we appear to hit a default timeout of around 6 minutes that seems to be unavoidable. We're at a scale of less than 10 total nodes, so it seems that this timeout could safely be made much shorter.
>
> I understand that I'm approaching an unsafe/risky situation and asking for it to be made more unsafe, but we'd like to get start time in the event of a total cluster failure as fast as possible (within reason, of course). Alternatively, any way to manually end the recovery window would be appreciated.
>
> Cheers, and thanks for your attention,
> Christian Kuntz
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org>
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220809/e250b1b6/attachment.htm>