<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<br>
Marion,<br>
<br>
You note the deadlock reoccurs on server reboot, so you’re really stuck. This is most likely due to recovery where operations from the clients are replayed.<br>
<br>
If you’re fine with letting any pending I/O fail in order to get the system back up, I would suggest a client side action: unmount (-f, and be patient) and /or shut down all of your clients. That will discard things the clients are trying to replay, (causing
pending I/O to fail). Then shut down your servers and start them up again. With no clients, there’s (almost) nothing to replay, and you probably won’t hit the issue on startup. (There’s also the abort_recovery option covered in the manual, but I personally
think this is easier.)<br>
<br>
There’s no guarantee this avoids your deadlock happening again, but it’s highly likely it’ll at least get you running.<br>
<br>
If you need to save your pending I/O, you’ll have to install patched software with a fix for this (sounds like WC has identified the bug) and then reboot.<br>
<br>
Good luck!<br>
- Patrick<br>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Marion Hakanson <hakansom@ohsu.edu><br>
<b>Sent:</b> Friday, October 19, 2018 1:32:10 AM<br>
<b>To:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> [lustre-discuss] LU-11465 OSS/MDS deadlock in 2.10.5</font>
<div> </div>
</div>
<meta content="text/html; charset=utf-8">
<div dir="auto">This issue is really kicking our behinds:
<div dir="ltr"></div>
<div><a href="https://jira.whamcloud.com/browse/LU-11465">https://jira.whamcloud.com/browse/LU-11465</a></div>
<div><br>
</div>
<div>While we're waiting for the issue to get some attention from Lustre developers, are there suggestions on how we can recover our cluster from this kind of deadlocked, stuck-threads-on-the-MDS (or OSS) situation? Rebooting the storage servers does not clear
the hang-up, as upon reboot the MDS quickly ends up with the same number of D-state threads (around the same number as we have clients). It seems to me like there is some state stashed away in the filesystem which restores the deadlock as soon as the MDS
comes up.</div>
<div><br>
</div>
<div>Thanks and regards,</div>
<div><br>
</div>
<div>Marion</div>
<div><br>
</div>
</div>
</body>
</html>