<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FR" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Hi François<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="font-size:10.0pt;font-family:"Arial",sans-serif">[root@server1 ~] rm: cannot remove ‘Logs:
<b>Cannot send after transport endpoint shutdown</b><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt;text-autospace:none"><span lang="EN-ZA" style="font-size:10.0pt;font-family:"Arial",sans-serif">[root@server1 ~] mv: cannot move /test/lustre/structure1 to ‘/test/lustre/structure2’:<b> Input/output error</b><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-ZA" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-ZA" style="mso-fareast-language:EN-US">These 2 error messages are typical from a client eviction issue. Your client was evicted (likely from the MDT) and you should expect ESHUTDOWN or EIO in that situation. Look at the kernel
logs (dmesg) from both client and server to try understand. The client usually reconnect quickly when this happens and following access should not return error.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-ZA" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-ZA" style="mso-fareast-language:EN-US">Aurélien<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b><span lang="EN-ZA" style="font-size:12.0pt;color:black">De :
</span></b><span lang="EN-ZA" style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> au nom de "Cloete, F. (Francois) via lustre-discuss" <lustre-discuss@lists.lustre.org><br>
<b>Répondre à : </b>"Cloete, F. (Francois)" <FrancoisClo@Nedbank.co.za><br>
<b>Date : </b>jeudi 20 octobre 2022 à 08:48<br>
<b>À : </b>"Spitz, Cory James" <cory.spitz@hpe.com>, Alastair Basden <a.g.basden@durham.ac.uk><br>
<b>Cc : </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Objet : </b>RE: [EXTERNAL][lustre-discuss] Lustre recycle bin<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA"><o:p> </o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="622" style="width:466.5pt;margin-left:35.4pt;border-collapse:collapse">
<tbody>
<tr style="height:15.25pt">
<td width="622" valign="top" style="width:466.5pt;border:solid #ED7D31 1.5pt;padding:0cm 5.4pt 0cm 5.4pt;height:15.25pt">
<p><strong><span style="font-family:"Calibri",sans-serif;color:black;background:#FFFF99">CAUTION</span></strong><span style="color:black;background:#FFFF99">: This email originated from outside of the organization. Do not click links or open attachments unless
you can confirm the sender and know the content is safe.</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US">Hi Cory,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US">They are both running the same versions on client and mgs server.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US">lfs 2.12.8_6_g5457c37<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US">Not sure if this could be related but our lustre environment started behaving starngly 2/3 weeks ago.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US">Also seeing the below when doing rsync of folders to new destinations.
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><i><span lang="EN-ZA" style="mso-fareast-language:EN-US">failed: Cannot send after transport endpoint shutdown (108)<o:p></o:p></span></i></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="font-size:10.0pt;font-family:"Arial",sans-serif">[root@server1 ~] rm: cannot remove ‘Logs:
<b>Cannot send after transport endpoint shutdown</b><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt;text-autospace:none"><span lang="EN-ZA" style="font-size:10.0pt;font-family:"Arial",sans-serif">[root@server1 ~] mv: cannot move /test/lustre/structure1 to ‘/test/lustre/structure2’:<b> Input/output error</b><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt;text-autospace:none"><span lang="EN-ZA" style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt;text-autospace:none"><span lang="EN-ZA" style="font-size:10.0pt;font-family:"Arial",sans-serif">[root@server1 ~] ~] ll<span style="color:#141414;background:gray;mso-highlight:gray"><o:p></o:p></span></span></p>
<p class="MsoNormal" style="margin-left:35.4pt;text-autospace:none"><span lang="EN-ZA" style="font-size:10.0pt;font-family:"Arial",sans-serif">[root@server1 ~] ls: cannot open directory .:
<b>Cannot send after transport endpoint shutdown<span style="color:#141414;background:gray;mso-highlight:gray"><o:p></o:p></span></b></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US">Regards<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Spitz, Cory James <cory.spitz@hpe.com>
<br>
<b>Sent:</b> Monday, 17 October 2022 20:56<br>
<b>To:</b> Cloete, F. (Francois) <FrancoisClo@Nedbank.co.za>; Alastair Basden <a.g.basden@durham.ac.uk><br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Lustre recycle bin<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-ZA"><o:p> </o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%">
<tbody>
<tr>
<td width="0%" style="width:.72%;background:#A6A6A6;padding:5.25pt 1.5pt 5.25pt 1.5pt">
</td>
<td width="97%" style="width:97.62%;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 11.25pt">
<div>
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<span style="font-size:9.0pt;font-family:"Segoe UI",sans-serif;color:#212121">You don't often get email from
<a href="mailto:cory.spitz@hpe.com">cory.spitz@hpe.com</a>. <a href="https://aka.ms/LearnAboutSenderIdentification">
Learn why this is important</a><o:p></o:p></span></p>
</div>
</td>
<td width="1%" style="width:1.66%;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 3.75pt;align:left">
</td>
</tr>
</tbody>
</table>
<div>
<div style="border-top:solid #F88017 1.0pt;border-left:none;border-bottom:solid #F88017 1.0pt;border-right:none;padding:1.0pt 0cm 2.0pt 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><strong><span lang="EN-US" style="font-size:8.5pt;font-family:"Arial",sans-serif;color:#F88017">CAUTION - EXTERNAL SENDER -</span></strong><strong><span lang="EN-US" style="font-family:"Arial",sans-serif;color:#F88017">
</span></strong><span lang="EN-US" style="font-size:9.0pt">Please be careful when opening links and attachments.
</span><span lang="EN-US" style="font-size:8.5pt">Nedbank - IT Information Security Department (ISD)</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">What version(s) are you using? Do you have an old client and a new-ish server?<br>
<br>
Very old client versions will disagree with the MDSes about how to clean up objects, resulting in orphans.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">-Cory<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:71.4pt"><span lang="EN-US">On 10/17/22, 3:44 AM, "lustre-discuss" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a>> wrote:<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:71.4pt"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:71.4pt"><span lang="EN-US">Thank-you! <br>
<br>
-----Original Message-----<br>
From: Alastair Basden <<a href="mailto:a.g.basden@durham.ac.uk">a.g.basden@durham.ac.uk</a>>
<br>
Sent: Monday, 17 October 2022 10:13<br>
To: Cloete, F. (Francois) <<a href="mailto:FrancoisClo@Nedbank.co.za">FrancoisClo@Nedbank.co.za</a>><br>
Cc: Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>>;
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
Subject: Re: [lustre-discuss] Lustre recycle bin<br>
<br>
[You don't often get email from <a href="mailto:a.g.basden@durham.ac.uk">a.g.basden@durham.ac.uk</a>. Learn why this is important at
<a href="https://aka.ms/LearnAboutSenderIdentification">https://aka.ms/LearnAboutSenderIdentification</a> ]<br>
<br>
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
CAUTION - EXTERNAL SENDER - Please be careful when opening links and attachments. Nedbank - IT Information Security Department (ISD)<br>
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
<br>
Hi Francois,<br>
<br>
We had something similar a few months back - I suspect a bug somewhere.<br>
<br>
Basically files weren't getting removed from the OST. Eventually, we mounted as ext, and removed them manually, I think.<br>
<br>
A reboot of the file system meant that rm operations then proceeded correctly after that.<br>
<br>
Cheers,<br>
Alastair.<br>
<br>
On Mon, 17 Oct 2022, Cloete, F. (Francois) via lustre-discuss wrote:<br>
<br>
> [EXTERNAL EMAIL]<br>
> Hi Andreas,<br>
> Our OSTs still display high file-system usage after removing folders.<br>
><br>
> Are there any commands that could be run to confirm if the allocated space which was used by those files have been released successfully ?<br>
><br>
> Thanks<br>
> Francois<br>
><br>
> From: Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>><br>
> Sent: Saturday, 15 October 2022 00:20<br>
> To: Cloete, F. (Francois) <<a href="mailto:FrancoisClo@Nedbank.co.za">FrancoisClo@Nedbank.co.za</a>><br>
> Cc: <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
> Subject: Re: [lustre-discuss] Lustre recycle bin<br>
><br>
> You don't often get email from <br>
> <a href="mailto:adilger@whamcloud.com%3cmailto:adilger@whamcloud.com">adilger@whamcloud.com<mailto:adilger@whamcloud.com</a>>. Learn why this is
<br>
> important<<a href="https://aka.ms/LearnAboutSenderIdentification%20">https://aka.ms/LearnAboutSenderIdentification
</a>><br>
> CAUTION - EXTERNAL SENDER - Please be careful when opening links and <br>
> attachments. Nedbank - IT Information Security Department (ISD) There isn't a recycle bin, but filenames are deleted from the filesystem quickly and the data objects are deleted in the background asynchronously (with transactions to prevent the space being
leaked). If there are a lot of files this may take some time, rebooting will not speed it up.<br>
><br>
><br>
> On Oct 14, 2022, at 10:00, Cloete, F. (Francois) via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org%3cmailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org</a>>> wrote:<br>
><br>
> Hi Community,<br>
> Is anyone aware of a recycle bin parameter for Lustre?<br>
><br>
> Just deleted a whole lot of files but for some reason the space is not getting cleared.<br>
><br>
> Server rebooted, file-system un-mounted etc.<br>
><br>
> Thanks<br>
><br>
> ********************<br>
> Nedbank Limited group of companies (Nedbank) disclaimer and confidentiality notice:<br>
> This email, including any attachments (email), contains information that is confidential and is meant only for intended recipients. You may not share or copy the email or any part of it, unless the sender has specifically allowed you to do so. If you are
not an intended recipient, please delete the email permanently and let Nedbank know that you have deleted it by replying to the sender or calling the Nedbank Contact Centre on 0860 555 111.<br>
> This email is not confirmation of a transaction or a Nedbank statement and is not offering or inviting anyone to take up any financial products or services, unless the content specifically indicates that it does so. Nedbank will not be liable for any errors
or omissions in this email. The views and opinions are those of the author and not necessarily those of Nedbank.<br>
> The names of the Nedbank Board of Directors and Company Secretary are available here:
<a href="http://www.nedbank.co.za/terms/DirectorsNedbank.htm">http://www.nedbank.co.za/terms/DirectorsNedbank.htm</a> <<a href="http://www.nedbank.co.za/terms/DirectorsNedbank.htm%20">http://www.nedbank.co.za/terms/DirectorsNedbank.htm
</a>>. Nedbank Ltd Reg No 1951/000009/06.<br>
> ********************<br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org%3cmailto:lustre-discuss@lists.lustre.org">
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org</a><br>
> ><br>
> <a href="https://eur01.safelinks.protection.outlook.com/?url=http://lists">https://eur01.safelinks.protection.outlook.com/?url=http://lists</a>
<br>
> .lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=05%7C0<br>
> 1%7CFrancoisClo%40Nedbank.co.za%7Caccd70e00cb84df3418608dab017807f%7C0<br>
> b1d23d810d140938cb7fd0bb32f81e1%7C0%7C0%7C638015912276367615%7CUnknown<br>
> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ<br>
> XVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=f4pQ6fAnpWfLFoihgUUGBNXH65%2B9Yc<br>
> Wgotw5E7UuUgU%3D&reserved=0<br>
><br>
> Cheers, Andreas<br>
> --<br>
> Andreas Dilger<br>
> Lustre Principal Architect<br>
> Whamcloud<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=05%7C01%7CFrancoisClo%40Nedbank.co.za%7Ccd127790e4714b6ffa0908dab0714a6d%7C0b1d23d810d140938cb7fd0bb32f81e1%7C0%7C0%7C638016297920336453%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rHCHfCCnDwn38lq8HE1QeF1a6yaLttJ%2BFnb1B08iEZ0%3D&reserved=0">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</body>
</html>