<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-ZA" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Cory,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">They are both running the same versions on client and mgs server.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">lfs 2.12.8_6_g5457c37<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Not sure if this could be related but our lustre environment started behaving starngly 2/3 weeks ago.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Also seeing the below when doing rsync of folders to new destinations.
<o:p></o:p></span></p>
<p class="MsoNormal"><i><span style="mso-fareast-language:EN-US">failed: Cannot send after transport endpoint shutdown (108)<o:p></o:p></span></i></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">[root@server1 ~] rm: cannot remove ‘Logs:
<b>Cannot send after transport endpoint shutdown</b><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">[root@server1 ~] mv: cannot move /test/lustre/structure1 to ‘/test/lustre/structure2’:<b> Input/output error</b><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">[root@server1 ~] ~] ll<span style="color:#141414;background:gray;mso-highlight:gray"><o:p></o:p></span></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">[root@server1 ~] ls: cannot open directory .:
<b>Cannot send after transport endpoint shutdown<span style="color:#141414;background:gray;mso-highlight:gray"><o:p></o:p></span></b></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Regards<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Spitz, Cory James <cory.spitz@hpe.com>
<br>
<b>Sent:</b> Monday, 17 October 2022 20:56<br>
<b>To:</b> Cloete, F. (Francois) <FrancoisClo@Nedbank.co.za>; Alastair Basden <a.g.basden@durham.ac.uk><br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Lustre recycle bin<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%">
<tbody>
<tr>
<td style="background:#A6A6A6;padding:5.25pt 1.5pt 5.25pt 1.5pt"></td>
<td width="100%" style="width:100.0%;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 11.25pt">
<div>
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<span style="font-size:9.0pt;font-family:"Segoe UI",sans-serif;color:#212121">You don't often get email from
<a href="mailto:cory.spitz@hpe.com">cory.spitz@hpe.com</a>. <a href="https://aka.ms/LearnAboutSenderIdentification">
Learn why this is important</a><o:p></o:p></span></p>
</div>
</td>
<td width="75" style="width:56.25pt;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 3.75pt;align:left">
</td>
</tr>
</tbody>
</table>
<div>
<div style="border-top:solid #F88017 1.0pt;border-left:none;border-bottom:solid #F88017 1.0pt;border-right:none;padding:1.0pt 0cm 2.0pt 0cm">
<p class="MsoNormal"><strong><span lang="EN-US" style="font-size:8.5pt;font-family:"Arial",sans-serif;color:#F88017">CAUTION - EXTERNAL SENDER -</span></strong><strong><span lang="EN-US" style="font-family:"Arial",sans-serif;color:#F88017">
</span></strong><span lang="EN-US" style="font-size:9.0pt">Please be careful when opening links and attachments.
</span><span lang="EN-US" style="font-size:8.5pt">Nedbank - IT Information Security Department (ISD)</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">What version(s) are you using? Do you have an old client and a new-ish server?<br>
<br>
Very old client versions will disagree with the MDSes about how to clean up objects, resulting in orphans.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">-Cory<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US">On 10/17/22, 3:44 AM, "lustre-discuss" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a>> wrote:<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US">Thank-you! <br>
<br>
-----Original Message-----<br>
From: Alastair Basden <<a href="mailto:a.g.basden@durham.ac.uk">a.g.basden@durham.ac.uk</a>>
<br>
Sent: Monday, 17 October 2022 10:13<br>
To: Cloete, F. (Francois) <<a href="mailto:FrancoisClo@Nedbank.co.za">FrancoisClo@Nedbank.co.za</a>><br>
Cc: Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>>;
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
Subject: Re: [lustre-discuss] Lustre recycle bin<br>
<br>
[You don't often get email from <a href="mailto:a.g.basden@durham.ac.uk">a.g.basden@durham.ac.uk</a>. Learn why this is important at
<a href="https://aka.ms/LearnAboutSenderIdentification">https://aka.ms/LearnAboutSenderIdentification</a> ]<br>
<br>
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
CAUTION - EXTERNAL SENDER - Please be careful when opening links and attachments. Nedbank - IT Information Security Department (ISD)<br>
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
<br>
Hi Francois,<br>
<br>
We had something similar a few months back - I suspect a bug somewhere.<br>
<br>
Basically files weren't getting removed from the OST. Eventually, we mounted as ext, and removed them manually, I think.<br>
<br>
A reboot of the file system meant that rm operations then proceeded correctly after that.<br>
<br>
Cheers,<br>
Alastair.<br>
<br>
On Mon, 17 Oct 2022, Cloete, F. (Francois) via lustre-discuss wrote:<br>
<br>
> [EXTERNAL EMAIL]<br>
> Hi Andreas,<br>
> Our OSTs still display high file-system usage after removing folders.<br>
><br>
> Are there any commands that could be run to confirm if the allocated space which was used by those files have been released successfully ?<br>
><br>
> Thanks<br>
> Francois<br>
><br>
> From: Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>><br>
> Sent: Saturday, 15 October 2022 00:20<br>
> To: Cloete, F. (Francois) <<a href="mailto:FrancoisClo@Nedbank.co.za">FrancoisClo@Nedbank.co.za</a>><br>
> Cc: <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
> Subject: Re: [lustre-discuss] Lustre recycle bin<br>
><br>
> You don't often get email from <br>
> <a href="mailto:adilger@whamcloud.com%3cmailto:adilger@whamcloud.com">adilger@whamcloud.com<mailto:adilger@whamcloud.com</a>>. Learn why this is
<br>
> important<<a href="https://aka.ms/LearnAboutSenderIdentification%20">https://aka.ms/LearnAboutSenderIdentification
</a>><br>
> CAUTION - EXTERNAL SENDER - Please be careful when opening links and <br>
> attachments. Nedbank - IT Information Security Department (ISD) There isn't a recycle bin, but filenames are deleted from the filesystem quickly and the data objects are deleted in the background asynchronously (with transactions to prevent the space being
leaked). If there are a lot of files this may take some time, rebooting will not speed it up.<br>
><br>
><br>
> On Oct 14, 2022, at 10:00, Cloete, F. (Francois) via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org%3cmailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org</a>>> wrote:<br>
><br>
> Hi Community,<br>
> Is anyone aware of a recycle bin parameter for Lustre?<br>
><br>
> Just deleted a whole lot of files but for some reason the space is not getting cleared.<br>
><br>
> Server rebooted, file-system un-mounted etc.<br>
><br>
> Thanks<br>
><br>
> ********************<br>
> Nedbank Limited group of companies (Nedbank) disclaimer and confidentiality notice:<br>
> This email, including any attachments (email), contains information that is confidential and is meant only for intended recipients. You may not share or copy the email or any part of it, unless the sender has specifically allowed you to do so. If you are
not an intended recipient, please delete the email permanently and let Nedbank know that you have deleted it by replying to the sender or calling the Nedbank Contact Centre on 0860 555 111.<br>
> This email is not confirmation of a transaction or a Nedbank statement and is not offering or inviting anyone to take up any financial products or services, unless the content specifically indicates that it does so. Nedbank will not be liable for any errors
or omissions in this email. The views and opinions are those of the author and not necessarily those of Nedbank.<br>
> The names of the Nedbank Board of Directors and Company Secretary are available here:
<a href="http://www.nedbank.co.za/terms/DirectorsNedbank.htm">http://www.nedbank.co.za/terms/DirectorsNedbank.htm</a> <<a href="http://www.nedbank.co.za/terms/DirectorsNedbank.htm%20">http://www.nedbank.co.za/terms/DirectorsNedbank.htm
</a>>. Nedbank Ltd Reg No 1951/000009/06.<br>
> ********************<br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org%3cmailto:lustre-discuss@lists.lustre.org">
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org</a><br>
> ><br>
> <a href="https://eur01.safelinks.protection.outlook.com/?url=http://lists">https://eur01.safelinks.protection.outlook.com/?url=http://lists</a>
<br>
> .lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=05%7C0<br>
> 1%7CFrancoisClo%40Nedbank.co.za%7Caccd70e00cb84df3418608dab017807f%7C0<br>
> b1d23d810d140938cb7fd0bb32f81e1%7C0%7C0%7C638015912276367615%7CUnknown<br>
> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ<br>
> XVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=f4pQ6fAnpWfLFoihgUUGBNXH65%2B9Yc<br>
> Wgotw5E7UuUgU%3D&reserved=0<br>
><br>
> Cheers, Andreas<br>
> --<br>
> Andreas Dilger<br>
> Lustre Principal Architect<br>
> Whamcloud<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=05%7C01%7CFrancoisClo%40Nedbank.co.za%7Ccd127790e4714b6ffa0908dab0714a6d%7C0b1d23d810d140938cb7fd0bb32f81e1%7C0%7C0%7C638016297920336453%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rHCHfCCnDwn38lq8HE1QeF1a6yaLttJ%2BFnb1B08iEZ0%3D&reserved=0">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
<o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>