[lustre-discuss] Lustre recycle bin

Cloete, F. (Francois) FrancoisClo at Nedbank.co.za
Wed Oct 19 23:47:25 PDT 2022


Hi Cory,
They are both running the same versions on client and mgs server.

lfs 2.12.8_6_g5457c37

Not sure if this could be related but our lustre environment started behaving starngly 2/3 weeks ago.

Also seeing the below when doing rsync of folders to new destinations.
failed: Cannot send after transport endpoint shutdown (108)


[root at server1 ~] rm: cannot remove 'Logs: Cannot send after transport endpoint shutdown
[root at server1 ~] mv: cannot move /test/lustre/structure1 to '/test/lustre/structure2': Input/output error

[root at server1 ~] ~] ll
[root at server1 ~] ls: cannot open directory .: Cannot send after transport endpoint shutdown

Regards

From: Spitz, Cory James <cory.spitz at hpe.com>
Sent: Monday, 17 October 2022 20:56
To: Cloete, F. (Francois) <FrancoisClo at Nedbank.co.za>; Alastair Basden <a.g.basden at durham.ac.uk>
Cc: lustre-discuss at lists.lustre.org
Subject: Re: [lustre-discuss] Lustre recycle bin

You don't often get email from cory.spitz at hpe.com<mailto:cory.spitz at hpe.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
CAUTION - EXTERNAL SENDER - Please be careful when opening links and attachments. Nedbank - IT Information Security Department (ISD)
What version(s) are you using?  Do you have an old client and a new-ish server?

Very old client versions will disagree with the MDSes about how to clean up objects, resulting in orphans.

-Cory

On 10/17/22, 3:44 AM, "lustre-discuss" <lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces at lists.lustre.org>> wrote:

Thank-you!

-----Original Message-----
From: Alastair Basden <a.g.basden at durham.ac.uk<mailto:a.g.basden at durham.ac.uk>>
Sent: Monday, 17 October 2022 10:13
To: Cloete, F. (Francois) <FrancoisClo at Nedbank.co.za<mailto:FrancoisClo at Nedbank.co.za>>
Cc: Andreas Dilger <adilger at whamcloud.com<mailto:adilger at whamcloud.com>>; lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] Lustre recycle bin

[You don't often get email from a.g.basden at durham.ac.uk<mailto:a.g.basden at durham.ac.uk>. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification   ]

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CAUTION - EXTERNAL SENDER -  Please be careful when opening links and attachments. Nedbank - IT Information Security Department (ISD)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Hi Francois,

We had something similar a few months back - I suspect a bug somewhere.

Basically files weren't getting removed from the OST.  Eventually, we mounted as ext, and removed them manually, I think.

A reboot of the file system meant that rm operations then proceeded correctly after that.

Cheers,
Alastair.

On Mon, 17 Oct 2022, Cloete, F. (Francois) via lustre-discuss wrote:

> [EXTERNAL EMAIL]
> Hi Andreas,
> Our OSTs still display high file-system usage after removing folders.
>
> Are there any commands that could be run to confirm if the allocated space which was used by those files have been released successfully ?
>
> Thanks
> Francois
>
> From: Andreas Dilger <adilger at whamcloud.com<mailto:adilger at whamcloud.com>>
> Sent: Saturday, 15 October 2022 00:20
> To: Cloete, F. (Francois) <FrancoisClo at Nedbank.co.za<mailto:FrancoisClo at Nedbank.co.za>>
> Cc: lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
> Subject: Re: [lustre-discuss] Lustre recycle bin
>
> You don't often get email from
> adilger at whamcloud.com<mailto:adilger at whamcloud.com<mailto:adilger at whamcloud.com%3cmailto:adilger at whamcloud.com>>. Learn why this is
> important<https://aka.ms/LearnAboutSenderIdentification <https://aka.ms/LearnAboutSenderIdentification%20> >
> CAUTION - EXTERNAL SENDER - Please be careful when opening links and
> attachments. Nedbank - IT Information Security Department (ISD) There isn't a recycle bin, but filenames are deleted from the filesystem quickly and the data objects are deleted in the background asynchronously (with transactions to prevent the space being leaked).  If there are a lot of files this may take some time, rebooting will not speed it up.
>
>
> On Oct 14, 2022, at 10:00, Cloete, F. (Francois) via lustre-discuss <lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org%3cmailto:lustre-discuss at lists.lustre.org>>> wrote:
>
> Hi Community,
> Is anyone aware of a recycle bin parameter for Lustre?
>
> Just deleted a whole lot of files but for some reason the space is not getting cleared.
>
> Server rebooted, file-system un-mounted etc.
>
> Thanks
>
> ********************
> Nedbank Limited group of companies (Nedbank) disclaimer and confidentiality notice:
> This email, including any attachments (email), contains information that is confidential and is meant only for intended recipients. You may not share or copy the email or any part of it, unless the sender has specifically allowed you to do so. If you are not an intended recipient, please delete the email permanently and let Nedbank know that you have deleted it by replying to the sender or calling the Nedbank Contact Centre on 0860 555 111.
> This email is not confirmation of a transaction or a Nedbank statement and is not offering or inviting anyone to take up any financial products or services, unless the content specifically indicates that it does so. Nedbank will not be liable for any errors or omissions in this email. The views and opinions are those of the author and not necessarily those of Nedbank.
> The names of the Nedbank Board of Directors and Company Secretary are available here: http://www.nedbank.co.za/terms/DirectorsNedbank.htm  <http://www.nedbank.co.za/terms/DirectorsNedbank.htm <http://www.nedbank.co.za/terms/DirectorsNedbank.htm%20> >. Nedbank Ltd Reg No 1951/000009/06.
> ********************
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org%3cmailto:lustre-discuss at lists.lustre.org>
> >
> https://eur01.safelinks.protection.outlook.com/?url=http://lists
> .lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=05%7C0
> 1%7CFrancoisClo%40Nedbank.co.za%7Caccd70e00cb84df3418608dab017807f%7C0
> b1d23d810d140938cb7fd0bb32f81e1%7C0%7C0%7C638015912276367615%7CUnknown
> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ
> XVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=f4pQ6fAnpWfLFoihgUUGBNXH65%2B9Yc
> Wgotw5E7UuUgU%3D&reserved=0
>
> Cheers, Andreas
> --
> Andreas Dilger
> Lustre Principal Architect
> Whamcloud
>
>
>
>
>
>
>
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=05%7C01%7CFrancoisClo%40Nedbank.co.za%7Ccd127790e4714b6ffa0908dab0714a6d%7C0b1d23d810d140938cb7fd0bb32f81e1%7C0%7C0%7C638016297920336453%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rHCHfCCnDwn38lq8HE1QeF1a6yaLttJ%2BFnb1B08iEZ0%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20221020/ac24556d/attachment-0001.htm>


More information about the lustre-discuss mailing list