<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Sep 12, 2019, at 04:20, Degremont, Aurelien <<a href="mailto:degremoa@amazon.com" class="">degremoa@amazon.com</a>> wrote:<br class="">
<div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">(Relocation this to lustre-devel)<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">The problem is related to the behavior when accessing files located on this OST.<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">If you replace this OST and some files are still referring it, you will have 2 different behaviors<o:p class=""></o:p></span></div>
<ul type="disc" style="margin-bottom: 0cm; margin-top: 0cm;" class="">
<li class="MsoListParagraph" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US" class="">If the object was not recreated, you will get some error accessing it (likely ENOENT)<o:p class=""></o:p></span></li><li class="MsoListParagraph" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<span lang="EN-US" class="">If the object was recreated, the file will be accessible without error BUT:<o:p class=""></o:p></span></li></ul>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">If the object is recreated and the stripe count is 1, file is empty, size is 0.<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">If stripe count was greater than 1, you will have different behavior with file size between 0 and the original size and file content with holes.<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">But, from the user point of view the file will look OK, without any error, even if the data were severely impacted.<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">That's why I'm not fond of this behavior were files with missing data looks nice from a user point of view.</span></div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
The reason that only recent objects are recreated is to handle files that were just created.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">If admins want to recreate the file objects because it is better for them, lfsck with create object option will take care of that. No need for the OST to do it automatically. </span><span style="font-size: 11pt;" class="">If admins
prefer to not recreate those file objects, that's not possible now.</span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">Looking at the code, the OST is recreating the last batch of object creation when starting after being replaced.<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">What do you think of not doing that in this case if the current object ID is 0.</span></div>
</div>
</blockquote>
<div><br class="">
</div>
</div>
<div>For files that were recently created, instead of creating all of the possible objects in that range, it would also be possible to have the OST create new objects for the case they are missing and the client tries to write to them. If no clients try to
write to those objects then they will not be recreated.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">Looking for OBD_FL_DELORPHAN flag and ofd_seq_last_oid() == 0 ?</span></div>
</div>
</blockquote>
<blockquote type="cite" class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">Do you see side effects?<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">Aurélien<o:p class=""></o:p></span></div>
<div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class=""><o:p class=""> </o:p></span></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;" class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 12pt;" class="">De :<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Andreas Dilger <<a href="mailto:adilger@whamcloud.com" style="color: purple; text-decoration: underline;" class="">adilger@whamcloud.com</a>><br class="">
<b class="">Date :<span class="Apple-converted-space"> </span></b>jeudi 12 septembre 2019 à 11:45<br class="">
<b class="">À :<span class="Apple-converted-space"> </span></b>"Degremont, Aurelien" <<a href="mailto:degremoa@amazon.com" style="color: purple; text-decoration: underline;" class="">degremoa@amazon.com</a>><br class="">
<b class="">Cc :<span class="Apple-converted-space"> </span></b>"<a href="mailto:lustre-discuss@lists.lustre.org" style="color: purple; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>" <<a href="mailto:lustre-discuss@lists.lustre.org" style="color: purple; text-decoration: underline;" class="">lustre-discuss@lists.lustre.org</a>><br class="">
<b class="">Objet :<span class="Apple-converted-space"> </span></b>Re: [lustre-discuss] Do not recreate OST objects on OST replacement<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<br class="">
<br class="">
<o:p class=""></o:p></div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On Sep 10, 2019, at 10:23, Degremont, Aurelien <<a href="mailto:degremoa@amazon.com" style="color: purple; text-decoration: underline;" class="">degremoa@amazon.com</a>> wrote:<o:p class=""></o:p></div>
</div>
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Hello<br class="">
<br class="">
When an OST dies and you have no choice but replacing it with a newly freshly formatted one (using mkfs --replace), Lustre runs a resynchronization mechanisms between the MDT and the OST.<br class="">
The MDT will sent the last object ID it knows for this OST and the OST will compare this value with its own counter (0 for a freshly formatted OST).<br class="">
If the difference is greater than 100,000 objects, it will recreate only the last 10,000, if not, it will recreate all the missing objects.<br class="">
<br class="">
I would like it to avoid recreating any objects. The missing ones are lost and just start recreating new ones. Is there a way to achieve that?<o:p class=""></o:p></div>
</div>
</div>
</blockquote>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
It isn't currently possible to completely avoid recreating these objects. Normally it isn't a huge problem, given the size of normal OSTs. This is done to ensure that if the MDS has previously allocated those objects there will be objects available for the
clients to write to them. LFSCK can be used to clean up these orphan objects if they are not in use.<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Cheers, Andreas<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">--<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Andreas Dilger<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Principal Lustre Architect<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm 0cm 0.0001pt 35.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span style="" class="">Whamcloud</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Principal Lustre Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>