<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:2056735727;
mso-list-type:hybrid;
mso-list-template-ids:-1933256766 1752702026 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi Einar,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As for the OST in bad shape, if you have not cleared the bad blocks on the storage system you’ll keep having IO errors when your server tries to access these blocks, that’s kind of a protection mechanism and lots of IO errors might give
you many issues. The procedure to clean them up is a bit of storage and filesystem surgery. I would suggest, this high level view plan:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">Obtain the bad blocks from the storage system (offset, size, etc…)<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">Map them to filesystem blocks: watch out, the storage system speaks probably and for old systems about 512bytes blocks and the filesystem blocks are 4KB, so you need to map storage
blocks to filesystem blocks<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">Clear the bad blocks on the storage system, each storage system has their own commands to clear those. You’ll probably no longer have IO errors accessing these sectors after clearing
the bad blocks<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">Optional, zero the bad storage blocks with dd (and just these bad blocks of course) to ignore the “trash” there might be on these blocks<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">Find out with debugfs which files are affected<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">Run e2fsck on the device<o:p></o:p></li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As I said, surgery, so if you really care about what you have on that device try to do a block level backup before… But the minimum for sure is that you need to clear the bad blocks, otherwise you get IO access error on the device.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">Diego</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Einar Næss Jensen <einar.nass.jensen@ntnu.no><br>
<b>Date: </b>Friday, 15 November 2019 at 10:01<br>
<b>To: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[lustre-discuss] one ost down<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div id="divtagdefaultwrapper">
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">Hello dear lustre community.<o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">We have a lustre file system, where one ost is having problems.<o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">The underlying diskarray, an old sfa10k from DDN (without support), have one raidset with ca 1300 bad blocks. The bad blocks came about when one disk in the raid failed while another drive
in other raidset was rebuilding.<o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">Now.<o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">The ost is offline, and the file system seems useable for new files, while old files on the corresponding ost is generating lots of kernel messages on the OSS.<o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">Quotainformation is not available though.<o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">Questions: <o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">May I assume that for new files, everything is fine, since they are not using the inactive device anyway?<o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">I tried to run e2fschk on the ost unmounted, while jobs were still running on the filesystem, and for a few minutes it seemd like this was working, as the filesystem seemed to come back
complete afterwards. After a few minutes the ost failed again, though.<o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">Any pointers on how to rebuild/fix the ost and get it back is very much appreciated. <o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">Also how to regenerate the quotainformation, which is currently unavailable would help. With or without the troublesome OST.<o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">Best Regards<o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black">Einar Næss Jensen (on flight to Denver)<o:p></o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p style="margin-left:36.0pt"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<div id="Signature">
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;color:black">--
<br>
Einar Næss Jensen<br>
NTNU HPC Section<br>
Norwegian University of Science and Technoloy<br>
Address: Høgskoleringen 7i<br>
N-7491 Trondheim, NORWAY<br>
tlf: +47 90990249<br>
email: einar.nass.jensen@ntnu.no<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>