<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:PMingLiU;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"\@PMingLiU";
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:"Baskerville Old Face";
panose-1:2 2 6 2 8 5 5 2 3 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:Consolas;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Not sure what happened to my post on Friday, but it seems to got tagged under another one.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Anyway, here is a resend – my apologies if you have read it already.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I shall be grateful for any advice.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#17365D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#17365D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#17365D">Peter Chiu<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><span style="font-family:"Baskerville Old Face","serif";color:#002060">STFC Rutherford Appleton Laboratory<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-family:"Baskerville Old Face","serif";color:#002060">RAL Space Department<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-family:"Baskerville Old Face","serif";color:#002060">Building R25, Room 2.02<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-family:"Baskerville Old Face","serif";color:#002060">Harwell Oxford<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-family:"Baskerville Old Face","serif";color:#002060">Didcot<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-family:"Baskerville Old Face","serif";color:#002060">OXON<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-family:"Baskerville Old Face","serif";color:#002060">OX11 0QX<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-family:"Baskerville Old Face","serif";color:#002060">UK<o:p></o:p></span></b></p>
<div style="mso-element:para-border-div;border:none;border-bottom:double windowtext 2.25pt;padding:0cm 0cm 1.0pt 0cm">
<p class="MsoNormal" style="border:none;padding:0cm"><b><span style="font-family:"Baskerville Old Face","serif";color:#002060"><o:p> </o:p></span></b></p>
</div>
<p class="MsoPlainText">Dear all,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">We have an odd problem on a lustre client in that a couple of OST elements have become "inactive".<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">However, lctl dl shows all devices are in "UP" state.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">The "UP" status is also echoed on the mds as well as on another client - please see below.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I have tried rebooting this client, but with no joy.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I have also searched around the discussion list, and can find a few others had experienced similar problems before.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">But unfortunatley none of them has reported back with a good recovery procedure, other than a complete reformatting of the lustre storage.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">So can I trouble you experts to give me some clue as to how to recover from this?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Many thanks.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Regards,<o:p></o:p></p>
<p class="MsoPlainText">Peter<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">PS. Client: SLES 11 with 2.6.32.29 kernel, Lustre 1.8.5<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">===================================================================================<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Client Capuchin:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">capuchin:~ # lfs df -h<o:p></o:p></p>
<p class="MsoPlainText">UUID bytes Used Available Use% Mounted on<o:p></o:p></p>
<p class="MsoPlainText">ceda3-MDT0000_UUID 350.0G 689.2M 329.3G 0% /disks/ceda3[MDT:0]<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0000_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:0]<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0001_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:1]<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0002_UUID : inactive device<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0003_UUID : inactive device<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0004_UUID 7.2T 6.8T 6.6G 94% /disks/ceda3[OST:4]<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0005_UUID 7.2T 6.8T 6.7G 94% /disks/ceda3[OST:5]<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0006_UUID 7.2T 6.8T 6.2G 94% /disks/ceda3[OST:6]<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">filesystem summary: 35.8T 34.0T 32.5G 94% /disks/ceda3<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">capuchin:~ # lctl dl<o:p></o:p></p>
<p class="MsoPlainText"> 0 UP mgc <a href="mailto:MGC130.246.191.64@tcp">MGC130.246.191.64@tcp</a> be6c0da2-275f-c59d-fa3a-c792134a0d9d 5<o:p></o:p></p>
<p class="MsoPlainText"> 1 UP lov ceda3-clilov-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 4<o:p></o:p></p>
<p class="MsoPlainText"> 2 UP mdc ceda3-MDT0000-mdc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5<o:p></o:p></p>
<p class="MsoPlainText"> 3 UP osc ceda3-OST0000-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5<o:p></o:p></p>
<p class="MsoPlainText"> 4 UP osc ceda3-OST0001-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5<o:p></o:p></p>
<p class="MsoPlainText"> 5 UP osc ceda3-OST0002-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5<o:p></o:p></p>
<p class="MsoPlainText"> 6 UP osc ceda3-OST0003-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5<o:p></o:p></p>
<p class="MsoPlainText"> 7 UP osc ceda3-OST0004-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5<o:p></o:p></p>
<p class="MsoPlainText"> 8 UP osc ceda3-OST0005-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5<o:p></o:p></p>
<p class="MsoPlainText"> 9 UP osc ceda3-OST0006-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 capuchin:~ #<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Client2 cmip-dn1:<o:p></o:p></p>
<p class="MsoPlainText">[root@aspre script]# ssh cmip-dn1 lfs df -h<o:p></o:p></p>
<p class="MsoPlainText">UUID bytes Used Available Use% Mounted on<o:p></o:p></p>
<p class="MsoPlainText">ceda3-MDT0000_UUID 350.0G 689.2M 329.3G 0% /disks/ceda3[MDT:0]<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0000_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:0]<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0001_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:1]<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0002_UUID 7.2T 451.6M 6.8T 0% /disks/ceda3[OST:2]<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0003_UUID 7.2T 451.8M 6.8T 0% /disks/ceda3[OST:3]<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0004_UUID 7.2T 6.8T 6.6G 94% /disks/ceda3[OST:4]<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0005_UUID 7.2T 6.8T 6.7G 94% /disks/ceda3[OST:5]<o:p></o:p></p>
<p class="MsoPlainText">ceda3-OST0006_UUID 7.2T 6.8T 6.2G 94% /disks/ceda3[OST:6]<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">filesystem summary: 50.1T 34.0T 13.6T 67% /disks/ceda3<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Mds:<o:p></o:p></p>
<p class="MsoPlainText">[root@mds02 ~]# lctl dl<o:p></o:p></p>
<p class="MsoPlainText"> 0 UP mgc <a href="mailto:MGC130.246.191.64@tcp">MGC130.246.191.64@tcp</a> 8aa29420-11f1-8ca3-a361-ce5135a09be2 5<o:p></o:p></p>
<p class="MsoPlainText"> 1 UP mdt MDS MDS_uuid 3<o:p></o:p></p>
<p class="MsoPlainText"> 2 UP lov ceda3-mdtlov ceda3-mdtlov_UUID 4<o:p></o:p></p>
<p class="MsoPlainText"> 3 UP mds ceda3-MDT0000 ceda3-MDT0000_UUID 9<o:p></o:p></p>
<p class="MsoPlainText"> 4 UP osc ceda3-OST0000-osc ceda3-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText"> 5 UP osc ceda3-OST0001-osc ceda3-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText"> 6 UP osc ceda3-OST0002-osc ceda3-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText"> 7 UP osc ceda3-OST0003-osc ceda3-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText"> 8 UP osc ceda3-OST0004-osc ceda3-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText"> 9 UP osc ceda3-OST0005-osc ceda3-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText">10 UP osc ceda3-OST0006-osc ceda3-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText">11 UP lov ceda4-mdtlov ceda4-mdtlov_UUID 4<o:p></o:p></p>
<p class="MsoPlainText">12 UP mds ceda4-MDT0000 ceda4-MDT0000_UUID 9<o:p></o:p></p>
<p class="MsoPlainText">13 UP osc ceda4-OST0000-osc ceda4-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText">14 UP osc ceda4-OST0001-osc ceda4-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText">15 UP osc ceda4-OST0002-osc ceda4-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText">16 UP osc ceda4-OST0003-osc ceda4-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText">17 UP osc ceda4-OST0004-osc ceda4-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText">18 UP osc ceda4-OST0005-osc ceda4-mdtlov_UUID 5<o:p></o:p></p>
<p class="MsoPlainText">[root@mds02 ~]#<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br><p>--
<BR>Scanned by iCritical.
</p>
<br></body>
</html>