<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:MingLiU;
panose-1:2 2 5 9 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Thanks. This is happening on all the clients so its not a DLM lock problem.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">We’ll try the fsck soon. If that come back clean is this likely a hardware corruption?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Could this have anything to do with our corruption?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Aug 29 12:13:34 hpfs2-eg3-mds0 kernel: LustreError: 0-0: hpfs2eg3-MDT0000: trigger OI scrub by RPC for [0x2000079cd:0x3988:0x0], rc = 0 [1]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">We make a file level backup of the MDT using the procedure in the lustre manual on a daily basis and keep a history of those. We’ve never had a problem so we’ve never had to restore anything
from the backups. Is the device level backup (dd) necessary or is file level sufficient?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">"Dilger, Andreas" <andreas.dilger@intel.com><br>
<b>Date: </b>Thursday, September 1, 2016 at 11:48 AM<br>
<b>To: </b>Darby Vicker <darby.vicker-1@nasa.gov><br>
<b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] Inaccessible directory<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<div>
<div>
<p class="MsoNormal">The first thing to try is just cancel the DLM locks on this client, to see if the problem is just a stale lock cached there:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> lctl set_param ldlm.namespaces.*.lru_size=clear<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">That likely won't help if this same problem is being hit on all of your clients. <br>
<br>
Next to try is running e2fsck on the MDT. It is recommended to use the most recent e2fsprogs-1.42.12-wc5, since this fixes a number of bugs in older versions. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">It is also strongly recommended to make a "dd" backup of the whole MDT before the e2fsck run, and to do this on a regular basis. This is convenient in case of MDT corruption (now or in the future), and you can restore only the MDT instead
of having to restore 400TB of data as well. I was just updating the Lustre User Manual about this <a href="http://review.whamcloud.com/21726">http://review.whamcloud.com/21726</a> .<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Running "e2fsck -fn" first can also give you an idea of what kind of corruption is present before Making the fix. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Cheers, Andreas<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
On Aug 31, 2016, at 10:54, Vicker, Darby (JSC-EG311) <<a href="mailto:darby.vicker-1@nasa.gov">darby.vicker-1@nasa.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Hello,<br>
<br>
We’ve run into a problem where an entire directory on our lustre file system has become inaccessible. <span style="font-family:MingLiU"><br>
</span><br>
# mount | grep lustre2<br>
192.52.98.142@tcp:/hpfs2eg3 on /lustre2 type lustre (rw,flock)<br>
# ls -l /lustre2/mirrors/cpas<br>
ls: cannot access /lustre2/mirrors/cpas: Stale file handle<br>
# ls -l /lustre2/mirrors/<br>
ls: cannot access /lustre2/mirrors/cpas: Stale file handle<br>
4 drwxr-s--- 5 root g27122 4096 Dec 23 2014 cev-repo/<br>
? d????????? ? ? ? ? ? cpas/<br>
4 drwxrwxr-x 5 root eg3 4096 Aug 21 2014 sls/<br>
<br>
<br>
<br>
Fortunately, we have a backup of this directory from about a week ago. However, I would like to figure out how this happened to prevent any further damage. I’m not sure if we’re dealing with corruption in the LFS, damage to the underlying RAID or something
else and I’d appreciate some help figuring this out. Here’s some info on our lustre servers:<br>
<br>
CentOS 6.4 2.6.32-358.23.2.el6_lustre.x86_64<br>
Lustre 2.4.3 (I know - we need to upgrade...)<br>
Hardware RAID 10 MDT (Adaptec 6805Q – SSD’s)<span style="font-family:MingLiU"><br>
</span>(19x) OSS’s - Hardware RAID 6 OST’s (Adaptec 6445)<span style="font-family:MingLiU"><br>
</span>1 27TB OST per OSS, ldiskfs<span style="font-family:MingLiU"><br>
</span>Dual homed via Ethernet and IB<br>
<br>
Most Ethernet clients (~50 total) are CentOS 7 using lustre-client-2.8.0-3.10.0_327.13.1.el7.x86_64.x86_64. Our compute nodes (~400 total) connect over IB and are still CentOS 6 using lustre-client-2.7.0-2.6.32_358.14.1.el6.x86_64.x86_64.<br>
<br>
<br>
The lustre server hardware is about 4 years old now. All RAID arrays are reported as healthy. Searched JIRA and the mailing lists and couldn’t find anything related. This sounded close at first:<span style="font-family:MingLiU"><br>
<br>
</span><a href="https://jira.hpdd.intel.com/browse/LU-3550">https://jira.hpdd.intel.com/browse/LU-3550</a><br>
<br>
But, as shown above, the issue shows up on a native lustre client, not an NFS export. We are exporting this LFS via SMB but I don’t think that’s related. <span style="font-family:MingLiU"><br>
<br>
</span>I think the next step is to run an e2fsck but we haven’t done that yet and would appreciate advise on stepping through this. <span style="font-family:MingLiU"><br>
<br>
</span>Thanks,<span style="font-family:MingLiU"><br>
</span>Darby<span style="font-family:MingLiU"><br>
<br>
<br>
<br>
<br>
<br>
</span>_______________________________________________<span style="font-family:MingLiU"><br>
</span>lustre-discuss mailing list<span style="font-family:MingLiU"><br>
</span><a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</body>
</html>