<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
p.msipfooter1a2dc593, li.msipfooter1a2dc593, div.msipfooter1a2dc593
{mso-style-name:msipfooter1a2dc593;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">This is fantastic output, you guessed correctly. We use an md raid on ldiskfs. We're getting ext4 io errors on boot and were able to reboot enough to get into a state where we can read the mount, so we are moving data off that mount onto
another and rebuilding(which is the intended design of this). We were sometimes able to restore the mount easily, and sometimes we would have to run fsck.ext4 in order to repair or move bad blocks. This back and forth landed us in the ability to move and rebuild
as we are doing now.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I will definitely be able to use your input in testing further.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<br>
JC<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="msipfooter1a2dc593" style="margin:0in"><span style="font-size:10.0pt;color:black">Internal</span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org>
<b>On Behalf Of </b>Cameron Harr via lustre-discuss<br>
<b>Sent:</b> Tuesday, August 9, 2022 5:41 PM<br>
<b>To:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Lustre 2.12.6 on RHEL 7.9 not able to mount disks after reboot<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div align="center">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="933" style="width:700.0pt;background:maroon">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p align="center" style="text-align:center"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:white">CAUTION: EXTERNAL MAIL. DO NOT CLICK ON LINKS OR OPEN ATTACHMENTS YOU DO NOT TRUST
<br>
ATTENTION : COURRIEL EXTERNE. NE CLIQUEZ PAS SUR DES LIENS ET N'OUVREZ PAS DE PIÈCES JOINTES AUXQUELS VOUS NE FAITES PAS CONFIANCE</span><span style="font-size:9.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>JC,<o:p></o:p></p>
<p>The message where it asks if the MGS is running is a pretty common error that you'll see when something isn't right. There's not a lot of detail in your message but first step is to make sure your OST device is present on the OSS server. You mentioned remounting
the RAID directories; is this software/MD RAID? Are you using ldiskfs or ZFS for the backend storage (I'll guess ldiskfs if using MD RAID).
<o:p></o:p></p>
<p>If you've already verified the OST volume is present, see if you can 'lctl ping' between the MDS and OSS nodes. I'm not sure what your knowledge base is so forgive me if this is too elementary, but on each node, type 'lctl list_nids' to get the Lustre node
identifier, then run 'lctl ping <NID>' to make sure you can talk Lustre/LNet between them:<o:p></o:p></p>
<pre>[root@tin1:~]# lctl list_nids<o:p></o:p></pre>
<pre><a href="mailto:192.168.101.1@o2ib1">192.168.101.1@o2ib1</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>[root@tin6:~]# lctl list_nids<o:p></o:p></pre>
<pre><a href="mailto:192.168.101.6@o2ib1">192.168.101.6@o2ib1</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>[root@tin6:~]# lctl ping <a href="mailto:192.168.101.1@o2ib1">192.168.101.1@o2ib1</a><o:p></o:p></pre>
<pre>12345-0@lo<o:p></o:p></pre>
<pre><a href="mailto:12345-192.168.101.1@o2ib1">12345-192.168.101.1@o2ib1</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>[root@tin1:~]# lctl ping <a href="mailto:192.168.101.6@o2ib1">192.168.101.6@o2ib1</a><o:p></o:p></pre>
<pre>12345-0@lo<o:p></o:p></pre>
<pre><a href="mailto:12345-192.168.101.6@o2ib1">12345-192.168.101.6@o2ib1</a><o:p></o:p></pre>
<p>If you get a failure (like I/O Error), then you have a communications problem and you'll want to make sure all the correct interfaces are up. If the pings do work, then you'll want to look for messages in /var/log/lustre and dmesg.<o:p></o:p></p>
<p>Cameron<o:p></o:p></p>
<div>
<p class="MsoNormal">On 8/9/22 06:45, Crowder, Jonathan via lustre-discuss wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hello, this is my first post here so I may need some guidance on the function of this system.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I am in a small team supporting some 36TB lustre servers for a business unit. Our configuration per mount point is one lustre master node and 3 lustre object stores. We had one of the object stores lost to an unidentified reboot and upon
getting it booted back into the lustre kernel by azure cloud teams, we saw behavior where we could not get it to remount the raid directories for storage to the local file paths we have set up for them. I can obtain the output soon here, it knows the MGS node,
but asks if it's running. I am having difficulty investigating more deeply into why this is happening as the other object stores are working without issue.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">JC<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="msipfooter1a2dc593" style="margin:0in"><span style="font-size:10.0pt;color:black">Internal</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>lustre-discuss mailing list<o:p></o:p></pre>
<pre><a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><o:p></o:p></pre>
<pre><a href="https://clicktime.symantec.com/15tStaNZJe7fN1Rd2QBxh?h=Sv2alGuGQqv2LT-fhBdOjFsaWjrYLSvMk3aOegXZzfI=&u=https://urldefense.us/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__%3B%21%21G2kpM7uM-TzIFchu%21l2FbiTAR6qhLwbOqf4kfzj8IRp8tfTexTXEOpPVB2ASGCAIVUTpJGN5isgF9Ugs%24">https://urldefense.us/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!G2kpM7uM-TzIFchu!l2FbiTAR6qhLwbOqf4kfzj8IRp8tfTexTXEOpPVB2ASGCAIVUTpJGN5isgF9Ugs$</a> <o:p></o:p></pre>
</blockquote>
</div>
</body>
</html>