<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Joe,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">So I gave that a try. I disabled ib0 so the node would not be on the lnet network.<o:p></o:p></span></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">It tried to mount and threw some errors about unable to connect to the MGS and then kernel panicked. Here is what came out of
dmesg just before it did that:<o:p></o:p></span></p>
<p class="MsoNormal" style="border:none;padding:0in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LDISKFS-fs (dm-27): mounted filesystem with ordered data mode. quota=on. Opts:<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LNetError: 3243:0:(o2iblnd_cb.c:1267:kiblnd_resolve_addr()) Failed to bind to a free privileged port<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LNetError: 3243:0:(o2iblnd_cb.c:1321:kiblnd_connect_peer()) Can't resolve addr for 10.100.1.11@o2ib: -99<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Lustre: 3243:0:(client.c:1926:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1422655534/real 1422655534] req@ffff880fc93dcc00 x1491748356358156/t0(0)
o250->MGC10.100.1.11@o2ib@10.100.1.11@o2ib:26/25 lens 400/544 e 0 to 1 dl 1422655539 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LustreError: 20160:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880fc9dfbc00 x1491748356358160/t0(0) o253->MGC10.100.1.11@o2ib@10.100.1.11@o2ib:26/25
lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LustreError: 20160:0:(obd_mount_server.c:1165:server_register_target()) WORK-OST0005: error registering with the MGS: rc = -5 (not fatal)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LustreError: 20160:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880fc9dfbc00 x1491748356358164/t0(0) o101->MGC10.100.1.11@o2ib@10.100.1.11@o2ib:26/25
lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LustreError: 20160:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880fc9dfbc00 x1491748356358168/t0(0) o101->MGC10.100.1.11@o2ib@10.100.1.11@o2ib:26/25
lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LNetError: 3243:0:(o2iblnd_cb.c:1267:kiblnd_resolve_addr()) Failed to bind to a free privileged port<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LNetError: 3243:0:(o2iblnd_cb.c:1321:kiblnd_connect_peer()) Can't resolve addr for 10.100.1.10@o2ib: -99<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Lustre: 3243:0:(client.c:1926:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1422655559/real 1422655559] req@ffff8804100edc00 x1491748356358176/t0(0)
o250->MGC10.100.1.11@o2ib@10.100.1.10@o2ib:26/25 lens 400/544 e 0 to 1 dl 1422655564 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LustreError: 20160:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880fc9dfbc00 x1491748356358172/t0(0) o101->MGC10.100.1.11@o2ib@10.100.1.10@o2ib:26/25
lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LNetError: 3243:0:(o2iblnd_cb.c:1267:kiblnd_resolve_addr()) Failed to bind to a free privileged port<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LNetError: 3243:0:(o2iblnd_cb.c:1321:kiblnd_connect_peer()) Can't resolve addr for 10.100.1.11@o2ib: -99<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Lustre: 3243:0:(client.c:1926:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1422655568/real 1422655568] req@ffff8808156a2c00 x1491748356358184/t0(0)
o38->WORK-MDT0000-lwp-OST0005@10.100.1.11@o2ib:12/10 lens 400/544 e 0 to 1 dl 1422655573 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LustreError: 20160:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880fc9dfb000 x1491748356358188/t0(0) o101->MGC10.100.1.11@o2ib@10.100.1.10@o2ib:26/25
lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">LustreError: 20160:0:(client.c:1083:ptlrpc_import_delay_req()) Skipped 1 previous similar message<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">[root@nas-spare ~]#<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Message from syslogd@nas-spare at Jan 30 14:06:28 ...<o:p></o:p></span></i></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">kernel:Kernel panic - not syncing: Fatal exception<o:p></o:p></span></i></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">So it is still causing a kernel panic anytime I mount it….<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Brian Andrus<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">ITACS/Research Computing<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Naval Postgraduate School<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Monterey, California<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">voice: 831-656-6238<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></a></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Mervini, Joseph A [mailto:jamervi@sandia.gov]
<br>
<b>Sent:</b> Thursday, January 29, 2015 2:12 PM<br>
<b>To:</b> Andrus, Brian Contractor<br>
<b>Subject:</b> Re: [EXTERNAL] [Lustre-discuss] how do I deactivate a very wonky OST<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Brian, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Have you tried mount the OST outside of the the file system? By that I mean mounting it like it is not connect to the LNET network? It should mount, just complain about not being able to communicate with the MGS/MDS. Also, if you have quotas
turned of on the device and it mounts I would think that you should be able to re-register it with the overall file system.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Just a thought...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black">====<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black">Joe Mervini<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black">Sandia National Laboratories<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black">High Performance Computing<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black">505.844.6770<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black"><a href="mailto:jamervi@sandia.gov">jamervi@sandia.gov</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Jan 23, 2015, at 11:40 AM, Andrus, Brian Contractor <<a href="mailto:bdandrus@nps.edu">bdandrus@nps.edu</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Joe,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We are still having the issue. I did do the commands you suggested, that was what got me to a point I could get MOST of the system back up.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Current state, the ‘bad’ OST (OST5) is unregistered. When I tried to start an lfsck, it kernel panicked all of the OSSes and when I would try to bring up OST5,
that would kernel panic the OSS it was on. So I did a writeconf on everything and then brought everything back up except it to keep it from registering.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Doing the lfs find does not work directly, it throws an error for every file that has anything on the bad OST, so I was able to capture the STDERR to get the
info for those files.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I haven't given up hope on the OST since it passes e2fsk with no issues. I can mount it as ldiskfs and see everything too. But, as long as it is unavailable,
I cannot even delete or unlink files that are on it.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Right now, we have the filesystem up, it is working except any file that has data on the bad OST is inaccessible and cannot be removed. It would be nice to
figure out what is wrong with the OST that makes the OSS panic if it gets mounted as part of the filesystem.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Brian Andrus</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">ITACS/Research Computing</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Naval Postgraduate School</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Monterey, California</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">voice: 831-656-6238</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span class="apple-converted-space"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Mervini,
Joseph A [<a href="mailto:jamervi@sandia.gov"><span style="color:purple">mailto:jamervi@sandia.gov</span></a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Tuesday, January 20, 2015 11:46 AM<br>
<b>To:</b><span class="apple-converted-space"> </span>Andrus, Brian Contractor<br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [EXTERNAL] [Lustre-discuss] how do I deactivate a very wonky OST</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Hey Brian,<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Are you still fighting the issue? <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I had something similar come up with ldiskfs and quotas on a MDT on 2.5.3 that was also kernel panicking the system. I was able to get past it by shutting off quotas and the MDT and then re-enabling them. (I didn't know that lustre was
using Linux quota now...)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Anyway, the commands that I used to turn off quota on the device was:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">dumpe2fs -h <device> (To see what was in the superblock.)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">tune2fs -O ^quota <device><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I was then able to mount the file system with no problem.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I then re-enabled it with:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">tune2fs -O quota <device><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">And everything works fine now.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Hope this helps.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Joe</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">====</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Joe Mervini</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Sandia National Laboratories</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">High Performance Computing</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">505.844.6770</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""><a href="mailto:jamervi@sandia.gov"><span style="color:purple">jamervi@sandia.gov</span></a></span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Jan 16, 2015, at 10:06 AM, Andrus, Brian Contractor <<a href="mailto:bdandrus@nps.edu"><span style="color:purple">bdandrus@nps.edu</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Joe,</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We are using OSTs that are on an LVM connected via Infiniband. I have 17 OSTs and all the others are fine. I have also tried mounting this ‘bad’ one on other
hardware with the same result.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Brian Andrus</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">ITACS/Research Computing</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Naval Postgraduate School</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Monterey, California</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">voice: 831-656-6238</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span class="apple-converted-space"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Mervini,
Joseph A [<a href="mailto:jamervi@sandia.gov"><span style="color:purple">mailto:jamervi@sandia.gov</span></a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Thursday, January 15, 2015 10:09 AM<br>
<b>To:</b><span class="apple-converted-space"> </span>Andrus, Brian Contractor<br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [EXTERNAL] Re: [Lustre-discuss] how do I deactivate a very wonky OST</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Brian,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Oh, sorry. I didn't realize you were using 2.6 - don't know anything about that version yet.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">It doesn't make a lot of sense that if you can fsck the file system it would panic the system on mount. What is the hardware that you are using (i.e., is the storage direct attached or external, is it a JBOD or raid device, etc..)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I happy to make suggestions that may help.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Joe<o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">====</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Joe Mervini</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Sandia National Laboratories</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">High Performance Computing</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">505.844.6770</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""><a href="mailto:jamervi@sandia.gov"><span style="color:purple">jamervi@sandia.gov</span></a></span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Jan 15, 2015, at 10:06 AM, Andrus, Brian Contractor <<a href="mailto:bdandrus@nps.edu"><span style="color:purple">bdandrus@nps.edu</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Joe,</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Is that something that can be done on an offline, unavailable OST under lustre 2.6? I thought quotas were now built-in.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">It is not a persistent mount parameter on the OST when I user tunefs.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I have done tune2fs and removed/rebuilt the quota info on the backing filesystem, that removes the quota errors from e2fsck, but it still kernel panics the
OSS when it mounts.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I cannot even seem to unlink any files that use the missing OST either. It seems that if you completely lose an OST that is not recoverable in lustre 2.6, you
cannot do much with the files that were on it….</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I haven’t given up hope on the OST yet, though…</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Brian Andrus</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">ITACS/Research Computing</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Naval Postgraduate School</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Monterey, California</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">voice: 831-656-6238</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span class="apple-converted-space"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Mervini,
Joseph A [<a href="mailto:jamervi@sandia.gov"><span style="color:purple">mailto:jamervi@sandia.gov</span></a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Wednesday, January 14, 2015 3:34 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Andrus, Brian Contractor<br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [EXTERNAL] Re: [Lustre-discuss] how do I deactivate a very wonky OST</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Brian,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I have one thought: You did say that you could run fsck against the file system but it was complaining about the quota file? You might try removing the quota parameter from the OST configuration and see if you can mount it then. If it
does, then you can reinsert the parameter. The only downside to that will be that you'll have to do a quota check again but that would be considerably better that trying to hack the file system back into a working order.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Hope this helps/works.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">====</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Joe Mervini</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Sandia National Laboratories</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">High Performance Computing</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">505.844.6770</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""><a href="mailto:jamervi@sandia.gov"><span style="color:purple">jamervi@sandia.gov</span></a></span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Jan 14, 2015, at 4:11 PM, Andrus, Brian Contractor <<a href="mailto:bdandrus@nps.edu"><span style="color:purple">bdandrus@nps.edu</span></a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks Sean,</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Right now neither help me as I had to bring the entire system up from scratch and NOT mount the bad OST.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">So, now OST5 is not listed anywhere. The only knows it is missing.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Doing ‘lctl dl’ only show the OSTs that have been brought up.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If I try to bring it up, it registers, the MDS becomes aware, the OSS kernel panics and the MDS starts making everyone wait for it to come back.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Part of ‘lfs df’:</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">OST0005 : Resource temporarily unavailable</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hassle is I cannot really do an ‘lfs find’ for the files on the bad OST because the OST is not registered… stuck in a loop here…</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If I could find a way to tag it as offline even though the MDS doesn’t see it yet, that may help.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Brian Andrus</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">ITACS/Research Computing</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Naval Postgraduate School</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Monterey, California</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">voice: 831-656-6238</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span class="apple-converted-space"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Sean
Brisbane [<a href="mailto:s.brisbane1@physics.ox.ac.uk"><span style="color:purple">mailto:s.brisbane1@physics.ox.ac.uk</span></a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Wednesday, January 14, 2015 3:04 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Andrus, Brian Contractor;<span class="apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org"><span style="color:purple">lustre-discuss@lists.lustre.org</span></a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>RE: how do I deactivate a very wonky OST</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">This caught me out in a recent upgrade:<br>
<br>
cat /proc/fs/lustre/lov/{yourmdt}/target_obd<br>
<br>
rather than<span class="apple-converted-space"> </span><br>
<br>
"lctl dl"<br>
<br>
Shows the state of the OST.<br>
<br>
Cheers,<br>
Sean</span><o:p></o:p></p>
<div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<div id="divRpF648475">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span class="apple-converted-space"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""><a href="mailto:lustre-discuss-bounces@lists.lustre.org"><span style="color:purple">lustre-discuss-bounces@lists.lustre.org</span></a><span class="apple-converted-space"> </span>[<a href="mailto:lustre-discuss-bounces@lists.lustre.org"><span style="color:purple">lustre-discuss-bounces@lists.lustre.org</span></a>]
on behalf of Andrus, Brian Contractor [<a href="mailto:bdandrus@nps.edu"><span style="color:purple">bdandrus@nps.edu</span></a>]<br>
<b>Sent:</b><span class="apple-converted-space"> </span>13 January 2015 17:28<br>
<b>To:</b><span class="apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org"><span style="color:purple">lustre-discuss@lists.lustre.org</span></a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>[Lustre-discuss] how do I deactivate a very wonky OST</span><o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">All,</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">We are still trying to move forward getting our filesystem at least partially up with a failed OST.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Currently the OST will kernel panic any device that mounts it. That seems to be a constant.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">So, the plan is to bring the system up without that OST and find what data will be lost.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Now, I am trying to deactivate the OST on the MGS, but it seems to have no effect.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Running lctl --device 14 deactivate does not change anything. The OST still shows ‘UP’</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Is there a way to force lustre to deactivate an OST altogether when it is showing ‘UP’ and the OST is not going to be happily mounted?</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">I can mount the filesystem, but many actions hang (ls, df, etc).</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Brian Andrus</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">ITACS/Research Computing</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Naval Postgraduate School</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Monterey, California</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">voice: 831-656-6238</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica","sans-serif"">_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org"><span style="color:purple">Lustre-discuss@lists.lustre.org</span></a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss"><span style="color:purple">http://lists.lustre.org/mailman/listinfo/lustre-discuss</span></a></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>