<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.emailquote, li.emailquote, div.emailquote
{mso-style-name:emailquote;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:1.0pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1134715607;
mso-list-template-ids:1316771816;}
@list l0:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1
{mso-list-id:1182620077;
mso-list-template-ids:489063392;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hi Grégoire,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks for the reply.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I thought that might be the issue as well. However, it doesn’t explain why the other two clients work successfully when both interfaces are on the same network.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>…Brian<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Gregoire Pichon [mailto:gregoire.pichon@bull.net] <br><b>Sent:</b> Wednesday, February 13, 2013 4:19 AM<br><b>To:</b> mages, brian; lustre-discuss@lists.lustre.org<br><b>Subject:</b> RE: Multirail IB Configuration Issue<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=FR style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Hi,<o:p></o:p></span></p><p class=MsoNormal><span lang=FR style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>The two LNet networks need separate IPoIB networks.<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>o2ib0 - 10.0.0.0/24<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>o2ib1 - 10.1.0.0/24<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Nodes with 2 physical interfaces (like you servers) have ib0 on o2ib0 and ib1 on o2ib1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Nodes with 1 physical interface (like your clients) have 2 logical interfaces: ib0 on o2ib0 and ib0:0 on o2ib1.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Grégoire.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span lang=FR style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>De :</span></b><span lang=FR style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> <a href="mailto:lustre-discuss-bounces@lists.lustre.org">lustre-discuss-bounces@lists.lustre.org</a> [<a href="mailto:lustre-discuss-bounces@lists.lustre.org">mailto:lustre-discuss-bounces@lists.lustre.org</a>] <b>De la part de</b> mages, brian<br><b>Envoyé :</b> mardi 12 février 2013 21:34<br><b>À :</b> <a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br><b>Objet :</b> [Lustre-discuss] Multirail IB Configuration Issue<o:p></o:p></span></p></div></div><p class=MsoNormal><span lang=FR><o:p> </o:p></span></p><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>Hi,<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>I’m having difficulty getting one of my clients to work with a multirail IB configuration. Here’s what I’ve got:<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <b>Host OS Version Lustre Version Function Storage Interface ib0 Interface ib1</b><o:p></o:p></span></p></div><ol start=1 type=1><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo3'><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>bmr1-s7 CentOS 5.7 2.1.1 MGS,MDS,OSS1 mdt,mdt2,ost1->6,ost13->18 192.168.1.25/24 192.168.1.35/24<o:p></o:p></span></li><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo3'><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>bmr1-s8 CentOS 5.7 2.1.1 OSS2 ost7->12,ost19->24 192.168.1.26/24 192.168.1.36/24<o:p></o:p></span></li><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo3'><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>bmr1-s5 CentOS 5.7 2.1.1 OSS3 ost25->30 192.168.1.20/24 192.168.1.30/24<o:p></o:p></span></li><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo3'><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>bmr1-s6 CentOS 5.7 2.1.1 OSS4 ost31->36 192.168.1.21/24 192.168.1.31/24<o:p></o:p></span></li><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo3'><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>bmr2-s9 CentOS 5.7 2.1.1 Client n/a 192.168.1.209/24<o:p></o:p></span></li></ol><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>The “/lustre” filesystem consists of mdt and ost1->12 (using bmr1-s7 and bmr1-s8).<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>The “/lustre2” filesystem consists of mdt2 and ost13->36 (using bmr1-s7, bmr1-s8, bmr1-s5, and bmr1-s6).<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>On each OSS, half the OSTs are available only on ib0 and the other half only on ib1.<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>From bmr1-s5 and bmr1-s6 (using as clients), I can successfully mount and access “/lustre”. I can also successfully mount “/lustre2”.<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>From bmr2-s9, I can neither mount “/lustre” nor “/lustre2”. Originally, the issue with bmr2-s9 was that it was running 1.8.6-wc1 (server on CentOS 5.6). Since this config (i.e., multirail) wasn’t supported on that version, I upgraded to 2.1.1. Originally, I tried installing and testing the 2.1.1 client without success. Then, since it had worked with the 2.1.1 server on both bmr1-s5 and bmr1-s6, I thought I’d try that next. Unfortunately, it still didn’t work.<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>1a) Here’s what I see on the client when I try to mount “/lustre”:<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr2-s9 ~]# mount -t lustre <a href="mailto:192.168.1.25@o2ib:/lustre">192.168.1.25@o2ib:/lustre</a> /mnt/lustre</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>mount.lustre: mount <a href="mailto:192.168.1.25@o2ib:/lustre">192.168.1.25@o2ib:/lustre</a> at /mnt/lustre failed: No such file or directory</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Is the MGS specification correct?</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Is the filesystem name correct?</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>If upgrading, is the copied client log valid? (see upgrade docs)</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr2-s9 ~]#</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>1b) Here’s an excerpt from “/var/log/messages” on the client (after executing the above command):<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: Lustre: 5512:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import <a href="mailto:MGC192.168.1.25@o2ib-%3eMGC192.168.1.25@o2ib_0">MGC192.168.1.25@o2ib->MGC192.168.1.25@o2ib_0</a> netid 50000: select flavor null</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: Lustre: <a href="mailto:MGC192.168.1.25@o2ib">MGC192.168.1.25@o2ib</a>: Reactivating import</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5523:0:(ldlm_lib.c:357:client_obd_setup()) can't add initial connection</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5523:0:(obd_config.c:522:class_setup()) setup lustre-OST0001-osc-ffff81045d783c00 failed (-2)</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5523:0:(obd_config.c:1361:class_config_llog_handler()) Err -2 on cfg command:</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: Lustre: cmd=cf003 0:lustre-OST0001-osc 1:lustre-OST0001_UUID 2:192.168.1.35@o2ib1 </span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 15c-8: <a href="mailto:MGC192.168.1.25@o2ib">MGC192.168.1.25@o2ib</a>: The configuration from log 'lustre-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5512:0:(llite_lib.c:950:ll_fill_super()) Unable to process log: -2</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 4923:0:(lov_obd.c:927:lov_cleanup()) lov tgt 0 not cleaned! deathrow=0, lovrc=1</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5512:0:(obd_config.c:567:class_cleanup()) Device 5 not setup</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5512:0:(ldlm_request.c:1172:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5512:0:(ldlm_request.c:1799:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: Lustre: client ffff81045d783c00 umount complete</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr2-s9 kernel: LustreError: 5512:0:(obd_mount.c:2164:lustre_fill_super()) Unable to mount (-2)</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>1c) Here’s an excerpt from “/var/log/messages” on the server (after executing the above command):<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr1-s7 kernel: Lustre: 25911:0:(ldlm_lib.c:877:target_handle_connect()) MGS: connection from <a href="mailto:2e13dea0-ec9c-0fbd-0f95-7b16246f2626@192.168.1.209@o2ib">2e13dea0-ec9c-0fbd-0f95-7b16246f2626@192.168.1.209@o2ib</a> t0 exp 0000000000000000 cur 1360699254 last 0</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:00:54 bmr1-s7 kernel: Lustre: 25911:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x50000c0a801d1_UUID netid 50000: select flavor null</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>2a) Here’s what I see on the client when I try to mount “/lustre” (using the other interface):<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr2-s9 ~]# mount -t lustre <a href="mailto:192.168.1.25@o2ib:/lustre">192.168.1.25@o2ib:/lustre</a> /mnt/lustre</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>mount.lustre: mount <a href="mailto:192.168.1.25@o2ib:/lustre">192.168.1.25@o2ib:/lustre</a> at /mnt/lustre failed: No such file or directory</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Is the MGS specification correct?</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Is the filesystem name correct?</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>If upgrading, is the copied client log valid? (see upgrade docs)</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr2-s9 ~]# mount -t lustre <a href="mailto:192.168.1.35@o2ib:/lustre">192.168.1.35@o2ib:/lustre</a> /mnt/lustre</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>mount.lustre: mount <a href="mailto:192.168.1.35@o2ib:/lustre">192.168.1.35@o2ib:/lustre</a> at /mnt/lustre failed: Invalid argument</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>This may have multiple causes.</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Is 'lustre' the correct filesystem name?</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Are the mount options correct?</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Check the syslog for more info.</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr2-s9 ~]#</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>2b) Here’s an excerpt from “/var/log/messages” on the client (after executing the above command):<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr2-s9 kernel: Lustre: 5580:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import <a href="mailto:MGC192.168.1.35@o2ib-%3eMGC192.168.1.35@o2ib_0">MGC192.168.1.35@o2ib->MGC192.168.1.35@o2ib_0</a> netid 50000: select flavor null</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr2-s9 kernel: Lustre: 4948:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1426793186721863 sent from <a href="mailto:MGC192.168.1.35@o2ib">MGC192.168.1.35@o2ib</a> to NID <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a> has failed due to network error: [sent 1360699617] [real_sent 1360699617] [current 1360699617] [deadline 5s] [delay -5s] req@ffff81043b76e400 x1426793186721863/t0(0) o-1-><a href="mailto:MGS@MGC192.168.1.35@o2ib_0:26/25">MGS@MGC192.168.1.35@o2ib_0:26/25</a> lens 368/512 e 0 to 1 dl 1360699622 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr2-s9 kernel: Lustre: 4948:0:(client.c:1778:ptlrpc_expire_one_request()) Skipped 1 previous similar message</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr2-s9 kernel: LustreError: 3074:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a> rejected: o2iblnd fatal error</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr2-s9 kernel: LustreError: 3074:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) Skipped 1 previous similar message</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:03 bmr2-s9 kernel: LustreError: 5580:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff81043b76e000 x1426793186721864/t0(0) o-1-><a href="mailto:MGS@MGC192.168.1.35@o2ib_0:26/25">MGS@MGC192.168.1.35@o2ib_0:26/25</a> lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:03 bmr2-s9 kernel: LustreError: 5580:0:(client.c:1049:ptlrpc_import_delay_req()) Skipped 6 previous similar messages</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:22 bmr2-s9 kernel: Lustre: 4949:0:(import.c:526:import_select_connection()) <a href="mailto:MGC192.168.1.35@o2ib">MGC192.168.1.35@o2ib</a>: tried all connections, increasing latency to 5s</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:22 bmr2-s9 kernel: Lustre: 4948:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1426793186721868 sent from <a href="mailto:MGC192.168.1.35@o2ib">MGC192.168.1.35@o2ib</a> to NID <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a> has failed due to network error: [sent 1360699642] [real_sent 1360699642] [current 1360699642] [deadline 10s] [delay -10s] req@ffff810430e30800 x1426793186721868/t0(0) o-1-><a href="mailto:MGS@MGC192.168.1.35@o2ib_0:26/25">MGS@MGC192.168.1.35@o2ib_0:26/25</a> lens 368/512 e 0 to 1 dl 1360699652 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:22 bmr2-s9 kernel: LustreError: 3074:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a> rejected: o2iblnd fatal error</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:24 bmr2-s9 kernel: LustreError: 5591:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff81045d7ce800 x1426793186721867/t0(0) o-1-><a href="mailto:MGS@MGC192.168.1.35@o2ib_0:26/25">MGS@MGC192.168.1.35@o2ib_0:26/25</a> lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/ffffffff/ffffffff rc 0/-1</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:24 bmr2-s9 kernel: LustreError: 5591:0:(client.c:1049:ptlrpc_import_delay_req()) Skipped 1 previous similar message</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:47 bmr2-s9 kernel: Lustre: 4949:0:(import.c:526:import_select_connection()) <a href="mailto:MGC192.168.1.35@o2ib">MGC192.168.1.35@o2ib</a>: tried all connections, increasing latency to 10s</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:47 bmr2-s9 kernel: Lustre: 4948:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1426793186721872 sent from <a href="mailto:MGC192.168.1.35@o2ib">MGC192.168.1.35@o2ib</a> to NID <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a> has failed due to network error: [sent 1360699667] [real_sent 1360699667] [current 1360699667] [deadline 15s] [delay -15s] req@ffff810444576c00 x1426793186721872/t0(0) o-1-><a href="mailto:MGS@MGC192.168.1.35@o2ib_0:26/25">MGS@MGC192.168.1.35@o2ib_0:26/25</a> lens 368/512 e 0 to 1 dl 1360699682 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:47 bmr2-s9 kernel: LustreError: 3074:0:(o2iblnd_cb.c:2615:kiblnd_rejected()) <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a> rejected: o2iblnd fatal error</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:54 bmr2-s9 kernel: LustreError: 156-2: The client profile 'lustre-client' could not be read from the MGS. Does that filesystem exist?</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:54 bmr2-s9 kernel: Lustre: client ffff81045f465800 umount complete</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:54 bmr2-s9 kernel: LustreError: 5580:0:(obd_mount.c:2164:lustre_fill_super()) Unable to mount (-22)</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>2c) Here’s an excerpt from “/var/log/messages” on the server (after executing the above command):<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr1-s7 kernel: LustreError: 9274:0:(o2iblnd_cb.c:2247:kiblnd_passive_connect()) Can't accept <a href="mailto:192.168.1.209@o2ib">192.168.1.209@o2ib</a> on <a href="mailto:192.168.1.25@o2ib">192.168.1.25@o2ib</a> (ib1:1:192.168.1.35): bad dst nid <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a></span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:06:57 bmr1-s7 kernel: LustreError: 9274:0:(o2iblnd_cb.c:2247:kiblnd_passive_connect()) Skipped 2 previous similar messages</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Feb 12 15:07:22 bmr1-s7 kernel: LustreError: 9274:0:(o2iblnd_cb.c:2247:kiblnd_passive_connect()) Can't accept <a href="mailto:192.168.1.209@o2ib">192.168.1.209@o2ib</a> on <a href="mailto:192.168.1.25@o2ib">192.168.1.25@o2ib</a> (ib1:1:192.168.1.35): bad dst nid <a href="mailto:192.168.1.35@o2ib">192.168.1.35@o2ib</a></span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>3) Here’s what one of the MDTs looks like (the other is similarly configured):<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr1-s7 ~]# tunefs.lustre --dryrun --writeconf /dev/sdp</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>checking for existing Lustre data: found CONFIGS/mountdata</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Reading CONFIGS/mountdata</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> </span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> Read previous values:</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Target: lustre-MDT0000</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Index: 0</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Lustre FS: lustre</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Mount type: ldiskfs</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Flags: 0x5</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> (MDT MGS )</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Parameters: <a href="mailto:mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1">mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1</a></span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> Permanent disk data:</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Target: lustre-MDT0000</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Index: 0</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Lustre FS: lustre</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Mount type: ldiskfs</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Flags: 0x105</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> (MDT MGS writeconf )</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Parameters: <a href="mailto:mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1">mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1</a></span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>exiting before disk write.</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr1-s7 ~]#</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>4) Here’s what one of the OSTs looks like (the others are similarly configured):<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr1-s7 ~]# tunefs.lustre --dryrun --writeconf /dev/sdf</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>checking for existing Lustre data: found CONFIGS/mountdata</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Reading CONFIGS/mountdata</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> </span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> Read previous values:</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Target: lustre-OST0000</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Index: 0</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Lustre FS: lustre</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Mount type: ldiskfs</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Flags: 0x2</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> (OST )</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Persistent mount opts: errors=remount-ro,extents,mballoc</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Parameters: <a href="mailto:mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1">mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1</a> network=o2ib0</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> </span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> </span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> Permanent disk data:</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Target: lustre-OST0000</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Index: 0</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Lustre FS: lustre</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Mount type: ldiskfs</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Flags: 0x102</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> (OST writeconf )</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Persistent mount opts: errors=remount-ro,extents,mballoc</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>Parameters: <a href="mailto:mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1">mgsnode=192.168.1.25@o2ib,192.168.1.35@o2ib1,10.244.78.88@tcp,192.168.1.25@tcp1</a> network=o2ib0</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'> </span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>exiting before disk write.</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Courier New"'>[root@bmr1-s7 ~]#</span><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>I’d appreciate any help or direction on a potential resolution. Let me know what additional information is needed, if any. Hopefully, I’m just missing something simple.<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>Thanks in advance,<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'>…Brian<o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span lang=FR style='font-size:10.0pt;font-family:"Calibri","sans-serif"'> <o:p></o:p></span></p></div></div></body></html>