<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:"Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">We’ve been testing DNE Phase II and tried scaling the number of MDSes(one MDT each for all of our tests) very high, but when we did that, we couldn’t mount the filesystem on a client. After trial and error,
we discovered that we were unable to mount the filesystem when there were 56 MDSes. 55 MDSes mounted without issue, and it appears any number below that will mount. This failure at 56 MDSes was replicable across different nodes being used for the MDSes, all
of which were tested with working configurations, so it doesn’t seem to be a bad server.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Here’s the error info we saw in dmesg on the client:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">LustreError: 28880:0:(obd_config.c:559:class_setup()) setup lustre-MDT0037-mdc-ffff95923d31b000 failed (-16)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">LustreError: 28880:0:(obd_config.c:1836:class_config_llog_handler()) MGCx.x.x.x@o2ib: cfg command failed: rc = -16<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Lustre: cmd=cf003 0:lustre-MDT0037-mdc 1:lustre-MDT0037_UUID 2:x.x.x.x@o2ib<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">LustreError: 15c-8: MGCx.x.x.x@o2ib: The configuration from log 'lustre-client' failed (-16). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors.
See the syslog for more information.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">LustreError: 28858:0:(obd_config.c:610:class_cleanup()) Device 58 not setup<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Lustre: Unmounted lustre-client<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">LustreError: 28858:0:(obd_mount.c:1608:lustre_fill_super()) Unable to mount (-16)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">OS: CentOS 7.6.1810 <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Kernel: 3.10.0-957.5.1.el7.x86_64<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Lustre: 2.12.1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Network card: Qlogic InfiniPath_QLE7340<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Other things to note for completeness’ sake: this happened with both ldiskfs and zfs backfstypes, and these tests were using files in memory as the backing devices.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Is there something I’m missing as to why more than 56 MDSes won’t mount?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Scott White<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Scientist, HPC <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Los Alamos National Laboratory<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</body>
</html>