<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle27
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle28
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle29
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle30
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle31
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle32
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle33
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle34
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle35
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle36
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle37
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle38
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle39
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle40
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle41
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">></span><span style="color:#1F497D">Hi James,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">></span><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">></span><span style="color:#1F497D">Are you able to disclose what OS version/stack you’re running on Summitdev where you have Lustre mounted? Is the machine running Ubuntu?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">></span><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">></span><span style="color:#1F497D">There seem to be significant pains to get the client working here on the ppcle platform, under RHEL/Centos 7.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Moved this topic to lustre-devel. So my Power8 clients are RHEL7.3. First to make life easy for you please download the latest lustre 2.10 client.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">git clone git://git.hpdd.intel.com/fs/lustre/lustre-release<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">git checkout –b b2_10 origin/b2_10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The latest has almost all the patches you need. The reason for your build failure is due to the lack of SNMP on the nodes. The lustre spec file<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">assumes SNMP is always there. This is wrong and I opened a ticket -
<a href="https://jira.hpdd.intel.com/browse/LU-9870">https://jira.hpdd.intel.com/browse/LU-9870</a>. I created a patch that resolves<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">this. Just download from here - <a href="https://review.whamcloud.com/#/c/28494">
https://review.whamcloud.com/#/c/28494</a> and apply. With that you should be able to build your rpms. Now you<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">try to mount and run into <a href="https://jira.hpdd.intel.com/browse/LU-9823">
https://jira.hpdd.intel.com/browse/LU-9823</a> I can’t help you since I don’t have a solution. I do have a patch in the works<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">to fix several of the config log issues (LU-7004) which I hope will fix this issue.<b><o:p></o:p></b></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Russell JONES
<br>
<b>Sent:</b> Wednesday, August 09, 2017 4:41 PM<br>
<b>To:</b> 'Simmons, James A.'; Donny COOPER; Leverman, Dustin B.; Mehta, Kshitij V.; Oral, H. Sarp; Hill, Jason J.<br>
<b>Subject:</b> RE: Lustre on Summitdev<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">No worries about the delay
</span><span style="font-family:Wingdings;color:#1F497D">J</span><span style="color:#1F497D"><br>
<br>
No, not cross compiling. I have been giving it that configure flag because without it, it was not detecting the architecture correctly (gave me an error about attempting to build for big endian when system is little endian). I suppose I should have pointed
that out at the beginning, sorry. I had been attempting to resolve these other issues long enough that I honestly forgot I was even adding it in there! This decision and the architecture error was prior to cloning the current lustre tree and applying your
patch.<br>
<br>
I started over with a clean copy of the source tree again, reapplied your patch, and re-ran autogen and configure with the following flags: ./configure --disable-server --with-o2ib=/usr/src/ofa_kernel/default --disable-tests.<br>
<br>
Configure completes as does make. Make rpms errors with:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">+ basemodpath=/tmp/rpmbuild-lustre-l0360328-nY6H5CMn/BUILDROOT/lustre-2.10.0_25_gc25132d_dirty-1.ppc64le/lib/modules/3.10.0-514.el7.ppc64le/extra/lustre-client<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">+ :<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">+ echo /usr/lib/systemd/system/lnet.service<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">+ echo /etc/init.d/lsvcgss<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">+ find /tmp/rpmbuild-lustre-l0360328-nY6H5CMn/BUILDROOT/lustre-2.10.0_25_gc25132d_dirty-1.ppc64le -name '*.so' -type f -exec chmod +x '{}' ';'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">+ rm -f /tmp/rpmbuild-lustre-l0360328-nY6H5CMn/BUILDROOT/lustre-2.10.0_25_gc25132d_dirty-1.ppc64le/usr/lib64/liblnetconfig.la<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">+ echo '%attr(-, root, root) /usr/lib64/liblnetconfig.a'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">+ echo '%attr(-, root, root) /usr/lib64/liblnetconfig.so'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">+ echo '%attr(-, root, root) /usr/lib64/liblnetconfig.so.*'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">+ '[' -d /tmp/rpmbuild-lustre-l0360328-nY6H5CMn/BUILDROOT/lustre-2.10.0_25_gc25132d_dirty-1.ppc64le/usr/lib64/lustre/snmp ']'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">+ mkdir -p /tmp/rpmbuild-lustre-l0360328-nY6H5CMn/BUILDROOT/lustre-2.10.0_25_gc25132d_dirty-1.ppc64le//usr/share/lustre<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">+ find /tmp/rpmbuild-lustre-l0360328-nY6H5CMn/BUILDROOT/lustre-2.10.0_25_gc25132d_dirty-1.ppc64le/usr/lib64/lustre -name '*.la' -type f -exec rm -f '{}' ';'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">find: '/tmp/rpmbuild-lustre-l0360328-nY6H5CMn/BUILDROOT/lustre-2.10.0_25_gc25132d_dirty-1.ppc64le/usr/lib64/lustre': No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Courier New";color:#1F497D">error: Bad exit status from /tmp/rpmbuild-lustre-l0360328-nY6H5CMn/TMP/rpm-tmp.XgLKEM (%install)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Simmons, James A. [<a href="mailto:simmonsja@ornl.gov">mailto:simmonsja@ornl.gov</a>]
<br>
<b>Sent:</b> Wednesday, August 09, 2017 3:18 PM<br>
<b>To:</b> Russell JONES; Donny COOPER; Leverman, Dustin B.; Mehta, Kshitij V.; Oral, H. Sarp; Hill, Jason J.<br>
<b>Subject:</b> RE: Lustre on Summitdev<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">>> Did you install libyaml-devel rpm?<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">>Yes, both libyaml and libyaml-devel are installed.<br>
><br>
><br>
>I have successfully cloned and applied the patch without errors. Autogen and configure finish just fine, however a make errors out with:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">></span><span style="font-family:"Courier New";color:#1F497D">gcc -DHAVE_CONFIG_H -I. -I../.. -D_GNU_SOURCE -D_LARGEFILE64_SOURCE=1 -D_FILE_OFFSET_BITS=64 -DLUSTRE_UTILS=1 -include
/home/l0360328/lustre->release/undef.h -include /home/l0360328/lustre-release/config.h -I/home/l0360328/lustre-release/libcfs/include -I/home/l0360328/lustre->release/lnet/include -I/home/l0360328/lustre-release/lustre/include -I/home/l0360328/lustre-release/lustre/include/uapi
-fPIC -g -O2 -MT >libcfsutil_a-parser.o -MD -MP -MF .deps/libcfsutil_a-parser.Tpo -c -o libcfsutil_a-parser.o `test -f 'util/parser.c' || echo './'`util/parser.c</span><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>In file included from <command-line>:0:0:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>/usr/include/stdc-predef.h:40:1: fatal error: /home/l0360328/lustre-release/undef.h: No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>#endif<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">>Configure line:</span><span style="font-family:"Courier New";color:#1F497D"> ./configure --build=ppc64le --disable-server --with-o2ib=/usr/src/ofa_kernel/default --disable-tests<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Never tried build=ppc64le. Are you cross compiling? BTW I can reproduce your rpm build issue. That is why I didn’t response right away. I was attempting to figure out<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">what is wrong. So basically autoconf is setting your libdir to /usr/lib64 and the rpm macros expect the libraries to be in /usr/lib. I see other projects have had issues<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">with this before but I didn’t find a good solution yet. It will take me a bit to figure it out.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Simmons, James A. [<a href="mailto:simmonsja@ornl.gov">mailto:simmonsja@ornl.gov</a>]
<br>
<b>Sent:</b> Tuesday, August 08, 2017 9:20 AM<br>
<b>To:</b> Russell JONES; Donny COOPER; Leverman, Dustin B.; Mehta, Kshitij V.; Oral, H. Sarp; Hill, Jason J.<br>
<b>Subject:</b> RE: Lustre on Summitdev<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">>James,<br>
><br>
>When setting the new configuration options I noticed that I didn’t appear to have lnetctl on the system. I found an older bug report that hinted I needed libyaml and libyaml-devel for the binary to get built. Installed those and >ran another rpmbuild, but
unfortunately I still didn’t get an RPM created that included that binary.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">Did you install libyaml-devel rpm?<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D"><br>
>I decided to try the route of patching the source tree to fix LUA-9758 and see if building that route would give me the binary, however I’m still getting the same error with the updated lustre.spec.in downloaded and put in place, >and a make clean, configure,
make, make rpms ran. I viewed the new lustre.spec that gets created after configure to make sure your changed lines appear there, and they seem to be there.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">Did the patch apply? The reason I ask is that I created that patch against a later lustre 2.10 version. Patches are still landing to the 2.10 branch for the 2.10.1 release. Try the
following<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">git clone git://git.hpdd.intel.com/fs/lustre-release<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">git checkout –b b2_10 origin/b2_10<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">Then apply the LU-9758 patch and with libyaml-devel installed try a build.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D"><br>
>I know this discussion is getting a bit long and technical, if there’s a better place to continue it (devel list or bug tracker?) I’ll be happy to move to wherever is more convenient for you.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">I’m fine where ever the discuss takes place.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Simmons, James A. [<a href="mailto:simmonsja@ornl.gov">mailto:simmonsja@ornl.gov</a>]
<br>
<b>Sent:</b> Monday, August 07, 2017 4:19 PM<br>
<b>To:</b> Donny COOPER; Russell JONES; Leverman, Dustin B.; Mehta, Kshitij V.; Oral, H. Sarp; Hill, Jason J.<br>
<b>Subject:</b> RE: Lustre on Summitdev<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">>James,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">>Is the configuration of OLCF Lustre filesystem that is connecting to the Summitdev (ppcle64) machine using mlx5 -> mlx5 on both Lustre client and server?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Is not so much a mlx4 vs mlx5 driver issue but what the hardware supports. So here is the correct technical explanation of what is going. IB hardware support<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">something called queue pairs. How deep the queue pair can go depends on the hardware. In our testbed the back end file system I was testing with, which<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">does have mlx4 based hardware, could support a queue depth of 64K. This is also true of our production file system storage back end. This setup allows us<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">to set our lnet peer credits to 63. The Power8 nodes Mellanox hardware in our testbed has a maximum queue pair depth of 32K. Because of this I couldn’t<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">push the lnet peer credits to 63. So what I did to get around that is turn on map_on_demand. The map_on_demand option in ko2iblnd turns on RDMA<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">transfers. This helped me to support the 63 peer credits I wanted on the Power8 nodes but it exposed a problem due to different page sizes. That is what<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">caused me some headaches. Now for our Summitdev machine its hardware, even with it using the mlx5 driver, doesn’t seem to have problems with our<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">back end production file system. So how this will impact you will depend on your setup. You just have to try and see with the LNet peer_credits you<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">are using.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Simmons, James A. [<a href="mailto:simmonsja@ornl.gov">mailto:simmonsja@ornl.gov</a>]
<br>
<b>Sent:</b> Monday, August 07, 2017 3:20 PM<br>
<b>To:</b> Russell JONES <<a href="mailto:russell.jones@external.total.com">russell.jones@external.total.com</a>>; Leverman, Dustin B. <<a href="mailto:leverman@ornl.gov">leverman@ornl.gov</a>>; Donny COOPER <<a href="mailto:donny.cooper@total.com">donny.cooper@total.com</a>>;
Mehta, Kshitij V. <<a href="mailto:mehtakv@ornl.gov">mehtakv@ornl.gov</a>>; Oral, H. Sarp <<a href="mailto:oralhs@ornl.gov">oralhs@ornl.gov</a>>; Hill, Jason J. <<a href="mailto:hilljj@ornl.gov">hilljj@ornl.gov</a>><br>
<b>Subject:</b> RE: Lustre on Summitdev<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">>Thanks for the feedback!<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">>Yes we do have a mlx5 -> mlx4 connection in play, our lustre servers are mlx4.
<br>
><br>
>I will start work on getting the configuration put into place you recommended. <o:p>
</o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">You might still have issues mounting due the LU-9823 bug. I collected the debug log but it is going to take some time for me to figure out what is wrong.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:#1F497D">As for the mlx5 <-> mlx4 if you have trouble, if you manage to get around LU-9823, let me know and I can help you with that.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Simmons, James A. [<a href="mailto:simmonsja@ornl.gov">mailto:simmonsja@ornl.gov</a>]
<br>
<b>Sent:</b> Monday, August 07, 2017 2:18 PM<br>
<b>To:</b> Russell JONES; Leverman, Dustin B.; Donny COOPER; Mehta, Kshitij V.; Oral, H. Sarp; Hill, Jason J.<br>
<b>Subject:</b> RE: Lustre on Summitdev<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">>Hi all,<br>
><br>
>Appreciate the assistance! Here’s a quick overview of what we are experiencing, and then further answers inline below.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">>If we do a simple rpmbuild against the 2.10 source RPM, the build completes and I am able to modprobe the lustre module and issue a mount command against our Lustre filesystem. However as soon as I do that,
or >attempt any writes, we begin getting the following errors recorded in /var/log/messages on the node, and the filesystem is unusable from this client:<br>
><br>
</span><span style="font-family:"Courier New";color:#1F497D">>Aug 7 13:32:49 p8eval kernel: Lustre: Lustre: Build Version: 2.10.0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>Aug 7 13:32:49 p8eval kernel: LNet: Added LNI
<a href="mailto:172.40.120.231@o2ib4">172.40.120.231@o2ib4</a> [8/256/0/180]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>Aug 7 13:33:06 p8eval kernel: Lustre: 61690:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent >1502130781/real 1502130781] req@c000000fd85e0300
x1575098273234960/t0(0) o250-><a href="mailto:MGC172.40.2.60@o2ib4@172.40.2.60@o2ib4:26/25">MGC172.40.2.60@o2ib4@172.40.2.60@o2ib4:26/25</a> lens 520/544 e 0 to >.1 dl 1502130786 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>Aug 7 13:33:07 p8eval kernel: LustreError: 61858:0:(mgc_request.c:251:do_config_log_add())
<a href="mailto:MGC172.40.2.60@o2ib4">MGC172.40.2.60@o2ib4</a>: failed processing log, type 1: >rc = -5<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>Aug 7 13:33:26 p8eval kernel: Lustre: Server MGS version (2.5.42.8) is much older than client. Consider upgrading server (2.10.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>Aug 7 13:33:26 p8eval kernel: Lustre: Mounted lustre4-client<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Courier New";color:#1F497D">>Aug 7 13:33:31 p8eval kernel:
<b>Lustre: 61690:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent >1502130806/real 1502130806] req@c000000fdb620300 x1575098273235408/t0(0) o8-><a href="mailto:lustre4-OST0006-osc-c000001e3c4e9000@172.40.2.62@o2ib4:28/4">lustre4-OST0006-osc-c000001e3c4e9000@172.40.2.62@o2ib4:28/4</a>
lens >520/544 e 0 to 1 dl 1502130811 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1</b></span><span style="color:#1F497D"><br>
> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I know this bug. I can reproduce it on the Cray ARM machine. I haven’t fix it yet since I can’t reproduce this one on a x86 platform.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The ticket URL is <a href="https://jira.hpdd.intel.com/browse/LU-9823">
https://jira.hpdd.intel.com/browse/LU-9823</a>. I will look to collect some debug logs today on the ARM machine to<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">track it down.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">>If I attempt to build from the source .tar.gz with the following configure flags, the configure and make complete, however “make rpms” gets an error:<br>
><br>
</span><span style="font-family:"Courier New";color:#1F497D">>./configure --disable-tests --disable-server --with-linux=/usr/src/kernels/3.10.0-514.el7.ppc64le/ --with-o2ib=/usr/src/ofa_kernel/default -->target=ppc64le<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>+ basemodpath=/tmp/rpmbuild-lustre-root-dLLJioYr/BUILDROOT/lustre-2.10.0-1.ppc64le/lib/modules/3.10.0-514.el7.ppc64le/extra/lustre-client<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>+ :<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>+ echo /usr/lib/systemd/system/lnet.service<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>+ echo /etc/init.d/lsvcgss<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>+ find /tmp/rpmbuild-lustre-root-dLLJioYr/BUILDROOT/lustre-2.10.0-1.ppc64le -name '*.so' -type f -exec chmod +x '{}' ';'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>+ '[' -d /tmp/rpmbuild-lustre-root-dLLJioYr/BUILDROOT/lustre-2.10.0-1.ppc64le/usr/lib64/lustre/snmp ']'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>+ mkdir -p /tmp/rpmbuild-lustre-root-dLLJioYr/BUILDROOT/lustre-2.10.0-1.ppc64le//usr/share/lustre<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>+ find /tmp/rpmbuild-lustre-root-dLLJioYr/BUILDROOT/lustre-2.10.0-1.ppc64le/usr/lib64/lustre -name '*.la' -type f -exec rm -f '{}' ';'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>find: '/tmp/rpmbuild-lustre-root-dLLJioYr/BUILDROOT/lustre-2.10.0-1.ppc64le/usr/lib64/lustre': No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>error: Bad exit status from /tmp/rpmbuild-lustre-root-dLLJioYr/TMP/rpm-tmp.QuU8wT (%install)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>RPM build errors:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">> Bad exit status from /tmp/rpmbuild-lustre-root-dLLJioYr/TMP/rpm-tmp.QuU8wT (%install)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I know this one since I fixed it
</span><span style="font-family:Wingdings;color:#1F497D">J</span><span style="color:#1F497D"> That is LU-9758 and I have a patch for 2.10 already. Just waiting to land. You can get it here:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><a href="https://review.whamcloud.com/#/c/28372">https://review.whamcloud.com/#/c/28372</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><br>
></span>> Is lnet running over InfiniBand interfaces? If so, is it using the mlx4 or mlx5 driver<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">></span><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">>Yes, here’s the output of lustre.conf:<br>
><br>
</span><span style="font-family:"Courier New";color:#1F497D">>[root@p8eval modprobe.d]# cat lustre.conf
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">>options lnet networks=o2ib4(ib0)<o:p></o:p></span></p>
<p><span style="color:#1F497D">>> Also what OFED stack are you running? </span> <span style="color:#1F497D">Are you using the Lustre 2.10 or 2.8 client?<o:p></o:p></span></p>
<p><span style="color:#1F497D">>Mellanox OFED 3.4, and attempting to use 2.9 and 2.10, same errors for both versions. The version of lustre on the servers is 2.5.<o:p></o:p></span></p>
<p><span style="color:#1F497D">This one is a but more complicated. If by default the queue pair depth is too small. You will need to create the following file:<o:p></o:p></span></p>
<p><span style="color:#1F497D">/etc/modprobe.d/ib_mad.conf<o:p></o:p></span></p>
<div style="border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p><span style="color:#1F497D">With the following:<o:p></o:p></span></p>
</div>
<p><span style="color:#1F497D"># Module parameters for infiniband core to increase queue pair size<o:p></o:p></span></p>
<p><span style="color:#1F497D">options ib_mad send_queue_size=4096<o:p></o:p></span></p>
<div style="border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p><span style="color:#1F497D">options ib_mad recv_queue_size=4096<o:p></o:p></span></p>
</div>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Now for the LNet configurate. Please don’t use the modprobe config file lustre.conf. That is deprecated. You should be using lnetctl.<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">First you need a /etc/sysconfig/lnet.conf file. Something like this.<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-------------------------------------------------<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">net:<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> - net: o2ib6<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> status: up<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> interfaces:<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> 0: ib0<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> lnd tunables:<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> peercredits_hiw: 63<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> #map_on_demand: 16<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> concurrent_sends: 31<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> fmr_pool_size: 1280<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> fmr_flush_trigger: 1024<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> fmr_cache: 1<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> tunables:<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> peer_timeout: 180<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> peer_credits: 63<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> peer_buffer_credits: 0<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> credits: 2560<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">route:<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> - net: o2ib<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> gateway: <a href="mailto:10.39.232.10@o2ib6">10.39.232.10@o2ib6</a><o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> hop: 1<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> priority: 0<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> - net: o2ib<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> gateway: <a href="mailto:10.39.232.11@o2ib6">10.39.232.11@o2ib6</a><o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> hop: 1<o:p></o:p></span></pre>
<div style="border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> priority: 0<o:p></o:p></span></pre>
</div>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Once you have that file you need to run the following command:<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">modprobe lnet; lnetctl lnet configure; lnetctl import < /etc/sysconfig/lnet.conf<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Now if you have a mlx4 <-> mlx5 connection then you will have problems with page size difference between x86 and PPC. Is that the case for you? <o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Let me know because in that case it will take some more magic to get it working. Hope that helps. Hmmm. I need to create a wiki for this on lustre.org.<o:p></o:p></span></pre>
<pre><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></pre>
</div>
</body>
</html>