<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Thank you, Peter. I figured that would be the response but wanted to ask. We were hoping to get away from maintaining a MOFED build but it looks like that may not be the way to go.<div class=""><br class=""></div><div class="">And you are correct about the JIRA ticket. I misspoke. It was the associated RH kernel bug that was \u201cprivate\u201d, IIRC. </div><div class=""><br class=""></div><div class="">Thank you again,</div><div class=""><br class=""></div><div class="">Charlie</div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Nov 29, 2017, at 8:09 AM, Jones, Peter A <<a href="mailto:peter.a.jones@intel.com" class="">peter.a.jones@intel.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif;" class="">
<div class="">
<div class="">Charles</div>
<div class=""><br class="">
</div>
<div class="">That ticket is completely open so you do have access to everything. As I understand it the options are to either use the latest MOFED update rather than relying on the in-kernel OFED (which I believe is the advise usually provided by Mellanox anyway) or
else apply the kernel patch Andreas has created that is referenced in the ticket.</div>
<div class=""><br class="">
</div>
<div class="">Peter</div>
<div class="">
<div id="MAC_OUTLOOK_SIGNATURE" class=""></div>
</div>
</div>
<div class=""><br class="">
</div>
<span id="OLK_SRC_BODY_SECTION" class="">
<div class="">
<div class="">On 2017-11-29, 2:50 AM, "lustre-discuss on behalf of Charles A Taylor" <<a href="mailto:lustre-discuss-bounces@lists.lustre.org" class="">lustre-discuss-bounces@lists.lustre.org</a> on behalf of
<a href="mailto:chasman@ufl.edu" class="">chasman@ufl.edu</a>> wrote:</div>
</div>
<div class=""><br class="">
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" class="" type="cite">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class="">
<div class="">Hi All,</div>
<div class=""><br class="">
</div>
We recently upgraded from Lustre 2.5.3.90 on EL6 to 2.10.1 on EL7 (details below) but have hit what looks like LU-10133 (order 8 page allocation failures).<br class="">
<br class="">
<div class="">
<div class=""><span style="background-color: rgb(255, 255, 0);" class="">We don\u2019t have access to look at the JIRA ticket in more detail</span> but from what we can tell the the fix is to change from vmalloc() to vmalloc_array() in the mlx4 drivers. However, the vmalloc_array()
infrastructure is in an upstream (far upstream) kernel so I\u2019m not sure when we\u2019ll see that fix.</div>
</div>
<div class=""><br class="">
</div>
<div class="">While this may not be a Lustre issue directly, I know we can\u2019t be the only Lustre site running 2.10.1 over IB on Mellanox ConnectX-3 HCAs. So far we have tried increasing vm.min_free_kbytes to 8GB but that does not help. Zone_reclaim_mode is
disabled (for other reasons that may not be valid under EL7) but order 8 chunks get depleted on both NUMA nodes so I\u2019m not sure that is the answer either (though we have not tried it yet).</div>
<div class=""><br class="">
</div>
<div class="">[root@ufrcmds1 ~]# cat /proc/buddyinfo <br class="">
<font face="Courier" class="">Node 0, zone DMA 1 0 0 0 2 1 1 0 1 1 3 <br class="">
Node 0, zone DMA32 1554 13496 11481 5108 150 0 0 0 0 0 0 <br class="">
Node 0, zone Normal 114119 208080 78468 35679 6215 690 0 0 0 0 0 <br class="">
Node 1, zone Normal 81295 184795 106942 38818 4485 293 1653 0 0 0 0 </font><br class="">
<br class="">
</div>
<div class="">I\u2019m wondering if other sites are hitting this and, if so, what are you doing to work around the issue on your OSSs. </div>
<div class=""><br class="">
</div>
<div class="">Regards,</div>
<div class=""><br class="">
</div>
<div class="">Charles Taylor</div>
<div class="">UF Research Computing</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Some Details:</div>
<div class="">-------------------</div>
<div class="">OS: RHEL 7.4 (Linux ufrcoss28.ufhpc 3.10.0-693.2.2.el7_lustre.x86_64)<br class="">
Lustre: 2.10.1 (lustre-2.10.1-1.el7.x86_64)<br class="">
Clients: ~1400 (still running 2.5.3.90 but we are in the process of upgrading)<br class="">
Servers: 10 HA OSS pairs (20 OSSs)<br class="">
128 GB RAM</div>
<div class=""> 6 OSTs (8+2 RAID-6) per OSS <br class="">
Mellanox ConnectX-3 IB/VPI HCAs <br class="">
RedHat Native IB Stack (i.e. not MOFED)<br class="">
mlx4_core driver:<br class="">
filename: /lib/modules/3.10.0-693.2.2.el7_lustre.x86_64/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko.xz<br class="">
version: 2.2-1<br class="">
license: Dual BSD/GPL<br class="">
description: Mellanox ConnectX HCA low-level driver<br class="">
author: Roland Dreier<br class="">
rhelversion: 7.4</div>
</div>
</div>
</div>
</blockquote>
</span>
</div>
</div></blockquote></div><br class=""></div></body></html>