<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mv="http://macVmlSchemaUri" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman";}
span.EmailStyle18
{mso-style-type:personal;
font-family:Arial;
color:#000040;}
span.EmailStyle19
{mso-style-type:personal-compose;
font-family:Arial;
color:#000040;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1027"/>
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1"/>
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">I am not sure about the checksums value: I see the same behaviour on my system. It may be a failsafe against permanently disabling checksums, since
there is a risk of data corruption.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">For max_pages_per_rpc, setting the RPC size larger than 1MB (256 pages) is only available in Lustre versions 2.9.0 and newer, or in Intel’s EE Lustre
version 3.1. Also, to make this work, one must also adjust the brw_size parameter on the OSTs to match the RPC size. The Lustre manual provides documentation on the feature:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><a href="https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#idm139670075738896">https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#idm139670075738896</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">Here are some rough notes I have to hand:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">To check the current settings of the brw_size attribute, log into each OSS and run the following command:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">lctl get_param obdfilter.*.brw_size<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">For example:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">[root@ct66-oss1 ~]# lctl get_param obdfilter.*.brw_size<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">obdfilter.demo-OST0000.brw_size=1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">obdfilter.demo-OST0004.brw_size=1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">The value returned is measured in MB.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">To change the setting temporarily on an OSS server:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">lctl set_param obdfilter.*.brw_size=<n><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">where <n> is an integer value between 1 and 16. Again, the value is a measurement in MB. To set brw_size persistently, login to the MGS and as root
use the following syntax:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">lctl set_param -P obdfilter.*.brw_size=<n><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">This will set the value for all OSTs across all file systems registered with the MGS. To scope the settings to an individual file system, change the
filter expression to include the file system name:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">lctl set_param -P obdfilter.<fsname>-*.brw_size=<n><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">To temporarily change the value of max_pages_per_rpc, use the following command on each client:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">lctl set_param osc.*.max_pages_per_rpc=<n><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">for example, to set max_pages_per_rpc to 1024 (4M):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">lctl set_param osc.*.max_pages_per_rpc=1024<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">To make the setting persistent, log into the MGS server and run the lctl set_param command using the -P flag:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">lctl set_param -P osc.*.max_pages_per_rpc=<n><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">Again, the scope can be reined by changing the pattern to match the file system name:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">lctl set_param -P osc.<fsname>-*.max_pages_per_rpc=<n><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">For example:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">lctl set_param -P osc.demo-*.max_pages_per_rpc=1024<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US">Note that I have found that if the brw_size is changed you may have to re-mount the clients before you’ll be able to set max_pages_per_rpc > 256.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:Arial;color:#0C5BB8">Malcolm Cowe</span><span lang="EN-US" style="font-size:10.0pt;font-family:Arial;color:#00003F"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:Arial;color:#000951">High Performance Data Division</span><span lang="EN-US" style="font-size:9.0pt;font-family:Arial;color:#00003F"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:Arial;color:#00003F"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:9.0pt;font-family:Arial;color:#000951">Intel Corporation</span></b><span lang="EN-US" style="font-size:9.0pt;font-family:Arial;color:#000951"> | www.intel.com</span><span lang="EN-US" style="font-size:9.0pt;font-family:Arial;color:#00003F"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Arial;color:#000040;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="font-family:Calibri;color:black">From:
</span></b><span style="font-family:Calibri;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Reinoud Bokhorst <rbokhorst@astron.nl><br>
<b>Date: </b>Friday, 7 April 2017 at 1:31 am<br>
<b>To: </b>Lustre discussion <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>[lustre-discuss] permanent configuration: "set_param -P" or "conf_param"<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<p style="margin-left:36.0pt"><span style="font-size:10.0pt;mso-fareast-language:EN-US">Hi all,<br>
Two days ago I made the following Lustre configuration changes:<br>
<br>
lctl set_param -P osc.*.checksums=0<br>
lctl set_param -P osc.*.max_pages_per_rpc=512<br>
lctl set_param -P osc.*.max_rpcs_in_flight=32<br>
lctl set_param -P osc.*.max_dirty_mb=128<br>
<br>
I ran these commands on the MGS. The -P flag promised to make a permanent change and doing this on the MGS would make it system-wide. Indeed directly after running the commands, I noticed that the settings were nicely propagated to other nodes.
<br>
<br>
When I look now, only "max_rpcs_in_flight" and "max_dirty_mb" still have those values, the others are back to their defaults, namely checksums=1 and max_pages_per_rpc=256. The compute nodes have been rebooted in the mean time.<br>
<br>
Two questions:<br>
- Why were the settings of checksums and max_pages_per_rpc lost? (I suspect during the reboot)<br>
- What is the proper way to make these changes permanent? Should I use "lctl conf_param"?<br>
<br>
Our lustre version:<br>
<br>
# lctl get_param version<br>
version=<br>
lustre: 2.7.0<br>
kernel: patchless_client<br>
build: 2.7.0-RC4--PRISTINE-3.10.0-327.36.3.el7.x86_64<br>
<br>
Thanks,<br>
Reinoud Bokhorst</span><span style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
</div>
</body>
</html>