<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
code
{mso-style-priority:99;
font-family:"Courier New";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:2124879125;
mso-list-template-ids:-1;}
@list l0:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p>Hi all,<br>
<br>
I am testing a Lustre system that includes 1 MGS, 2 MDS, and 8 OSS with 8 OSTs running RAID 6 (8d+2p). Each OST's performance is approximately 16 GB/s for WRITE and 33 GB/s for READ (measured with FIO test: blocksize=1m, iodepth=64, numjob=2 sequential test).
The system has 16 clients.<o:p></o:p></p>
<p>I am encountering issues with performance testing using IOR with the following options:<o:p></o:p></p>
<p class="MsoNormal">```<o:p></o:p></p>
<pre><code>mpirun --allow-run-as-root --mca pml ucx -x UCX_TLS=rc_mlx5,ud_mlx5,self -x UCX_NET_DEVICES=mlx5_0:1 --mca btl ^openib --hostfile mphost10 -np <number_of_process> -map-by node ior -w -r -b 2m -t 2m -C -s 4000 -k -e -o /lustre/testFS/ior/iortest</code><o:p></o:p></pre>
<p class="MsoNormal">```<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p>The stripe_count is set equal to the number of processes (overstriping), and the stripe_size is equal to the block size (2m). The issues I am facing are:<o:p></o:p></p>
<ol start="1" type="1">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Performance does not increase beyond 2 processes per client. With 1 client and 1 OST, I achieve approximately 2 GB/s for WRITE. With 2 clients and 4 processes, I achieve 4 GB/s. To reach 16 GB/s, I need to use 16 clients with 2 processes per client.<o:p></o:p></li></ol>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="0" style="width:0in;border-collapse:collapse">
<tbody>
<tr style="height:15.75pt">
<td valign="bottom" style="border:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Stripe count<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border:solid #CCCCCC 1.0pt;border-left:none;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">NP<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border:solid #CCCCCC 1.0pt;border-left:none;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Write (MB/s)<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border:solid #CCCCCC 1.0pt;border-left:none;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Read (MB/s)<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.75pt">
<td valign="bottom" style="border:solid #CCCCCC 1.0pt;border-top:none;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1843.57<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1618.57<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.75pt">
<td valign="bottom" style="border:solid #CCCCCC 1.0pt;border-top:none;background:white;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">1</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;background:white;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">2</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;background:white;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">2079.28</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1914.32<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.75pt">
<td valign="bottom" style="border:solid #CCCCCC 1.0pt;border-top:none;background:white;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">2</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;background:white;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">2</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;background:white;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">2579.28</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">2298.19<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.75pt">
<td valign="bottom" style="border:solid #CCCCCC 1.0pt;border-top:none;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">2<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">4<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1337.38<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1310.23<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.75pt">
<td valign="bottom" style="border:solid #CCCCCC 1.0pt;border-top:none;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">16<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">16<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1313.24<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1345.24<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.75pt">
<td valign="bottom" style="border:solid #CCCCCC 1.0pt;border-top:none;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">16<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">32<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1455.45<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1398.23<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.75pt">
<td valign="bottom" style="border:solid #CCCCCC 1.0pt;border-top:none;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">32<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">32<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1477.75<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1410.68<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.75pt">
<td valign="bottom" style="border:solid #CCCCCC 1.0pt;border-top:none;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">800<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">32<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1326.41<o:p></o:p></span></p>
</td>
<td valign="bottom" style="border-top:none;border-left:none;border-bottom:solid #CCCCCC 1.0pt;border-right:solid #CCCCCC 1.0pt;padding:1.5pt 2.25pt 1.5pt 2.25pt;height:15.75pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">1210.13<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><o:p> </o:p></p>
<ol start="2" type="1">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Performance does not improve by adding more OSTs. With 2 OSTs and 2 clients, the performance remains at 4 GB/s, and with 16 clients, the performance is only equivalent to 1 OST.<o:p></o:p></li></ol>
<p>I am wondering why the performance does not scale after 2 processes per client. Could it be that overstriping alone is not sufficient to enhance performance for Single Shared File mode? Are there any additional settings I should consider configuring beyond
overstriping? <br>
<br>
The results of obdfilter-survey and lnet do not show any bottleneck. <br>
<br>
I am using Lustre 2.15.4 with Rocky Linux 8.9 and kernel 4.18.0-513.9.1.el8_lustre.x86_64.<br>
- Information of MGS/MDS/OSS: 16 CPUs, 32 GB RAM. <br>
- Information of Clients: AMD EPYC 7662 64 core x2, 512 GB RAM.<br>
The network connection is InfiniBand with 400 Gbps bandwidth.<o:p></o:p></p>
<p>Other settings on the Lustre cluster:<o:p></o:p></p>
<p>```<o:p></o:p></p>
<p># Clients:<o:p></o:p></p>
<p>options lnet networks="o2ib(ib0)"<o:p></o:p></p>
<p>options ko2iblnd peer_credits=32 peer_credits_hiw=16 credits=256 concurrent_sends=64<o:p></o:p></p>
<p>lctl set_param osc.*.max_pages_per_rpc=4096<o:p></o:p></p>
<p>lctl set_param osc.*.checksums=0<o:p></o:p></p>
<p>lctl set_param osc.*.max_rpcs_in_flight=16<o:p></o:p></p>
<p><o:p> </o:p></p>
<p># OSSs:<br>
options lnet networks="o2ib(ib0)"<o:p></o:p></p>
<p>options libcfs cpu_npartitions=1<o:p></o:p></p>
<p>options ko2iblnd peer_credits=32 peer_credits_hiw=16 credits=256 concurrent_sends=64 nscheds=8<o:p></o:p></p>
<p>options ost oss_num_threads=128<o:p></o:p></p>
<p>lctl set_param *.*.brw_size=16<o:p></o:p></p>
<p>lctl set_param osd-ldiskfs.*.writethrough_cache_enable=0<o:p></o:p></p>
<p>lctl set_param osd-ldiskfs.*.read_cache_enable=0<o:p></o:p></p>
<p><br>
# MGS – MDSs<o:p></o:p></p>
<p>options lnet networks="o2ib(ib0)"<o:p></o:p></p>
<p>options libcfs cpu_npartitions=1<o:p></o:p></p>
<p>options ko2iblnd peer_credits=32 peer_credits_hiw=16 credits=256 concurrent_sends=64 nscheds=8<o:p></o:p></p>
<p>```<o:p></o:p></p>
<p class="MsoNormal">Thank you for your helping.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>