<div dir="ltr"><div>One possible cause of bandwidth limitation might come from PCIE slot limitation - HDR200 will require at least PCIE 4.0 16x lanes to work at maximum bandwidth, while many x86 servers give only 8x lanes to IO adapters.<br></div><div><br></div><div>Generally 100Gb bandwidth limitation will not be be problem, because most of OST configurations available today cannot endure such workloads. Yet when 200Gb throughput must be provided to OSS, there is an alternative such as dynamic load balancing on multi-rail configuration ( using 2 x 100Gb ports links for OSSes).</div><div> <br></div><div>regards,</div><div>Jongwoo Han</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">2021년 10월 19일 (화) 오후 7:31, 홍재기 via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>>님이 작성:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow:auto">
<p>Hi, all</p>
<p> </p>
<p>I am setting up Lustre on local cluster based on infiniband HDR(200Gb, single port) network.</p>
<p>I could successfuly setup Lustre using 5 servers(1-MDS, 4-OSS)</p>
<p> </p>
<p>Even though I have verified the IB HDR bandwidth(200Gb/s) with 'ib_read_bw' or 'ib_write_bw' tools, (I used CPU#0 for the test)</p>
<p>when I run LNet-Selftest between servers, <strong>it only shows around 100Gb/s(around 12GB/s, just half of maximum bandwidth)</strong></p>
<p>(~12GB/s in case of read, ~13GB/s for write test)</p>
<p> </p>
<p>so I tried to change LNet tunables and fixed the CPT: "[0]" for IB with the following kernel module options.</p>
<p>but it doesn't show big difference in lnet-self test.</p>
<p> </p>
<p>It seems like LNet is not fully compatible with HDR or PCIe gen4 interfaces.</p>
<p>Is there anyone who can give me advice why the LNet performance is not reaching HDR BW?</p>
<p>or, Are there specific options or tunables that I have to modify?</p>
<p>Please share your experience if you have setup Lustre with HDR network.</p>
<p> </p>
<p>Thank you.</p>
<p> </p>
<p>-----------lustre.conf---------------</p>
<p>options lnet networks=o2ib0(ib0)<strong>[0]</strong></p>
<p> </p>
<p>-----------ko2iblnd.conf-------------</p>
<p>options ko2iblnd peer_credits=256 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=0 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=1</p>
<p> </p>
<p>----------lnet tunables--------------</p>
<p>tunables:<br> peer_timeout: 180<br> peer_credits: 255<br> peer_buffer_credits: 0<br> credits: 1024<br> peercredits_hiw: 127<br> map_on_demand: 0<br> concurrent_sends: 256<br> fmr_pool_size: 2048<br> fmr_flush_trigger: 512<br> fmr_cache: 1<br> ntx: 2048<br> conns_per_peer: 1<br></p>
<p> </p>
<p>I listed up some HW/SW environment that I used.</p>
<p> </p>
<p>------------------------------------------------------------</p>
<p>[Environment]</p>
<p>- CPUs: Epyc 7302 *2 socket, supports PCIe Gen4 </p>
<p>- OS: CentOS 8.3 (kernel: 4.18.0-240.1.1.el8_lustre.x86_64)</p>
<p>- Lustre: 2.14.0 (downloaded from repository <a href="https://downloads.whamcloud.com/public/lustre/lustre-2.14.0-ib/" target="_blank">https://downloads.whamcloud.com/public/lustre/lustre-2.14.0-ib/</a> )</p>
<p>- OFED driver: tried 2 different versions MLNX_OFED_LINUX-5.2-1.0.4.0, MLNX_OFED_LINUX-5.4-1.0.3.0</p>
<p> </p>
<p>Finally, I used the following LNet selftest script for test.</p>
<p>I tried to change concurrency, but the bandwidth is saturated when CN>=4 </p>
<p> </p>
<p> </p>
<p>----------------------------------------------------------</p>
<p># Concurrency<br>CN=32<br>#Size<br>SZ=1M<br># Length of time to run test (secs)<br>TM=20<br># Which BRW test to run (read or write)<br>BRW=read<br># Checksum calculation (simple or full)<br>CKSUM=simple</p>
<p># The LST "from" list -- e.g. Lustre clients. Space separated list of NIDs.<br>LFROM="<a href="mailto:192.168.8.4@o2ib0" target="_blank">192.168.8.4@o2ib0</a>"<br>#LFROM=${LFROM:?ERROR: the LFROM variable is not set}<br># The LST "to" list -- e.g. Lustre servers. Space separated list of NIDs.<br>LTO="<a href="mailto:192.168.8.6@o2ib0" target="_blank">192.168.8.6@o2ib0</a>"<br>#LTO=${LTO:?ERROR: the LTO variable is not set}</p>
<p>### End of customisation.</p>
<p>export LST_SESSION=$$<br>echo LST_SESSION = ${LST_SESSION}<br>lst new_session lst${BRW}<br>lst add_group lfrom ${LFROM}<br>lst add_group lto ${LTO}<br>lst add_batch bulk_${BRW}<br>lst add_test --batch bulk_${BRW} --distribute 3:1 --from lfrom --to lto brw ${BRW} \<br> --concurrency=${CN} check=${CKSUM} size=${SZ}<br>lst run bulk_${BRW}<br>echo -n "Capturing statistics for ${TM} secs "<br>lst stat --mbs lfrom lto &<br>LSTPID=$!<br># Delay loop with interval markers displayed every 5 secs.<br># Test time is rounded up to the nearest 5 seconds.<br>i=1<br>j=$((${TM}/5))<br>if [ $((${TM}%5)) -ne 0 ]; then let j++; fi<br>while [ $i -le $j ]; do<br> sleep 5<br> let i++<br>done<br>kill ${LSTPID} && wait ${LISTPID} >/dev/null 2>&1<br>echo<br>lst show_error lfrom lto<br>lst stop bulk_${BRW}<br>lst end_session<br></p>
<p> </p><table id="gmail-m_-3486098207722552753bannersignimg"><tbody><tr><td><p> </p>
</td></tr></tbody></table><table id="gmail-m_-3486098207722552753confidentialsignimg"><tbody><tr><td><p><img style="border: 0px solid currentcolor; width: 520px; height: 144px; display: inline-block;" src="cid:17cc67c2745c204bfcc1"> </p>
</td></tr></tbody></table></div><table style="display:none"><tbody><tr><td><img style="display: none;" border="0" src="http://ext.samsung.net/mail/ext/v1/external/status/update?userid=jaeki.hong&do=bWFpbElEPTIwMjExMDE5MTAzMTE0ZXBjbXMycDNkMzQzYTg5MDMxOGM5YjM3N2RmY2U3ZTJlMjFjZDliNCZyZWNpcGllbnRBZGRyZXNzPWx1c3RyZS1kaXNjdXNzQGxpc3RzLmx1c3RyZS5vcmc_" width="0" height="0"></td></tr></tbody></table>_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><font face="'courier new', monospace">Jongwoo Han</font><div><font face="'courier new', monospace">+82-505-227-6108</font></div></div>