<HTML><HEAD>
<META content=IE=5 http-equiv=X-UA-Compatible>
<META content="text/html; charset=utf-8" http-equiv=Content-Type>
<STYLE id=mysingle_style type=text/css>.search-word {
BACKGROUND-COLOR: #ffee94
}
P {
FONT-SIZE: 10pt; MARGIN-BOTTOM: 5px; FONT-FAMILY: 맑은 고딕, arial; MARGIN-TOP: 5px
}
TD {
FONT-SIZE: 10pt; MARGIN-BOTTOM: 5px; FONT-FAMILY: 맑은 고딕, arial; MARGIN-TOP: 5px
}
LI {
FONT-SIZE: 10pt; MARGIN-BOTTOM: 5px; FONT-FAMILY: 맑은 고딕, arial; MARGIN-TOP: 5px
}
BODY {
FONT-SIZE: 10pt; FONT-FAMILY: 맑은 고딕, arial; MARGIN: 10px; LINE-HEIGHT: 1.4
}
</STYLE>
<STYLE class=cui-content-default data-cafe-default="true">BODY {
MARGIN: 10px; DISPLAY: block
}
BODY OL {
PADDING-LEFT: 40px; MARGIN: 0px
}
BODY UL {
PADDING-LEFT: 40px; MARGIN: 0px
}
BODY P {
MARGIN: 0px auto; LINE-HEIGHT: 1.9
}
BODY LI {
MARGIN: 0px auto; LINE-HEIGHT: 1.9
}
TABLE.cui-div {
WIDTH: 100%; DISPLAY: block
}
TABLE.cui-div > TBODY {
DISPLAY: block
}
TABLE.cui-div > TBODY > TR {
DISPLAY: block
}
TABLE.cui-div > TBODY > TR > TD {
DISPLAY: block
}
TABLE.cui-div > TBODY > TR > TH {
DISPLAY: block
}
TABLE.cui-pasted-table TH {
LINE-HEIGHT: normal
}
TABLE.cui-pasted-table TD {
LINE-HEIGHT: normal
}
TABLE.cui-pasted-table P {
LINE-HEIGHT: normal
}
TABLE.cui-pasted-table H1 {
LINE-HEIGHT: normal
}
TABLE.cui-pasted-table H2 {
LINE-HEIGHT: normal
}
TABLE.cui-pasted-table H3 {
LINE-HEIGHT: normal
}
TABLE.cui-pasted-table H4 {
LINE-HEIGHT: normal
}
TABLE.cui-pasted-table H5 {
LINE-HEIGHT: normal
}
TABLE.cui-pasted-table H6 {
LINE-HEIGHT: normal
}
TABLE.cui-pasted-table LI {
LINE-HEIGHT: normal
}
IMG[data-cui-alt-image] {
BACKGROUND: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABYAAAAUCAYAAACJfM0wAAABH0lEQVQ4jbXU26qEIBQGYF+wtxLMgg4QdGdBREUhFEFBBx/tn6sGpcPezTTCAi/0Q11rSQghhDGGJ4P8An3j20QIgSzLvookSfawUgrrukIp9VVcwkIIcM7h+z66rnsGbprGeCvf95+B8zw3YM75M/A4jnAcB3pS9Y3zPKPv+0vYKDf9jcdxRJ7naJrG2DRNE8IwBGMMZVkewmQbOnwVy7IgiqL3TWzbRtu2x+gVnKYpXNdF27ZY1xVxHO+awHVdDMMAKaWJnsFZlhkn8zzvtMOCIACl1No8Sqm1S55SCkVR3GrdI3QH13V9/7PRxmG5SSlh2/bHKGMMVVXtYc75bZRSaunX1w92+9vUT6mju3WfJuqvv/zf8FmZXq7/BfoCA1VRsvK4AfgAAAAASUVORK5CYII=) #c1c1c1 no-repeat center 50%
}
DIV[data-cui-alt-image] {
BACKGROUND: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABYAAAAUCAYAAACJfM0wAAABH0lEQVQ4jbXU26qEIBQGYF+wtxLMgg4QdGdBREUhFEFBBx/tn6sGpcPezTTCAi/0Q11rSQghhDGGJ4P8An3j20QIgSzLvookSfawUgrrukIp9VVcwkIIcM7h+z66rnsGbprGeCvf95+B8zw3YM75M/A4jnAcB3pS9Y3zPKPv+0vYKDf9jcdxRJ7naJrG2DRNE8IwBGMMZVkewmQbOnwVy7IgiqL3TWzbRtu2x+gVnKYpXNdF27ZY1xVxHO+awHVdDMMAKaWJnsFZlhkn8zzvtMOCIACl1No8Sqm1S55SCkVR3GrdI3QH13V9/7PRxmG5SSlh2/bHKGMMVVXtYc75bZRSaunX1w92+9vUT6mju3WfJuqvv/zf8FmZXq7/BfoCA1VRsvK4AfgAAAAASUVORK5CYII=) #c1c1c1 no-repeat center 50%
}
UNKNOWN {
BORDER-TOP-STYLE: none !important
}
UNKNOWN {
BORDER-TOP-STYLE: none !important
}
UNKNOWN {
}
</STYLE>
<STYLE class=cui-content-default data-user-config="true">BODY {
FONT-SIZE: 10pt; FONT-FAMILY: '맑은 고딕'; MARGIN: 10px; LINE-HEIGHT: 1.9
}
BODY {
FONT-FAMILY: '맑은 고딕'; LINE-HEIGHT: 1.9
}
BODY P {
FONT-FAMILY: '맑은 고딕'; LINE-HEIGHT: 1.9
}
BODY LI {
FONT-FAMILY: '맑은 고딕'; LINE-HEIGHT: 1.9
}
BODY H1 {
FONT-FAMILY: '맑은 고딕'; LINE-HEIGHT: 1.9
}
BODY H2 {
FONT-FAMILY: '맑은 고딕'; LINE-HEIGHT: 1.9
}
BODY H3 {
FONT-FAMILY: '맑은 고딕'; LINE-HEIGHT: 1.9
}
BODY H4 {
FONT-FAMILY: '맑은 고딕'; LINE-HEIGHT: 1.9
}
BODY H5 {
FONT-FAMILY: '맑은 고딕'; LINE-HEIGHT: 1.9
}
BODY H6 {
FONT-FAMILY: '맑은 고딕'; LINE-HEIGHT: 1.9
}
</STYLE>
<STYLE id=knox_style type=text/css>P {
FONT-SIZE: 10pt; MARGIN-BOTTOM: 5px; FONT-FAMILY: 맑은 고딕, arial; MARGIN-TOP: 5px
}
</STYLE>
<META name=GENERATOR content=ActiveSquare></HEAD>
<BODY style="OVERFLOW: auto">
<P>Hi, all</P>
<P> </P>
<P>I am setting up Lustre on local cluster based on infiniband HDR(200Gb, single port) network.</P>
<P>I could successfuly setup Lustre using 5 servers(1-MDS, 4-OSS)</P>
<P> </P>
<P>Even though I have verified the IB HDR bandwidth(200Gb/s) with 'ib_read_bw' or 'ib_write_bw' tools, (I used CPU#0 for the test)</P>
<P>when I run LNet-Selftest between servers, <STRONG>it only shows around 100Gb/s(around 12GB/s, just half of maximum bandwidth)</STRONG></P>
<P>(~12GB/s in case of read, ~13GB/s for write test)</P>
<P> </P>
<P>so I tried to change LNet tunables and fixed the CPT: "[0]" for IB with the following kernel module options.</P>
<P>but it doesn't show big difference in lnet-self test.</P>
<P> </P>
<P>It seems like LNet is not fully compatible with HDR or PCIe gen4 interfaces.</P>
<P>Is there anyone who can give me advice why the LNet performance is not reaching HDR BW?</P>
<P>or, Are there specific options or tunables that I have to modify?</P>
<P>Please share your experience if you have setup Lustre with HDR network.</P>
<P> </P>
<P>Thank you.</P>
<P> </P>
<P>-----------lustre.conf---------------</P>
<P>options lnet networks=o2ib0(ib0)<STRONG>[0]</STRONG></P>
<P> </P>
<P>-----------ko2iblnd.conf-------------</P>
<P>options ko2iblnd peer_credits=256 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=0 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=1</P>
<P> </P>
<P>----------lnet tunables--------------</P>
<P>tunables:<BR> peer_timeout: 180<BR> peer_credits: 255<BR> peer_buffer_credits: 0<BR> credits: 1024<BR> peercredits_hiw: 127<BR> map_on_demand: 0<BR> concurrent_sends: 256<BR> fmr_pool_size: 2048<BR> fmr_flush_trigger: 512<BR> fmr_cache: 1<BR> ntx: 2048<BR> conns_per_peer: 1<BR></P>
<P> </P>
<P>I listed up some HW/SW environment that I used.</P>
<P> </P>
<P>------------------------------------------------------------</P>
<P>[Environment]</P>
<P>- CPUs: Epyc 7302 *2 socket, supports PCIe Gen4 </P>
<P>- OS: CentOS 8.3 (kernel: 4.18.0-240.1.1.el8_lustre.x86_64)</P>
<P>- Lustre: 2.14.0 (downloaded from repository <A href="https://downloads.whamcloud.com/public/lustre/lustre-2.14.0-ib/">https://downloads.whamcloud.com/public/lustre/lustre-2.14.0-ib/</A> )</P>
<P>- OFED driver: tried 2 different versions MLNX_OFED_LINUX-5.2-1.0.4.0, MLNX_OFED_LINUX-5.4-1.0.3.0</P>
<P> </P>
<P>Finally, I used the following LNet selftest script for test.</P>
<P>I tried to change concurrency, but the bandwidth is saturated when CN>=4 </P>
<P> </P>
<P> </P>
<P>----------------------------------------------------------</P>
<P># Concurrency<BR>CN=32<BR>#Size<BR>SZ=1M<BR># Length of time to run test (secs)<BR>TM=20<BR># Which BRW test to run (read or write)<BR>BRW=read<BR># Checksum calculation (simple or full)<BR>CKSUM=simple</P>
<P># The LST "from" list -- e.g. Lustre clients. Space separated list of NIDs.<BR>LFROM="<A href="mailto:192.168.8.4@o2ib0">192.168.8.4@o2ib0</A>"<BR>#LFROM=${LFROM:?ERROR: the LFROM variable is not set}<BR># The LST "to" list -- e.g. Lustre servers. Space separated list of NIDs.<BR>LTO="<A href="mailto:192.168.8.6@o2ib0">192.168.8.6@o2ib0</A>"<BR>#LTO=${LTO:?ERROR: the LTO variable is not set}</P>
<P>### End of customisation.</P>
<P>export LST_SESSION=$$<BR>echo LST_SESSION = ${LST_SESSION}<BR>lst new_session lst${BRW}<BR>lst add_group lfrom ${LFROM}<BR>lst add_group lto ${LTO}<BR>lst add_batch bulk_${BRW}<BR>lst add_test --batch bulk_${BRW} --distribute 3:1 --from lfrom --to lto brw ${BRW} \<BR> --concurrency=${CN} check=${CKSUM} size=${SZ}<BR>lst run bulk_${BRW}<BR>echo -n "Capturing statistics for ${TM} secs "<BR>lst stat --mbs lfrom lto &<BR>LSTPID=$!<BR># Delay loop with interval markers displayed every 5 secs.<BR># Test time is rounded up to the nearest 5 seconds.<BR>i=1<BR>j=$((${TM}/5))<BR>if [ $((${TM}%5)) -ne 0 ]; then let j++; fi<BR>while [ $i -le $j ]; do<BR> sleep 5<BR> let i++<BR>done<BR>kill ${LSTPID} && wait ${LISTPID} >/dev/null 2>&1<BR>echo<BR>lst show_error lfrom lto<BR>lst stop bulk_${BRW}<BR>lst end_session<BR></P>
<P> </P><table id=bannersignimg data-cui-lock="true" namo_lock><tr><td><p> </p>
</td></tr></table><table id=confidentialsignimg data-cui-lock="true" namo_lock><tr><td><p><img style="border: 0px solid currentColor; border-image: none; width: 520px; height: 144px; display: inline-block;" unselectable="on" data-cui-image="true" src="cid:cafe_image_0@s-core.co.kr"> </p>
</td></tr></table></BODY></HTML><table style='display: none;'><tbody><tr><td><img style='display: none;' border=0 src='http://ext.samsung.net/mail/ext/v1/external/status/update?userid=jaeki.hong&do=bWFpbElEPTIwMjExMDE5MTAzMTE0ZXBjbXMycDNkMzQzYTg5MDMxOGM5YjM3N2RmY2U3ZTJlMjFjZDliNCZyZWNpcGllbnRBZGRyZXNzPWx1c3RyZS1kaXNjdXNzQGxpc3RzLmx1c3RyZS5vcmc_' width=0 height=0></td></tr></tbody></table>