<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 5, 2024, at 11:37, Michael DiDomenico via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">i could use a little help with lustre clients over omni path. when i<br class="">
run ib_write_bw tests between two compute nodes i get +10GB/sec.<br class="">
compute nodes are rhel9.4 with rhel hw drivers<br class="">
<br class="">
however, when i run lnet_selftest between the same two compute nodes<br class="">
<br class="">
1m i/o size<br class="">
16 concurrency<br class="">
<br class="">
node1-node3<br class="">
read 1m i/o ~7.1GB/sec<br class="">
write 1m i/o ~4.7GB/sec<br class="">
<br class="">
node3-node1<br class="">
read 1m i/o ~6.6GB/sec<br class="">
write 1m i/o ~4.9GB/sec<br class="">
<br class="">
varying the i/o size and concurrency changes the numbers, but not<br class="">
dramatically. i've gone through the tuning guide for omnipath and my<br class="">
lnd tunables all match, but i can't seem to drive the bandwidth any<br class="">
higher between nodes.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Please provide the actual tuning parameters in use.</div>
<div><br class="">
</div>
Even when we were part of Intel, the OPA tuning parameters suggested by</div>
<div>the OPA team were not necessarily the best in all cases. There was some</div>
<div>kind of memory registration they kept suggesting, but it was always worse</div>
<div>in practice than in theory.</div>
<div><br class="">
</div>
<div>The biggest win was from setting conns_per_peer=4 or so, because OPA</div>
<div>needs more CPU resources for good performance than IB.</div>
<div><br class="">
</div>
<div>That said, it has been several years since I've had to deal with it, so I can't</div>
<div>say if your current performance is good or bad..</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">can anyone suggest where i might be dropping some performance or is<br class="">
this the end? i feel like there should be more performance here, but<br class="">
since we recently retooled from rhel7 to rhel9, i'm unsure if there's<br class="">
a tunable not tuned. (unfortunately i don't have/can't seem to find<br class="">
previous numbers to compare)<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>