<div dir="ltr">Hi All,<div><br></div><div> We are gradually updating a cluster (OS, etc.) in-place, basically switching blocks of nodes from the old head node to the new. Until we can re-arrange the fabric at the next scheduled machine room power shutdown event, we are running two independent Infiniband subnets. As I can't find useful documentation on proper IB routing between subnets, I have configured one node with an HCA on each IB subnet that does simple IPoIB routing and LNET routing.</div><div><br></div><div>Brief description:</div><div> Router node has 24 cores, 128GB RAM, and is running with the in-kernel IB drivers from Centos7.3. It connects to the new IB fabric via a Mellanox EDR card (MT4115) on ib0, and to the old via a Truescale QDR card (QLE7340) on ib1. The old IB is on <a href="http://10.2.0.0/16">10.2.0.0/16</a> (o2ib0), and the new is <a href="http://10.201.32.0/19">10.201.32.0/19</a> (o2ib1).</div><div><br></div><div> The new 2.10.0 server is on the EDR side, and the old 2.8.0 server is on the QDR side. Nodes with QDR HCAs already coexist with EDR nodes on the EDR subnet without problems.</div><div><br></div><div>All Lustre config via /etc/lnet.conf:</div><div>#####</div><div><div><font face="monospace, monospace">net:</font></div><div><font face="monospace, monospace"> - net type: o2ib1</font></div><div><font face="monospace, monospace"> local NI(s):</font></div><div><font face="monospace, monospace"> - nid: 10.201.32.11@o2ib1</font></div><div><font face="monospace, monospace"> interfaces:</font></div><div><font face="monospace, monospace"> 0: ib0</font></div><div><font face="monospace, monospace"> tunables:</font></div><div><font face="monospace, monospace"> peer_timeout: 180</font></div><div><font face="monospace, monospace"> peer_credits: 62</font></div><div><font face="monospace, monospace"> peer_buffer_credits: 512</font></div><div><font face="monospace, monospace"> credits: 1024</font></div><div><font face="monospace, monospace"> lnd tunables:</font></div><div><font face="monospace, monospace"> peercredits_hiw: 64</font></div><div><font face="monospace, monospace"> map_on_demand: 256</font></div><div><font face="monospace, monospace"> concurrent_sends: 62</font></div><div><font face="monospace, monospace"> fmr_pool_size: 2048</font></div><div><font face="monospace, monospace"> fmr_flush_trigger: 512</font></div><div><font face="monospace, monospace"> fmr_cache: 1</font></div><div><font face="monospace, monospace"> ntx: 2048</font></div><div><span style="font-family:monospace,monospace"> - net type: o2ib0</span><br></div><div><font face="monospace, monospace"> local NI(s):</font></div><div><font face="monospace, monospace"> - nid: 10.2.1.22@o2ib0</font></div><div><font face="monospace, monospace"> interfaces:</font></div><div><font face="monospace, monospace"> 0: ib1</font></div><div><font face="monospace, monospace"> tunables:</font></div><div><font face="monospace, monospace"> peer_timeout: 180</font></div><div><font face="monospace, monospace"> peer_credits: 8</font></div><div><font face="monospace, monospace"> peer_buffer_credits: 512</font></div><div><font face="monospace, monospace"> credits: 1024</font></div><div><font face="monospace, monospace"> lnd tunables:</font></div><div><font face="monospace, monospace"> </font><span style="font-family:monospace,monospace"> map_on_demand: 32</span></div><div><font face="monospace, monospace"> concurrent_sends: 16</font></div><div><font face="monospace, monospace"> fmr_pool_size: 2048</font></div><div><font face="monospace, monospace"> fmr_flush_trigger: 512</font></div><div><font face="monospace, monospace"> fmr_cache: 1</font></div><div><font face="monospace, monospace"> ntx: 2048</font></div><div><font face="monospace, monospace">routing:</font></div><div><font face="monospace, monospace"> - small: 16384</font></div><div><font face="monospace, monospace"> large: 2048</font></div><div><font face="monospace, monospace"> enable: 1</font></div></div><div>####</div><div><br></div><div> While the setup works, I had to drop peer_credits to 8 on the QDR side to avoid long periods of stalled traffic. It is probably going to be adequate for the remaining month before total shutdown and removal of routers, but I would still like to have a better solution in hand.</div><div><br></div><div>Questions:</div><div>1) Is there a well-known good config for a qib<-->mlx5 LNET router?</div><div>2) Where should I look to identify the cause of stalled traffic, which still appears at higher load?</div><div>3) What parameters should I be playing with to optimize the router?</div><div><br></div><div>Thanks,</div><div>Nate </div><div><br></div><div><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><pre>Dr. Nathan Crawford <a href="mailto:nathan.crawford@uci.edu" target="_blank">nathan.crawford@uci.edu</a>
Modeling Facility Director
Department of Chemistry
1102 Natural Sciences II Office: 2101 Natural Sciences II
University of California, Irvine Phone: 949-824-4508
Irvine, CA 92697-2025, USA</pre></div></div>
</div></div>