<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>I was hoping to get some advise on how to write a valid lnet.conf
and also how we should rearchitect our current LNET layout. I
tried following the Lustre docs for lnet.conf but they were not
helpful and I ended up not being able to successfully set up a
LNET router programmatically. You can see my attempt to do so in
puppet here: </p>
<p><a class="moz-txt-link-freetext" href="https://github.com/fasrc/puppet-lnet">https://github.com/fasrc/puppet-lnet</a></p>
<p>I'm pretty sure I am missing something but I don't know what.</p>
<p>As for our current architecture it is as follows. Right now we
have two data centers separated by 100 km each with Lustre
filesystems in them and their own IB islands. To complicate
matters we will have a third IB island coming online soon as well,
so what we set up should be extensible. I want to code this in
Puppet so I can easily lay down new lnet.conf's and spin up new
LNET layers. Here are the systems in each place as well as the
Lustre versions.</p>
<p><ins>Boston</ins></p>
<p>boslfs: 5PB Lustre IEEL Filesystem, Lustre 2.5.34, IB only
export, routed via boslnet<span class="error">[01-02]</span> as
o2ib1</p>
<p>boslnet<span class="error">[01,02]</span>: Lustre 2.5.34 bridges
boslfs IB to our 10 GBE ethernet network</p>
<p><ins>Holyoke</ins></p>
<p>holylfs: 5PB Lustre IEEL Filesystem, Lustre 2.5.34, IB only
export, routed via holylnet<span class="error">[01-02]</span> as
o2ib0</p>
<p>holylfs02: 5PB Lustre Filesystem, Lustre 2.10.4, IB only export,
routed via holylnet<span class="error">[03-04]</span> as o2ib2</p>
<p>holylfs03: 3PD Lustre Filesystem, Lustre 2.10.6, IB only export,
routed via holylnet<span class="error">[01-02]</span> as o2ib0</p>
<p>scratchlfs: 2PB, DDN Exascaler, Lustre 2.10.5, IB only export,
routed via holylnet<span class="error">[01-02]</span> as o2ib0</p>
<p>holylnet<span class="error">[01-04]</span>: Lustre 2.5.34,
bridges FDR IB to GbE ethernet network</p>
<p>panlfs2, panlfs3, kuanglfs: Various Lustre 2.5.3, exported via IB
and GbE do not use LNET routers, mounted via o2ib or tcp0
(depending on if they are GbE or IB connected)</p>
<p>All these systems are exported to the compute which live in both
datacenters both on and off of the IB fabric. The compute is
running Lustre 2.10.3. As noted soon we will have an independent
IB fabric at Holyoke that is EDR/HDR that we want to bridge with
the FDR fabric using an LNET router as well.</p>
<p>What I would like to do is both bridge the Boston network from IB
to GbE and then back to IB (right now it just does IB to GbE),
make all the Lustre hosts at Holyoke that aren't dual homed use
the same block of LNET routers that we can expand easily and
programatically, finally lay the ground work for the LNET bridging
from FDR to EDR fabrics. It would also be good to use Lustre
2.10.x for the routers or whatever the latest version is so we an
use lnet.conf. I tried this on my own but I couldn't get a conf
that worked even though I thought I had one. I wasn't sure what I
was doing wrong.</p>
<p>If you like I can share with you our current configs but given
that I'm happy to throw them out I'm good with just scrapping all
I have to start from scratch and do it right. Also happy to send
a diagram if that would be helpful.</p>
<p>Thanks for your help in advance!</p>
<p>-Paul Edmon-<br>
</p>
</body>
</html>