<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Right, when you format a Lustre target, it registers itself with the MGS. Part of that registration is telling the MGS what NIDs the target can be reached at (the MGS, in turn, passes this information to the clients). If you add or delete
NIDs then you need to ensure that information is updated with the MGS. This is the procedure I linked in the Ops manual.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">lctl list_nids does not tell you which NIDs are registered with the MGS. It only tells you what NIDs are currently defined on the local host. There is some way to inspect the config log to see what NIDs are in there, but I can’t recall
the specifics off the top of my head.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Chris Horn<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Laura Hild via lustre-discuss <lustre-discuss@lists.lustre.org><br>
<b>Date: </b>Thursday, November 30, 2023 at 8:22 AM<br>
<b>To: </b>Philipp Grau <phgrau@zedat.fu-berlin.de><br>
<b>Cc: </b>Lustre User Discussion Mailing List <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] Lustre mds/ods Server with IB/omnipath and Ethernet clients (dual homed?)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Hi Philipp-<br>
<br>
I don't do this a ton so I'm hazy, but do you set nids or nets when you mkfs.lustre? So then maybe you have to tunefs those in when you add more?<br>
<br>
-Laura<br>
<br>
<br>
________________________________________<br>
Od: lustre-discuss <lustre-discuss-bounces@lists.lustre.org> v imenu Philipp Grau <phgrau@zedat.fu-berlin.de><br>
Poslano: sreda, 29. november 2023 06:37<br>
Za: lustre-discuss@lists.lustre.org<br>
Zadeva: [lustre-discuss] Lustre mds/ods Server with IB/omnipath and Ethernet clients (dual homed?)<br>
<br>
Hello,<br>
<br>
some questions regarding network connection setup for ethernet based<br>
clients.<br>
<br>
We have a working Luste installation with two MDS servers and seven<br>
ODS systems connected to our cluster via omnipath/ib. This part is<br>
working fine.<br>
<br>
Now we want to add some clients that have only a ethernet connection<br>
to the Lustre servers (with the ethernet cards in the servers).<br>
<br>
Our MDS and ODS servers have the following lnet setup:<br>
<br>
net:<br>
- net type: lo<br>
local NI(s):<br>
- nid: 0@lo<br>
status: up<br>
- net type: o2ib<br>
local NI(s):<br>
- nid: 10.149.0.XXX@o2ib # IP of the local ib interface<br>
status: up<br>
interfaces:<br>
0: ib0<br>
- net type: tcp<br>
local NI(s):<br>
- nid: xxx.xxx.5.XXX@tcp # IP of the local ethernet interface<br>
status: up<br>
interfaces:<br>
0: eno1<br>
<br>
<br>
Our test ethernet node:<br>
<br>
lnetctl net show<br>
net:<br>
- net type: lo<br>
local NI(s):<br>
- nid: 0@lo<br>
status: up<br>
- net type: tcp<br>
local NI(s):<br>
- nid: xxx.xxx.4.XXX@tcp # same subnet as above, it is a /23<br>
status: up<br>
interfaces:<br>
0: enp225s0f0<br>
<br>
So far so good.<br>
<br>
I'm able to lnetctl ping in both directions:<br>
<br>
Ping the client:<br>
<br>
lnetctl ping xxx.xxx.4.xxx@tcp<br>
ping:<br>
- primary nid: xxx.xxx.4.xxx@tcp<br>
Multi-Rail: True<br>
peer ni:<br>
- nid: xxx.xxx.4.xxx@tcp<br>
<br>
Ping the server:<br>
<br>
lnetctl ping xxx.xxx.5.xxx@tcp<br>
ping:<br>
- primary nid: xxx.xxx.5.xxx@tcp<br>
Multi-Rail: True<br>
peer ni:<br>
- nid: 10.149.0.183@o2ib<br>
- nid: xxx.xxx.5.xxx@tcp<br>
<br>
But the mount fails, output from dmesg (are there other sources of<br>
debug information?):<br>
<br>
LustreError: 25758:0:(ldlm_lib.c:494:client_obd_setup()) can't add initial connection<br>
LustreError: 25758:0:(obd_config.c:559:class_setup()) setup scratch-MDT0000-mdc-ffff8b63003d4000 failed (-2)<br>
LustreError: 25758:0:(obd_config.c:1835:class_config_llog_handler()) MGCxxx.xxx.5.xxx@tcp: cfg command failed: rc = -2<br>
Lustre: cmd=cf003 0:scratch-MDT0000-mdc 1:scratch-MDT0000_UUID 2:10.149.0.183@o2ib<br>
LustreError: 15c-8: MGC160.45.5.246@tcp: The configuration from log 'scratch-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br>
LustreError: 25734:0:(obd_config.c:610:class_cleanup()) Device 3 not setup<br>
Lustre: Unmounted scratch-client<br>
LustreError: 25734:0:(obd_mount.c:1604:lustre_fill_super()) Unable to mount (-2)<br>
<br>
Does some one have some ideas or reference documentation on this topic?<br>
<br>
Do I need some "lnetctl route" stuff?<br>
<br>
Do I need some "lnetctl peer add ..." to make the Lustre servers and<br>
clients known to each other?<br>
<br>
Any hints are welcome!<br>
<br>
Kind regards,<br>
<br>
Philipp<br>
<br>
--<br>
Philipp Grau | Freie Universitaet Berlin<br>
phgrau@ZEDAT.FU-Berlin.DE | FU-IT - Infrastruktur<br>
Tel: +49 (30) 838 56583 | Fabeckstr. 32<br>
Fax: +49 (30) 838 56721 | 14195 Berlin<br>
<br>
_______________________________________________<br>
lustre-discuss mailing list<br>
lustre-discuss@lists.lustre.org<br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
<o:p></o:p></p>
</div>
</div>
</body>
</html>