[lustre-discuss] Need help with Creating Single Unified LNET yaml config
S, Praveen Kumar
praveen-kumar.s4 at hpe.com
Sat Jul 18 06:32:19 UTC 2026
I haven't got chance to check with master. I'll check with master asap and will keep you posted.
Thanks.
Praveen Kumar S
Systems Software Engineer
ClusterStor/Lustre, HPC & AI
praveen-kumar.s4 at hpe.com
Hewlett Packard Enterprise
HPE.com
[cid:d30d9b93-d47f-449f-8e8c-fe9944211f24]
________________________________
From: Chris Horn <chorn at ddn.com>
Sent: Tuesday, July 14, 2026 7:25 PM
To: S, Praveen Kumar <praveen-kumar.s4 at hpe.com>; lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
Subject: Re: Need help with Creating Single Unified LNET yaml config
Does this issue reproduce with master or just HPE Lustre?
Chris
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of S, Praveen Kumar via lustre-discuss <lustre-discuss at lists.lustre.org>
Date: Monday, July 13, 2026 at 6:15 AM
To: lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
Subject: [lustre-discuss] Need help with Creating Single Unified LNET yaml config
Greetings Everyone,
I'm trying to config LNET on Servers via unified yaml and I'm facing problem with configuring routes: See below
My expected config:
[root at kjcf05n00 pravs]# pdsh -g lustre 'lctl list_nids' | dshbak -c
----------------
kjcf05n02
----------------
10.16.100.52 at o2ib
10.16.100.58 at o2ib
10.16.100.52 at o2ib10
10.16.100.58 at o2ib10
----------------
kjcf05n03
----------------
10.16.100.53 at o2ib
10.16.100.59 at o2ib
10.16.100.53 at o2ib11
10.16.100.59 at o2ib11
----------------
kjcf05n04
----------------
10.16.100.54 at o2ib
10.16.100.54 at o2ib10
----------------
kjcf05n05
----------------
10.16.100.55 at o2ib
10.16.100.55 at o2ib11
----------------
kjcf05n06
----------------
10.16.100.56 at o2ib
10.16.100.60 at o2ib
10.16.100.56 at o2ib10
10.16.100.60 at o2ib10
----------------
kjcf05n07
----------------
10.16.100.57 at o2ib
10.16.100.61 at o2ib
10.16.100.57 at o2ib11
10.16.100.61 at o2ib11
----------------
kjcf05n08
----------------
10.16.100.62 at o2ib
10.16.100.63 at o2ib
10.16.100.62 at o2ib10
10.16.100.63 at o2ib10
----------------
kjcf05n09
----------------
10.16.100.64 at o2ib
10.16.100.65 at o2ib
10.16.100.64 at o2ib11
10.16.100.65 at o2ib11
[root at kjcf05n00 admin]# pdsh -g lustre 'lctl get_param routes' | dshbak -c
----------------
kjcf05n02
----------------
routes=
Routing Disabled
net hops priority state router
gni -1 0 up 10.16.100.10 at o2ib10
gni -1 0 up 10.16.100.18 at o2ib10
gni -1 0 up 10.16.100.20 at o2ib10
gni -1 0 up 10.16.100.16 at o2ib10
gni -1 0 up 10.16.100.12 at o2ib10
gni -1 0 up 10.16.100.14 at o2ib10
----------------
kjcf05n03
----------------
routes=
Routing Disabled
net hops priority state router
gni -1 0 up 10.16.100.11 at o2ib11
gni -1 0 up 10.16.100.17 at o2ib11
gni -1 0 up 10.16.100.15 at o2ib11
gni -1 0 up 10.16.100.19 at o2ib11
gni -1 0 up 10.16.100.13 at o2ib11
gni -1 0 up 10.16.100.21 at o2ib11
----------------
kjcf05n04
----------------
routes=
Routing Disabled
net hops priority state router
gni -1 0 up 10.16.100.16 at o2ib10
gni -1 0 up 10.16.100.10 at o2ib10
gni -1 0 up 10.16.100.18 at o2ib10
gni -1 0 up 10.16.100.12 at o2ib10
gni -1 0 up 10.16.100.14 at o2ib10
gni -1 0 up 10.16.100.20 at o2ib10
----------------
kjcf05n05
----------------
routes=
Routing Disabled
net hops priority state router
gni -1 0 up 10.16.100.19 at o2ib11
gni -1 0 up 10.16.100.13 at o2ib11
gni -1 0 up 10.16.100.21 at o2ib11
gni -1 0 up 10.16.100.17 at o2ib11
gni -1 0 up 10.16.100.15 at o2ib11
gni -1 0 up 10.16.100.11 at o2ib11
----------------
kjcf05n06
----------------
routes=
Routing Disabled
net hops priority state router
gni -1 0 up 10.16.100.12 at o2ib10
gni -1 0 up 10.16.100.10 at o2ib10
gni -1 0 up 10.16.100.18 at o2ib10
gni -1 0 up 10.16.100.14 at o2ib10
gni -1 0 up 10.16.100.16 at o2ib10
gni -1 0 up 10.16.100.20 at o2ib10
----------------
kjcf05n07
----------------
routes=
Routing Disabled
net hops priority state router
gni -1 0 up 10.16.100.15 at o2ib11
gni -1 0 up 10.16.100.11 at o2ib11
gni -1 0 up 10.16.100.21 at o2ib11
gni -1 0 up 10.16.100.19 at o2ib11
gni -1 0 up 10.16.100.13 at o2ib11
gni -1 0 up 10.16.100.17 at o2ib11
----------------
kjcf05n08
----------------
routes=
Routing Disabled
net hops priority state router
gni -1 0 up 10.16.100.16 at o2ib10
gni -1 0 up 10.16.100.20 at o2ib10
gni -1 0 up 10.16.100.10 at o2ib10
gni -1 0 up 10.16.100.18 at o2ib10
gni -1 0 up 10.16.100.12 at o2ib10
gni -1 0 up 10.16.100.14 at o2ib10
----------------
kjcf05n09
----------------
routes=
Routing Disabled
net hops priority state router
gni -1 0 up 10.16.100.21 at o2ib11
gni -1 0 up 10.16.100.11 at o2ib11
gni -1 0 up 10.16.100.13 at o2ib11
gni -1 0 up 10.16.100.17 at o2ib11
gni -1 0 up 10.16.100.19 at o2ib11
gni -1 0 up 10.16.100.15 at o2ib11
I'm using the below yaml
ip2nets:
- net-spec: o2ib
interfaces:
0: ib0
1: ib1
ip-range:
0: 10.16.100.[52,53,56,57,62,64]
1: 10.16.100.[58,59,60,61,63,65]
- net-spec: o2ib
interfaces:
0: ib0
ip-range:
0: 10.16.100.[50,51,54,55]
- net-spec: o2ib10
interfaces:
0: ib0
1: ib1
ip-range:
0: 10.16.100.[52,56,62]
1: 10.16.100.[58,60,63]
- net-spec: o2ib10
interfaces:
0: ib0
ip-range:
0: 10.16.100.[54]
- net-spec: o2ib11
interfaces:
0: ib0
1: ib1
ip-range:
0: 10.16.100.[53,57,64]
1: 10.16.100.[59,61,65]
- net-spec: o2ib11
interfaces:
0: ib0
ip-range:
0: 10.16.100.[55]
route:
- net: gni
gateway: 10.16.100.10 at o2ib10
hop: -1
priority: 0
- net: gni
gateway: 10.16.100.12 at o2ib10
hop: -1
priority: 0
- net: gni
gateway: 10.16.100.14 at o2ib10
hop: -1
priority: 0
- net: gni
gateway: 10.16.100.16 at o2ib10
hop: -1
priority: 0
- net: gni
gateway: 10.16.100.18 at o2ib10
hop: -1
priority: 0
- net: gni
gateway: 10.16.100.20 at o2ib10
hop: -1
priority: 0
- net: gni
gateway: 10.16.100.11 at o2ib11
hop: -1
priority: 0
- net: gni
gateway: 10.16.100.13 at o2ib11
hop: -1
priority: 0
- net: gni
gateway: 10.16.100.15 at o2ib11
hop: -1
priority: 0
- net: gni
gateway: 10.16.100.17 at o2ib11
hop: -1
priority: 0
- net: gni
gateway: 10.16.100.19 at o2ib11
hop: -1
priority: 0
- net: gni
gateway: 10.16.100.21 at o2ib11
hop: -1
priority: 0
peer:
- primary nid: 10.16.100.10 at o2ib10
Multi-Rail: True
peer ni:
- nid: 10.16.100.10 at o2ib10
- primary nid: 10.16.100.12 at o2ib10
Multi-Rail: True
peer ni:
- nid: 10.16.100.12 at o2ib10
- primary nid: 10.16.100.14 at o2ib10
Multi-Rail: True
peer ni:
- nid: 10.16.100.14 at o2ib10
- primary nid: 10.16.100.16 at o2ib10
Multi-Rail: True
peer ni:
- nid: 10.16.100.16 at o2ib10
- primary nid: 10.16.100.18 at o2ib10
Multi-Rail: True
peer ni:
- nid: 10.16.100.18 at o2ib10
- primary nid: 10.16.100.20 at o2ib10
Multi-Rail: True
peer ni:
- nid: 10.16.100.20 at o2ib10
- primary nid: 10.16.100.11 at o2ib11
Multi-Rail: True
peer ni:
- nid: 10.16.100.11 at o2ib11
- primary nid: 10.16.100.13 at o2ib11
Multi-Rail: True
peer ni:
- nid: 10.16.100.13 at o2ib11
- primary nid: 10.16.100.15 at o2ib11
Multi-Rail: True
peer ni:
- nid: 10.16.100.15 at o2ib11
- primary nid: 10.16.100.17 at o2ib11
Multi-Rail: True
peer ni:
- nid: 10.16.100.17 at o2ib11
- primary nid: 10.16.100.19 at o2ib11
Multi-Rail: True
peer ni:
- nid: 10.16.100.19 at o2ib11
- primary nid: 10.16.100.21 at o2ib11
Multi-Rail: True
peer ni:
- nid: 10.16.100.21 at o2ib11
and NIDs are configured properly but the routes aren't picking up of odd nodes:
----------------
kjcf05n02
----------------
routes=
Routing Disabled
net hops priority state router
gni -1 0 up 10.16.100.20 at o2ib10
gni -1 0 up 10.16.100.14 at o2ib10
gni -1 0 up 10.16.100.18 at o2ib10
gni -1 0 up 10.16.100.16 at o2ib10
gni -1 0 up 10.16.100.10 at o2ib10
gni -1 0 up 10.16.100.12 at o2ib10
----------------
kjcf05n[03,05,07,09]
----------------
routes=
Routing Disabled
net hops priority state router
----------------
kjcf05n04
----------------
routes=
Routing Disabled
net hops priority state router
gni -1 0 up 10.16.100.18 at o2ib10
gni -1 0 up 10.16.100.14 at o2ib10
gni -1 0 up 10.16.100.16 at o2ib10
gni -1 0 up 10.16.100.12 at o2ib10
gni -1 0 up 10.16.100.20 at o2ib10
gni -1 0 up 10.16.100.10 at o2ib10
----------------
kjcf05n06
----------------
routes=
Routing Disabled
net hops priority state router
gni -1 0 up 10.16.100.16 at o2ib10
gni -1 0 up 10.16.100.14 at o2ib10
gni -1 0 up 10.16.100.12 at o2ib10
gni -1 0 up 10.16.100.20 at o2ib10
gni -1 0 up 10.16.100.18 at o2ib10
gni -1 0 up 10.16.100.10 at o2ib10
----------------
kjcf05n08
----------------
routes=
Routing Disabled
net hops priority state router
gni -1 0 up 10.16.100.18 at o2ib10
gni -1 0 up 10.16.100.10 at o2ib10
gni -1 0 up 10.16.100.20 at o2ib10
gni -1 0 up 10.16.100.16 at o2ib10
gni -1 0 up 10.16.100.12 at o2ib10
gni -1 0 up 10.16.100.14 at o2ib10
Can someone please help me to identify what am I missing in yaml ? How can I change the yaml to get the expected config ?
Thanks.
Praveen Kumar S
Systems Software Engineer
ClusterStor/Lustre, HPC & AI
praveen-kumar.s4 at hpe.com
Hewlett Packard Enterprise
HPE.com
[cid:961a4a31-8821-4835-bfa7-6c3aa6aac540]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260718/94f9c8e0/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 341 bytes
Desc: image.png
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260718/94f9c8e0/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 341 bytes
Desc: image.png
URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260718/94f9c8e0/attachment-0001.png>
More information about the lustre-discuss
mailing list