[lustre-discuss] lustre-discuss Digest, Vol 1, Issue 1
Ms. Megan Larko
dobsonunit at gmail.com
Mon Jul 20 13:04:29 UTC 2026
In the output I am seeing “gni” tagged as “o2ib10”. Can gni network have an o2ibX tag, compared with a “gni” tag? Does gni use a nodeID with tag rather than IP num?
Cheers,
Megan
From mobile Megan.
> On Jul 18, 2026, at 2:35 AM, lustre-discuss-request at lists.lustre.org wrote:
>
> Send lustre-discuss mailing list submissions to
> lustre-discuss at lists.lustre.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org
>
> or, via email, send a message with subject or body 'help' to
> lustre-discuss-request at lists.lustre.org
>
> You can reach the person managing the list at
> lustre-discuss-owner at lists.lustre.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of lustre-discuss digest..."
>
>
> Today's Topics:
>
> 1. Re: Need help with Creating Single Unified LNET yaml config
> (S, Praveen Kumar)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 18 Jul 2026 06:32:19 +0000
> From: "S, Praveen Kumar" <praveen-kumar.s4 at hpe.com>
> To: Chris Horn <chorn at ddn.com>,
> "lustre-discuss at lists.lustre.org" <lustre-discuss at lists.lustre.org>
> Subject: Re: [lustre-discuss] Need help with Creating Single Unified
> LNET yaml config
> Message-ID:
> <MW5PR84MB17718565ECF423DE31A6141497C52 at MW5PR84MB1771.NAMPRD84.PROD.OUTLOOK.COM>
>
> Content-Type: text/plain; charset="utf-8"
>
> I haven't got chance to check with master. I'll check with master asap and will keep you posted.
>
> Thanks.
>
>
> Praveen Kumar S
> Systems Software Engineer
> ClusterStor/Lustre, HPC & AI
> praveen-kumar.s4 at hpe.com
> Hewlett Packard Enterprise
> HPE.com
>
> [cid:d30d9b93-d47f-449f-8e8c-fe9944211f24]
> ________________________________
> From: Chris Horn <chorn at ddn.com>
> Sent: Tuesday, July 14, 2026 7:25 PM
> To: S, Praveen Kumar <praveen-kumar.s4 at hpe.com>; lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
> Subject: Re: Need help with Creating Single Unified LNET yaml config
>
> Does this issue reproduce with master or just HPE Lustre?
>
> Chris
>
> From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of S, Praveen Kumar via lustre-discuss <lustre-discuss at lists.lustre.org>
> Date: Monday, July 13, 2026 at 6:15 AM
> To: lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
> Subject: [lustre-discuss] Need help with Creating Single Unified LNET yaml config
>
> Greetings Everyone,
>
> I'm trying to config LNET on Servers via unified yaml and I'm facing problem with configuring routes: See below
>
> My expected config:
>
> [root at kjcf05n00 pravs]# pdsh -g lustre 'lctl list_nids' | dshbak -c
> ----------------
> kjcf05n02
> ----------------
> 10.16.100.52 at o2ib
> 10.16.100.58 at o2ib
> 10.16.100.52 at o2ib10
> 10.16.100.58 at o2ib10
> ----------------
> kjcf05n03
> ----------------
> 10.16.100.53 at o2ib
> 10.16.100.59 at o2ib
> 10.16.100.53 at o2ib11
> 10.16.100.59 at o2ib11
> ----------------
> kjcf05n04
> ----------------
> 10.16.100.54 at o2ib
> 10.16.100.54 at o2ib10
> ----------------
> kjcf05n05
> ----------------
> 10.16.100.55 at o2ib
> 10.16.100.55 at o2ib11
> ----------------
> kjcf05n06
> ----------------
> 10.16.100.56 at o2ib
> 10.16.100.60 at o2ib
> 10.16.100.56 at o2ib10
> 10.16.100.60 at o2ib10
> ----------------
> kjcf05n07
> ----------------
> 10.16.100.57 at o2ib
> 10.16.100.61 at o2ib
> 10.16.100.57 at o2ib11
> 10.16.100.61 at o2ib11
> ----------------
> kjcf05n08
> ----------------
> 10.16.100.62 at o2ib
> 10.16.100.63 at o2ib
> 10.16.100.62 at o2ib10
> 10.16.100.63 at o2ib10
> ----------------
> kjcf05n09
> ----------------
> 10.16.100.64 at o2ib
> 10.16.100.65 at o2ib
> 10.16.100.64 at o2ib11
> 10.16.100.65 at o2ib11
>
> [root at kjcf05n00 admin]# pdsh -g lustre 'lctl get_param routes' | dshbak -c
> ----------------
> kjcf05n02
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> gni -1 0 up 10.16.100.10 at o2ib10
> gni -1 0 up 10.16.100.18 at o2ib10
> gni -1 0 up 10.16.100.20 at o2ib10
> gni -1 0 up 10.16.100.16 at o2ib10
> gni -1 0 up 10.16.100.12 at o2ib10
> gni -1 0 up 10.16.100.14 at o2ib10
> ----------------
> kjcf05n03
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> gni -1 0 up 10.16.100.11 at o2ib11
> gni -1 0 up 10.16.100.17 at o2ib11
> gni -1 0 up 10.16.100.15 at o2ib11
> gni -1 0 up 10.16.100.19 at o2ib11
> gni -1 0 up 10.16.100.13 at o2ib11
> gni -1 0 up 10.16.100.21 at o2ib11
> ----------------
> kjcf05n04
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> gni -1 0 up 10.16.100.16 at o2ib10
> gni -1 0 up 10.16.100.10 at o2ib10
> gni -1 0 up 10.16.100.18 at o2ib10
> gni -1 0 up 10.16.100.12 at o2ib10
> gni -1 0 up 10.16.100.14 at o2ib10
> gni -1 0 up 10.16.100.20 at o2ib10
> ----------------
> kjcf05n05
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> gni -1 0 up 10.16.100.19 at o2ib11
> gni -1 0 up 10.16.100.13 at o2ib11
> gni -1 0 up 10.16.100.21 at o2ib11
> gni -1 0 up 10.16.100.17 at o2ib11
> gni -1 0 up 10.16.100.15 at o2ib11
> gni -1 0 up 10.16.100.11 at o2ib11
> ----------------
> kjcf05n06
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> gni -1 0 up 10.16.100.12 at o2ib10
> gni -1 0 up 10.16.100.10 at o2ib10
> gni -1 0 up 10.16.100.18 at o2ib10
> gni -1 0 up 10.16.100.14 at o2ib10
> gni -1 0 up 10.16.100.16 at o2ib10
> gni -1 0 up 10.16.100.20 at o2ib10
> ----------------
> kjcf05n07
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> gni -1 0 up 10.16.100.15 at o2ib11
> gni -1 0 up 10.16.100.11 at o2ib11
> gni -1 0 up 10.16.100.21 at o2ib11
> gni -1 0 up 10.16.100.19 at o2ib11
> gni -1 0 up 10.16.100.13 at o2ib11
> gni -1 0 up 10.16.100.17 at o2ib11
> ----------------
> kjcf05n08
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> gni -1 0 up 10.16.100.16 at o2ib10
> gni -1 0 up 10.16.100.20 at o2ib10
> gni -1 0 up 10.16.100.10 at o2ib10
> gni -1 0 up 10.16.100.18 at o2ib10
> gni -1 0 up 10.16.100.12 at o2ib10
> gni -1 0 up 10.16.100.14 at o2ib10
> ----------------
> kjcf05n09
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> gni -1 0 up 10.16.100.21 at o2ib11
> gni -1 0 up 10.16.100.11 at o2ib11
> gni -1 0 up 10.16.100.13 at o2ib11
> gni -1 0 up 10.16.100.17 at o2ib11
> gni -1 0 up 10.16.100.19 at o2ib11
> gni -1 0 up 10.16.100.15 at o2ib11
>
>
> I'm using the below yaml
>
> ip2nets:
> - net-spec: o2ib
> interfaces:
> 0: ib0
> 1: ib1
> ip-range:
> 0: 10.16.100.[52,53,56,57,62,64]
> 1: 10.16.100.[58,59,60,61,63,65]
>
> - net-spec: o2ib
> interfaces:
> 0: ib0
> ip-range:
> 0: 10.16.100.[50,51,54,55]
>
> - net-spec: o2ib10
> interfaces:
> 0: ib0
> 1: ib1
> ip-range:
> 0: 10.16.100.[52,56,62]
> 1: 10.16.100.[58,60,63]
>
> - net-spec: o2ib10
> interfaces:
> 0: ib0
> ip-range:
> 0: 10.16.100.[54]
>
> - net-spec: o2ib11
> interfaces:
> 0: ib0
> 1: ib1
> ip-range:
> 0: 10.16.100.[53,57,64]
> 1: 10.16.100.[59,61,65]
>
> - net-spec: o2ib11
> interfaces:
> 0: ib0
> ip-range:
> 0: 10.16.100.[55]
>
> route:
> - net: gni
> gateway: 10.16.100.10 at o2ib10
> hop: -1
> priority: 0
> - net: gni
> gateway: 10.16.100.12 at o2ib10
> hop: -1
> priority: 0
> - net: gni
> gateway: 10.16.100.14 at o2ib10
> hop: -1
> priority: 0
> - net: gni
> gateway: 10.16.100.16 at o2ib10
> hop: -1
> priority: 0
> - net: gni
> gateway: 10.16.100.18 at o2ib10
> hop: -1
> priority: 0
> - net: gni
> gateway: 10.16.100.20 at o2ib10
> hop: -1
> priority: 0
> - net: gni
> gateway: 10.16.100.11 at o2ib11
> hop: -1
> priority: 0
> - net: gni
> gateway: 10.16.100.13 at o2ib11
> hop: -1
> priority: 0
> - net: gni
> gateway: 10.16.100.15 at o2ib11
> hop: -1
> priority: 0
> - net: gni
> gateway: 10.16.100.17 at o2ib11
> hop: -1
> priority: 0
> - net: gni
> gateway: 10.16.100.19 at o2ib11
> hop: -1
> priority: 0
> - net: gni
> gateway: 10.16.100.21 at o2ib11
> hop: -1
> priority: 0
>
> peer:
> - primary nid: 10.16.100.10 at o2ib10
> Multi-Rail: True
> peer ni:
> - nid: 10.16.100.10 at o2ib10
>
> - primary nid: 10.16.100.12 at o2ib10
> Multi-Rail: True
> peer ni:
> - nid: 10.16.100.12 at o2ib10
>
> - primary nid: 10.16.100.14 at o2ib10
> Multi-Rail: True
> peer ni:
> - nid: 10.16.100.14 at o2ib10
>
> - primary nid: 10.16.100.16 at o2ib10
> Multi-Rail: True
> peer ni:
> - nid: 10.16.100.16 at o2ib10
>
> - primary nid: 10.16.100.18 at o2ib10
> Multi-Rail: True
> peer ni:
> - nid: 10.16.100.18 at o2ib10
>
> - primary nid: 10.16.100.20 at o2ib10
> Multi-Rail: True
> peer ni:
> - nid: 10.16.100.20 at o2ib10
>
> - primary nid: 10.16.100.11 at o2ib11
> Multi-Rail: True
> peer ni:
> - nid: 10.16.100.11 at o2ib11
>
> - primary nid: 10.16.100.13 at o2ib11
> Multi-Rail: True
> peer ni:
> - nid: 10.16.100.13 at o2ib11
>
> - primary nid: 10.16.100.15 at o2ib11
> Multi-Rail: True
> peer ni:
> - nid: 10.16.100.15 at o2ib11
>
> - primary nid: 10.16.100.17 at o2ib11
> Multi-Rail: True
> peer ni:
> - nid: 10.16.100.17 at o2ib11
>
> - primary nid: 10.16.100.19 at o2ib11
> Multi-Rail: True
> peer ni:
> - nid: 10.16.100.19 at o2ib11
>
> - primary nid: 10.16.100.21 at o2ib11
> Multi-Rail: True
> peer ni:
> - nid: 10.16.100.21 at o2ib11
>
> and NIDs are configured properly but the routes aren't picking up of odd nodes:
>
> ----------------
> kjcf05n02
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> gni -1 0 up 10.16.100.20 at o2ib10
> gni -1 0 up 10.16.100.14 at o2ib10
> gni -1 0 up 10.16.100.18 at o2ib10
> gni -1 0 up 10.16.100.16 at o2ib10
> gni -1 0 up 10.16.100.10 at o2ib10
> gni -1 0 up 10.16.100.12 at o2ib10
> ----------------
> kjcf05n[03,05,07,09]
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> ----------------
> kjcf05n04
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> gni -1 0 up 10.16.100.18 at o2ib10
> gni -1 0 up 10.16.100.14 at o2ib10
> gni -1 0 up 10.16.100.16 at o2ib10
> gni -1 0 up 10.16.100.12 at o2ib10
> gni -1 0 up 10.16.100.20 at o2ib10
> gni -1 0 up 10.16.100.10 at o2ib10
> ----------------
> kjcf05n06
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> gni -1 0 up 10.16.100.16 at o2ib10
> gni -1 0 up 10.16.100.14 at o2ib10
> gni -1 0 up 10.16.100.12 at o2ib10
> gni -1 0 up 10.16.100.20 at o2ib10
> gni -1 0 up 10.16.100.18 at o2ib10
> gni -1 0 up 10.16.100.10 at o2ib10
> ----------------
> kjcf05n08
> ----------------
> routes=
> Routing Disabled
> net hops priority state router
> gni -1 0 up 10.16.100.18 at o2ib10
> gni -1 0 up 10.16.100.10 at o2ib10
> gni -1 0 up 10.16.100.20 at o2ib10
> gni -1 0 up 10.16.100.16 at o2ib10
> gni -1 0 up 10.16.100.12 at o2ib10
> gni -1 0 up 10.16.100.14 at o2ib10
>
> Can someone please help me to identify what am I missing in yaml ? How can I change the yaml to get the expected config ?
>
> Thanks.
>
> Praveen Kumar S
> Systems Software Engineer
> ClusterStor/Lustre, HPC & AI
> praveen-kumar.s4 at hpe.com
> Hewlett Packard Enterprise
> HPE.com
>
> [cid:961a4a31-8821-4835-bfa7-6c3aa6aac540]
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260718/94f9c8e0/attachment.html>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image.png
> Type: image/png
> Size: 341 bytes
> Desc: image.png
> URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260718/94f9c8e0/attachment.png>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image.png
> Type: image/png
> Size: 341 bytes
> Desc: image.png
> URL: <http://lists.lustre.org/pipermail/lustre-discuss_lists.lustre.org/attachments/20260718/94f9c8e0/attachment-0001.png>
>
> ------------------------------
>
> Subject: Digest Footer
>
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss_lists.lustre.org
>
>
> ------------------------------
>
> End of lustre-discuss Digest, Vol 1, Issue 1
> ********************************************
More information about the lustre-discuss
mailing list