[lustre-discuss] lustre-discuss Digest, Vol 228, Issue 4
Berry-Lozano, Erica
erica.berry-lozano at hpe.com
Wed Mar 5 20:44:45 PST 2025
Remove me from this email distribution. Thanks
-----Original Message-----
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> On Behalf Of lustre-discuss-request at lists.lustre.org
Sent: Wednesday, March 5, 2025 10:43 PM
To: lustre-discuss at lists.lustre.org
Subject: lustre-discuss Digest, Vol 228, Issue 4
Send lustre-discuss mailing list submissions to
lustre-discuss at lists.lustre.org
To subscribe or unsubscribe via the World Wide Web, visit
https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!NpxR!iLqFg9mHSFdTvFBBdIctwSExoOvHn9aTqkO3lh98X6RuKATpU7m6URNvsD02ThhhDvx7OcvFTi8J-UMgYDTpr5zNWPRU16KLKZxY_Q$
or, via email, send a message with subject or body 'help' to
lustre-discuss-request at lists.lustre.org
You can reach the person managing the list at
lustre-discuss-owner at lists.lustre.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of lustre-discuss digest..."
Today's Topics:
1. Re: Lustre MDT/OST Mount Failures During Virtual Machine
Reboot with Pacemaker (Laura Hild)
2. Re: multi-hop routing (John White)
3. Re: multi-hop routing (John White)
4. Re: lustre-discuss Digest, Vol 228, Issue 3 (Berry-Lozano, Erica)
----------------------------------------------------------------------
Message: 1
Date: Wed, 5 Mar 2025 22:12:00 +0000
From: Laura Hild <lsh at jlab.org>
To: "chenzufei at gmail.com" <chenzufei at gmail.com>
Cc: lustre-discuss <lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] Lustre MDT/OST Mount Failures During
Virtual Machine Reboot with Pacemaker
Message-ID:
<BLAPR09MB62116113A1788BD2D8B6DAD2DCCB2 at BLAPR09MB6211.namprd09.prod.outlook.com>
Content-Type: text/plain; charset="iso-8859-2"
I'm not sure what to say about how Pacemaker *should* behave, but I *can* say I virtually never try to (cleanly) reboot a host from which I have not already evacuated all resources, e.g. with `pcs node standby` or by putting Pacemaker in maintenance mode and unmounting/exporting everything manually. If I can't evacuate all resources and complete a lustre_rmmod, the host is getting power-cycled.
So maybe I can say, my guess would be that in the host's shutdown process, stopping the Pacemaker service happens before filesystems are unmounted, and that Pacemaker doesn't want to make an assumption whether its own shut-down means it should standby or initiate maintenance mode, and therefore the other host ends up knowing only that its partner has disappeared, while the filesystems have yet to be unmounted.
------------------------------
Message: 2
Date: Wed, 5 Mar 2025 14:29:10 -0800
From: John White <jwhite at lbl.gov>
To: "Horn, Chris" <chris.horn at hpe.com>
Cc: "lustre-discuss at lists.lustre.org"
<lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] multi-hop routing
Message-ID: <76DA0A9B-F47B-4EF8-84FC-FC7283C48BF4 at lbl.gov>
Content-Type: text/plain; charset=utf-8
Oh, so don?t even tell the client about tcp! That seems to have immediately kicked things into place!
I owe you a beverage of your choice if we ever meet up!
Seriously, the imposter syndrome was getting _bad_ the last few days here.
> On Mar 5, 2025, at 12:05?PM, Horn, Chris <chris.horn at hpe.com> wrote:
>
> You need LNet routes configured on all nodes. It should look something like this:
>
> # pdsh -w n0[0-3] 'lctl list_nids; lctl show_route' | dshbak -c
> ----------------
> server
> ----------------
> 172.18.2.5 at o2ib
> net o2ib2 hops 2 gw 172.18.2.6 at o2ib up pri 0
> ----------------
> router1
> ----------------
> 172.18.2.6 at o2ib
> 172.18.2.2 at tcp
> net o2ib2 hops 1 gw 172.18.2.3 at tcp up pri 0
> ----------------
> router2
> ----------------
> 172.18.2.7 at o2ib2
> 172.18.2.3 at tcp
> net o2ib hops 1 gw 172.18.2.2 at tcp up pri 0
> ----------------
> client
> ----------------
> 172.18.2.8 at o2ib2
> net o2ib hops 2 gw 172.18.2.7 at o2ib2 up pri 0
> #
> Chris Horn
> From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on
> behalf of John White via lustre-discuss
> <lustre-discuss at lists.lustre.org>
> Date: Wednesday, March 5, 2025 at 1:17?PM
> To: lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
> Subject: [lustre-discuss] multi-hop routing Hello folks. I have a
> rare situation that I?m told some centers are successfully pulling off and am looking for guidance - multi-hop lnet routing.
> In short, I have 2 distinct o2ib fabrics at disparate geo sites joined by a routed ethernet fabric. I?m looking to use a 2-lnet-router chain to plumb the two o2ib fabrics together.
>
> servers on the left, clients on the right
> o2ib0(10.5.0.0/16) <-> router(o2ib0,tcp0) <-> routed eth
> (10.37.0.0/16, 10.38.0.0/16) <-> router(tcp0,o2ib2) <->
> o2ib2(10.6.0.0/16)
>
> I have both sets of routers up but traffic absolutely fails the 2nd hop in either direction (I can `lctl ping` tcp0 from o2ib2 and o2ib0 but no further).
>
> I?ve tried adding a route ON the routers, that didn?t help.
>
> I?ve tried defining the 2nd hop on the client:
> options lnet routes="tcp0 10.6.0.[250-251]@o2ib2;\
> o2ib0 10.37.250.[162-163]@tcp0?
>
> but that failed with the following kern message on lnet load:
> 74067:0:(router.c:644:lnet_add_route()) Cannot add route with gateway
> 10.37.250.162 at tcp. There is no local interface configured on LNet tcp
>
> Does anyone have any hints here? It feels like I?m a syntax change or a routing hint away from getting this working.
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustr
> e-discuss-lustre.org__;!!NpxR!keuGPb7MHd7CQc6Zi_uwIvFahK68FJfbq9MNIXgH
> pd0W8bi5vOYFHf-IixYY5DiOnJKx0z9-Ht8VqH1ew82XWtaTRaoq$
------------------------------
Message: 3
Date: Wed, 5 Mar 2025 15:47:34 -0800
From: John White <jwhite at lbl.gov>
To: "Horn, Chris" <chris.horn at hpe.com>
Cc: "lustre-discuss at lists.lustre.org"
<lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] multi-hop routing
Message-ID: <2AC528AD-1331-46BA-9EEA-FAC797FB6961 at lbl.gov>
Content-Type: text/plain; charset=utf-8
Just a quick follow-up for posterity, I did seem to need to add a route for tcp to the server-side. lctl ping was working but MGS communication was failing saying it couldn?t talk back to the router:
[Wed Mar 5 15:28:26 2025] LNetError: 28576:0:(lib-move.c:2078:lnet_handle_find_routed_path()) no route to 10.38.0.250 at tcp from 10.5.250.22 at o2ib [Wed Mar 5 15:28:26 2025] LNetError: 28576:0:(lib-move.c:3991:lnet_parse_get()) 10.5.250.22 at o2ib: Unable to send REPLY for GET from 12345-10.38.0.250 at tcp: -113
Adding a route to tcp from it?s geo-local router fixed that and we?ve got mounts passing IO. Didn?t seem to need to do the same for clients at all.
> On Mar 5, 2025, at 2:29?PM, John White <jwhite at lbl.gov> wrote:
>
> Oh, so don?t even tell the client about tcp! That seems to have immediately kicked things into place!
> I owe you a beverage of your choice if we ever meet up!
>
> Seriously, the imposter syndrome was getting _bad_ the last few days here.
>
>> On Mar 5, 2025, at 12:05?PM, Horn, Chris <chris.horn at hpe.com> wrote:
>>
>> You need LNet routes configured on all nodes. It should look something like this:
>>
>> # pdsh -w n0[0-3] 'lctl list_nids; lctl show_route' | dshbak -c
>> ----------------
>> server
>> ----------------
>> 172.18.2.5 at o2ib
>> net o2ib2 hops 2 gw 172.18.2.6 at o2ib up pri 0
>> ----------------
>> router1
>> ----------------
>> 172.18.2.6 at o2ib
>> 172.18.2.2 at tcp
>> net o2ib2 hops 1 gw 172.18.2.3 at tcp up pri 0
>> ----------------
>> router2
>> ----------------
>> 172.18.2.7 at o2ib2
>> 172.18.2.3 at tcp
>> net o2ib hops 1 gw 172.18.2.2 at tcp up pri 0
>> ----------------
>> client
>> ----------------
>> 172.18.2.8 at o2ib2
>> net o2ib hops 2 gw 172.18.2.7 at o2ib2 up pri 0
>> #
>> Chris Horn
>> From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on
>> behalf of John White via lustre-discuss
>> <lustre-discuss at lists.lustre.org>
>> Date: Wednesday, March 5, 2025 at 1:17?PM
>> To: lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
>> Subject: [lustre-discuss] multi-hop routing Hello folks. I have a
>> rare situation that I?m told some centers are successfully pulling off and am looking for guidance - multi-hop lnet routing.
>> In short, I have 2 distinct o2ib fabrics at disparate geo sites joined by a routed ethernet fabric. I?m looking to use a 2-lnet-router chain to plumb the two o2ib fabrics together.
>>
>> servers on the left, clients on the right
>> o2ib0(10.5.0.0/16) <-> router(o2ib0,tcp0) <-> routed eth
>> (10.37.0.0/16, 10.38.0.0/16) <-> router(tcp0,o2ib2) <->
>> o2ib2(10.6.0.0/16)
>>
>> I have both sets of routers up but traffic absolutely fails the 2nd hop in either direction (I can `lctl ping` tcp0 from o2ib2 and o2ib0 but no further).
>>
>> I?ve tried adding a route ON the routers, that didn?t help.
>>
>> I?ve tried defining the 2nd hop on the client:
>> options lnet routes="tcp0 10.6.0.[250-251]@o2ib2;\
>> o2ib0 10.37.250.[162-163]@tcp0?
>>
>> but that failed with the following kern message on lnet load:
>> 74067:0:(router.c:644:lnet_add_route()) Cannot add route with gateway
>> 10.37.250.162 at tcp. There is no local interface configured on LNet tcp
>>
>> Does anyone have any hints here? It feels like I?m a syntax change or a routing hint away from getting this working.
>> _______________________________________________
>> lustre-discuss mailing list
>> lustre-discuss at lists.lustre.org
>> https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lust
>> re-discuss-lustre.org__;!!NpxR!keuGPb7MHd7CQc6Zi_uwIvFahK68FJfbq9MNIX
>> gHpd0W8bi5vOYFHf-IixYY5DiOnJKx0z9-Ht8VqH1ew82XWtaTRaoq$
>
>
------------------------------
Message: 4
Date: Thu, 6 Mar 2025 04:42:18 +0000
From: "Berry-Lozano, Erica" <erica.berry-lozano at hpe.com>
To: "lustre-discuss at lists.lustre.org"
<lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] lustre-discuss Digest, Vol 228, Issue 3
Message-ID:
<PH0PR84MB1407E5335DCA60E74F5B3A02D0CA2 at PH0PR84MB1407.NAMPRD84.PROD.OUTLOOK.COM>
Content-Type: text/plain; charset="us-ascii"
Please remove me from this email distribution list. Thanks
-----Original Message-----
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> On Behalf Of lustre-discuss-request at lists.lustre.org
Sent: Wednesday, March 5, 2025 2:05 PM
To: lustre-discuss at lists.lustre.org
Subject: lustre-discuss Digest, Vol 228, Issue 3
Send lustre-discuss mailing list submissions to
lustre-discuss at lists.lustre.org
To subscribe or unsubscribe via the World Wide Web, visit
https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!NpxR!nW3d5kUkGUumTD2D9jXSF5CvA3MWMg1Ye4tGQs4BwUrstkBSP9l5HNq08rXZwbINHfO2eMRHdwzqA7IXwpZAmNEC1W0u3ef_v-URSg$
or, via email, send a message with subject or body 'help' to
lustre-discuss-request at lists.lustre.org
You can reach the person managing the list at
lustre-discuss-owner at lists.lustre.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of lustre-discuss digest..."
Today's Topics:
1. multi-hop routing (John White)
2. Re: multi-hop routing (Horn, Chris)
----------------------------------------------------------------------
Message: 1
Date: Wed, 5 Mar 2025 11:14:49 -0800
From: John White <jwhite at lbl.gov>
To: lustre-discuss at lists.lustre.org
Subject: [lustre-discuss] multi-hop routing
Message-ID: <32AB4536-9924-41B8-8226-B3E3BF19F77E at lbl.gov>
Content-Type: text/plain; charset=utf-8
Hello folks. I have a rare situation that I?m told some centers are successfully pulling off and am looking for guidance - multi-hop lnet routing.
In short, I have 2 distinct o2ib fabrics at disparate geo sites joined by a routed ethernet fabric. I?m looking to use a 2-lnet-router chain to plumb the two o2ib fabrics together.
servers on the left, clients on the right
o2ib0(10.5.0.0/16) <-> router(o2ib0,tcp0) <-> routed eth (10.37.0.0/16, 10.38.0.0/16) <-> router(tcp0,o2ib2) <-> o2ib2(10.6.0.0/16)
I have both sets of routers up but traffic absolutely fails the 2nd hop in either direction (I can `lctl ping` tcp0 from o2ib2 and o2ib0 but no further).
I?ve tried adding a route ON the routers, that didn?t help.
I?ve tried defining the 2nd hop on the client:
options lnet routes="tcp0 10.6.0.[250-251]@o2ib2;\
o2ib0 10.37.250.[162-163]@tcp0?
but that failed with the following kern message on lnet load:
74067:0:(router.c:644:lnet_add_route()) Cannot add route with gateway 10.37.250.162 at tcp. There is no local interface configured on LNet tcp
Does anyone have any hints here? It feels like I?m a syntax change or a routing hint away from getting this working.
------------------------------
Message: 2
Date: Wed, 5 Mar 2025 20:05:02 +0000
From: "Horn, Chris" <chris.horn at hpe.com>
To: John White <jwhite at lbl.gov>, "lustre-discuss at lists.lustre.org"
<lustre-discuss at lists.lustre.org>
Subject: Re: [lustre-discuss] multi-hop routing
Message-ID:
<PH7PR84MB1438479204DBEC1E027B8FC79ECB2 at PH7PR84MB1438.NAMPRD84.PROD.OUTLOOK.COM>
Content-Type: text/plain; charset="utf-8"
You need LNet routes configured on all nodes. It should look something like this:
# pdsh -w n0[0-3] 'lctl list_nids; lctl show_route' | dshbak -c
----------------
server
----------------
172.18.2.5 at o2ib<mailto:172.18.2.5 at o2ib>
net o2ib2 hops 2 gw 172.18.2.6 at o2ib<mailto:172.18.2.6 at o2ib> up pri 0
----------------
router1
----------------
172.18.2.6 at o2ib<mailto:172.18.2.6 at o2ib>
172.18.2.2 at tcp<mailto:172.18.2.2 at tcp>
net o2ib2 hops 1 gw 172.18.2.3 at tcp<mailto:172.18.2.3 at tcp> up pri 0
----------------
router2
----------------
172.18.2.7 at o2ib2<mailto:172.18.2.7 at o2ib2>
172.18.2.3 at tcp<mailto:172.18.2.3 at tcp>
net o2ib hops 1 gw 172.18.2.2 at tcp<mailto:172.18.2.2 at tcp> up pri 0
----------------
client
----------------
172.18.2.8 at o2ib2<mailto:172.18.2.8 at o2ib2>
net o2ib hops 2 gw 172.18.2.7 at o2ib2<mailto:172.18.2.7 at o2ib2> up pri 0
#
Chris Horn
From: lustre-discuss <lustre-discuss-bounces at lists.lustre.org> on behalf of John White via lustre-discuss <lustre-discuss at lists.lustre.org>
Date: Wednesday, March 5, 2025 at 1:17?PM
To: lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
Subject: [lustre-discuss] multi-hop routing Hello folks. I have a rare situation that I?m told some centers are successfully pulling off and am looking for guidance - multi-hop lnet routing.
In short, I have 2 distinct o2ib fabrics at disparate geo sites joined by a routed ethernet fabric. I?m looking to use a 2-lnet-router chain to plumb the two o2ib fabrics together.
servers on the left, clients on the right
o2ib0(10.5.0.0/16) <-> router(o2ib0,tcp0) <-> routed eth (10.37.0.0/16, 10.38.0.0/16) <-> router(tcp0,o2ib2) <-> o2ib2(10.6.0.0/16)
I have both sets of routers up but traffic absolutely fails the 2nd hop in either direction (I can `lctl ping` tcp0 from o2ib2 and o2ib0 but no further).
I?ve tried adding a route ON the routers, that didn?t help.
I?ve tried defining the 2nd hop on the client:
options lnet routes="tcp0 10.6.0.[250-251]@o2ib2;\
o2ib0 10.37.250.[162-163]@tcp0?
but that failed with the following kern message on lnet load:
74067:0:(router.c:644:lnet_add_route()) Cannot add route with gateway 10.37.250.162 at tcp. There is no local interface configured on LNet tcp
Does anyone have any hints here? It feels like I?m a syntax change or a routing hint away from getting this working.
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!NpxR!keuGPb7MHd7CQc6Zi_uwIvFahK68FJfbq9MNIXgHpd0W8bi5vOYFHf-IixYY5DiOnJKx0z9-Ht8VqH1ew82XWtaTRaoq$<https://urldefense.com/v3/__http:/lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!NpxR!keuGPb7MHd7CQc6Zi_uwIvFahK68FJfbq9MNIXgHpd0W8bi5vOYFHf-IixYY5DiOnJKx0z9-Ht8VqH1ew82XWtaTRaoq$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://urldefense.com/v3/__http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20250305/bfa13970/attachment.htm__;!!NpxR!nW3d5kUkGUumTD2D9jXSF5CvA3MWMg1Ye4tGQs4BwUrstkBSP9l5HNq08rXZwbINHfO2eMRHdwzqA7IXwpZAmNEC1W0u3efbP49wvQ$ >
------------------------------
Subject: Digest Footer
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!NpxR!nW3d5kUkGUumTD2D9jXSF5CvA3MWMg1Ye4tGQs4BwUrstkBSP9l5HNq08rXZwbINHfO2eMRHdwzqA7IXwpZAmNEC1W0u3ef_v-URSg$
------------------------------
End of lustre-discuss Digest, Vol 228, Issue 3
**********************************************
------------------------------
Subject: Digest Footer
_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org
https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!NpxR!iLqFg9mHSFdTvFBBdIctwSExoOvHn9aTqkO3lh98X6RuKATpU7m6URNvsD02ThhhDvx7OcvFTi8J-UMgYDTpr5zNWPRU16KLKZxY_Q$
------------------------------
End of lustre-discuss Digest, Vol 228, Issue 4
**********************************************
More information about the lustre-discuss
mailing list