[lustre-discuss] lustre-discuss Digest, Vol 177, Issue 4
Ms. Megan Larko
dobsonunit at gmail.com
Fri Dec 4 14:07:31 PST 2020
WRT Subject lnet routing issue - 2.12.5 client with 2.10.3 server:
I concur with ExecStart=/usr/sbin/lnetctl set discovery=0 in the
/usr/lib/systemd/system/lnet.conf file. I also add a line below the
lnetctl line is to add a line to instantiate the lustre peers desired.
Example: ExecStart=/usr/sbin/lnetctl peer add A.B.C.[R-Z]@tcp9 --non-mr
The --non-mr is "no multirail" re-enforcing the discovery=0. This would be
done on the Lustre 2.12.x (2.12.5) box. Newer versions of Lustre are
"network greedy" (one might also say that lnet is not practicing social
distancing in networks). As a reminder, newer Lustre 2.12.x uses
/etc/lnet.conf, a commented-out example is included in the Lustre 2.12.x
client install.
Cheers,
megan
On Tue, Dec 1, 2020 at 12:45 PM <lustre-discuss-request at lists.lustre.org>
wrote:
> Send lustre-discuss mailing list submissions to
> lustre-discuss at lists.lustre.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> or, via email, send a message with subject or body 'help' to
> lustre-discuss-request at lists.lustre.org
>
> You can reach the person managing the list at
> lustre-discuss-owner at lists.lustre.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of lustre-discuss digest..."
>
>
> Today's Topics:
>
> 1. Re: lnet routing issue - 2.12.5 client with 2.10.3 server
> (Mark Lundie)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 1 Dec 2020 12:58:27 +0000
> From: Mark Lundie <mark.lundie at manchester.ac.uk>
> To: "Degremont, Aurelien" <degremoa at amazon.com>, f?rat y?lmaz
> <firatyilmazz at gmail.com>
> Cc: "lustre-discuss at lists.lustre.org"
> <lustre-discuss at lists.lustre.org>
> Subject: Re: [lustre-discuss] lnet routing issue - 2.12.5 client with
> 2.10.3 server
> Message-ID:
> <
> AM6PR0102MB3112477460E5A537EB5C06F8DEF40 at AM6PR0102MB3112.eurprd01.prod.exchangelabs.com
> >
>
> Content-Type: text/plain; charset="iso-8859-3"
>
> Hi Aur?lien,
>
> Many thanks! Sorry I missed that. I'll try disabling discovery as
> suggested.
>
> Thanks,
>
> Mark
> ________________________________
> From: Degremont, Aurelien <degremoa at amazon.com>
> Sent: 01 December 2020 12:42
> To: Mark Lundie <mark.lundie at manchester.ac.uk>; f?rat y?lmaz <
> firatyilmazz at gmail.com>
> Cc: lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
> Subject: Re: [lustre-discuss] lnet routing issue - 2.12.5 client with
> 2.10.3 server
>
>
> This is a known issue, see https://jira.whamcloud.com/browse/LU-11840 and
> https://jira.whamcloud.com/browse/LU-13548
>
>
>
> Aur?lien
>
>
>
> De : lustre-discuss <lustre-discuss-bounces at lists.lustre.org> au nom de
> Mark Lundie <mark.lundie at manchester.ac.uk>
> Date : mardi 1 d?cembre 2020 ? 13:16
> ? : f?rat y?lmaz <firatyilmazz at gmail.com>
> Cc : "lustre-discuss at lists.lustre.org" <lustre-discuss at lists.lustre.org>
> Objet : RE: [EXTERNAL] [lustre-discuss] lnet routing issue - 2.12.5 client
> with 2.10.3 server
>
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi Firat,
>
> Thanks for your reply. Apologies if I am being silly here, but there is no
> route configured for that network. We have the networks tcp (10.110.0.0/16)
> and tcp1 (10.10.0.0/16). The servers have interfaces on both, but the
> clients only have an interface on tcp1. I'm not sure why the client is
> trying to route to 10.110.0.21 at tcp:
>
>
>
> client # mount /net/lustre/
>
> mount.lustre: mount hmeta1 at tcp1:hmeta2 at tcp1:/lustre at /net/lustre
> failed: Input/output error
>
> Is the MGS running?
>
>
>
> hmeta1 resolves to 10.10.0.91, on tcp1.
>
>
>
> Thanks,
>
>
>
> Mark
>
> ________________________________
>
> From: f?rat y?lmaz <firatyilmazz at gmail.com>
> Sent: 01 December 2020 11:55
> To: Mark Lundie <mark.lundie at manchester.ac.uk>
> Cc: lustre-discuss at lists.lustre.org <lustre-discuss at lists.lustre.org>
> Subject: Re: [lustre-discuss] lnet routing issue - 2.12.5 client with
> 2.10.3 server
>
>
>
> Hi Mark,
>
>
>
> [Tue Dec 1 11:07:55 2020] LNetError:
> 2127:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to
> 10.110.0.21 at tcp from <?>
>
>
>
> I would suggest checking lnetctl routing show and remove the route to
> 10.110.0.21 at tcp and try to mount.
>
> https://wiki.lustre.org/LNet_Router_Config_Guide
>
>
>
>
>
>
>
> On Tue, Dec 1, 2020 at 2:41 PM Mark Lundie <mark.lundie at manchester.ac.uk
> <mailto:mark.lundie at manchester.ac.uk>> wrote:
>
> Hi all,
>
>
>
> I've just run in to an issue mounting on a newly upgraded client running
> 2.12.5 with 2.10.3 servers. Just to give some background, we're about to
> replace our existing Lustre storage, but will run it concurrently with the
> replacement for a couple of months. We'll be running 2.12.5 server on the
> new MDS and OSSs and I plan to update all clients to the same version. I
> would like to avoid updating the existing servers though.
>
>
>
> The problem is this. The servers have two tcp LNET networks, tcp and tcp1,
> on separate subnets and VLANs. The clients only see tcp1 (a small number
> are also on tcp3, routed via 2 lnet routers), which has been fine until
> now. With the 2.12.5 client, however, it is trying to mount from tcp.
> 2.10.3 to 2.12.5 is obviously a bit of a jump, but does anyone have any
> ideas on what has changed and what I could do here please?
>
>
>
> meta# lnetctl net show
>
> net:
>
> - net type: lo
>
> local NI(s):
>
> - nid: 0 at lo
>
> status: up
>
> - net type: tcp
>
> local NI(s):
>
> - nid: 10.110.0.21 at tcp
>
> status: up
>
> interfaces:
>
> 0: bond0.22
>
> - net type: tcp1
>
> local NI(s):
>
> - nid: 10.10.0.91 at tcp1
>
> status: up
>
> interfaces:
>
> 0: bond0
>
>
>
> meta# lnetctl route show
>
> route:
>
> - net: tcp2
>
> gateway: 10.10.0.254 at tcp1
>
> - net: tcp3
>
> gateway: 10.10.0.254 at tcp1
>
>
>
> client# lnetctl net show
>
> net:
>
> - net type: lo
>
> local NI(s):
>
> - nid: 0 at lo
>
> status: up
>
> - net type: o2ib
>
> local NI(s):
>
> - nid: 10.12.170.47 at o2ib
>
> status: up
>
> interfaces:
>
> 0: ib0
>
> - net type: tcp1
>
> local NI(s):
>
> - nid: 10.10.170.47 at tcp1
>
> status: up
>
> interfaces:
>
> 0: em1
>
>
>
> [Tue Dec 1 11:07:55 2020] LNetError:
> 2127:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to
> 10.110.0.21 at tcp from <?>
>
> [Tue Dec 1 11:08:01 2020] LustreError:
> 1792:0:(mgc_request.c:249:do_config_log_add()) MGC10.10.0.91 at tcp1: failed
> processing log, type 1: rc = -5
>
> [Tue Dec 1 11:08:08 2020] LustreError:
> 2169:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
>
> [Tue Dec 1 11:08:19 2020] LNetError:
> 2127:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to
> 10.110.0.22 at tcp from <?>
>
> [Tue Dec 1 11:08:30 2020] LustreError: 15c-8: MGC10.10.0.91 at tcp1: The
> configuration from log 'lustre-client' failed (-5). This may be the result
> of communication errors between this node and the MGS, a bad configuration,
> or other errors. See the syslog for more information.
>
>
>
> client# lctl ping 10.10.0.91 at tcp1
>
> 12345-0 at lo
>
> 12345-10.110.0.21 at tcp
>
> 12345-10.10.0.91 at tcp1
>
>
>
> Any suggestions will be greatly appreciated!
>
>
>
> Many thanks,
>
>
>
> Mark
>
>
>
> Dr Mark Lundie | Research IT Systems Administrator | Research IT |
> Directorate of IT Services | B39, Sackville Street Building | The
> University of Manchester | Manchester | M1 3WE | 0161 275 8403 |
> ri.itservices.manchester.ac.uk
>
>
>
> Working Hours: Tues - Thurs 0730-1730; Fri 0730-1630
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20201201/67c784f2/attachment.html
> >
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
> ------------------------------
>
> End of lustre-discuss Digest, Vol 177, Issue 4
> **********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20201204/a735dae1/attachment.html>
More information about the lustre-discuss
mailing list